The Procedures That Would Have Prevented the FAA Ground Stop (Part 2 of 2)
This is part two of a two-part analysis of the FAA national ground stop on November 13th, 2026. In part one (here) we ranked the likely technical causes. Our leading hypothesis is an identity collision: the new router was built from a copy of the old router's config, the identity fields weren't changed, and two routers ended up answering to the same name. This post covers the procedures that would have prevented it.
Disclaimer: We have no inside knowledge of how the FAA or its contractor runs changes. Where I say something "would have" helped, that is my opinion based on public reporting.
TL;DR, none of this is a technology problem. Every failure mode in part one has a known prevention. I mapped each one to the ITIL 4 practice that owns it, because a procedure only sticks when someone owns it.
| # | ITIL practice | What it adds | When |
|---|---|---|---|
| 1 | Service Validation and Testing | Lab that mirrors production | Before |
| 2 | Knowledge Management | Mandatory checklist from past successes | Before |
| 3 | Service Configuration Management | Config checked against the record | Before |
| 4 | Release Management and Deployment Management | Rollback decided in advance | Before |
| 5 | Change Enablement | Risk assessment and scheduling | Before |
| 6 | Supplier Management | Contractor follows the same process | Before |
| 7 | Availability Management | Designed for the failure | Before |
| 8 | Monitoring and Event Management | Correlate change with anomaly | During |
| 9 | Incident Management | Major incident procedure | During |
| 10 | Capacity and Performance Management | Contain the flood | During |
| 11 | Service Continuity Management | Timed recovery path | During |
| 12 | Problem Management | Root cause, then the pattern | After |
| 13 | Service Level Management and Continual Improvement | Measure it, prove the fix | After |
Before the Change
1. A lab that mirrors production (Service Validation and Testing). Build and run the new router in a staging lab first: same router models, same software version, same config templates, lab peers standing in for the real neighbors. Load the config, bring up OSPF, bring up BGP, and watch.
A duplicate router ID fight shows up in that lab within minutes. The lab would also catch an HSRP split-brain, an MTU mismatch, and a bad BGP password, all before a single passenger is affected. It doesn't have to be the whole FAA network, just enough to run the cutover once, end to end. In my opinion the cost is a rounding error next to roughly 12,000 canceled flights. No test evidence in the change record, no approval to deploy.
2. A checklist from prior successes (Knowledge Management). L3Harris says it has completed more than 70 percent of the planned copper-to-fiber conversion, and Duffy said the maintenance work had been done "thousands of times before without issue." What worked should be written down and every item made mandatory:
a. Traffic drained off the old router, verified by interface counters, not by assumption. b. Identity fields changed and diffed against the old config by a second engineer. c. Control plane brought up one protocol at a time, each verified before the next. d. Routing table checked against the expected state. e. Soak period before the old router is decommissioned. f. Each step signed off by someone other than the person doing the work.
Pilots run checklists before every flight because memory is unreliable under pressure. I don't know whether the FAA's contractor had a checklist like this. If it did, it wasn't enforced.
3. The config checked against the record (Service Configuration Management). A duplicate router ID is a configuration management failure. The configuration management system should hold the authoritative record of every router's identity: router IDs, IP addresses, HSRP virtual IPs. A new config that duplicates any of them gets flagged before deployment. Here is the kind of line that has to differ (example values):
! old router (in production)
router ospf 1
router-id 10.255.0.1
! new router, cloned config, not yet edited
router ospf 1
router-id 10.255.0.1
Same ID, two routers. The system catches that, not a human remembering to check. This is the most direct prevention for our leading hypothesis!
4. Rollback, rehearsed and pre-decided (Release Management and Deployment Management). The way back has to be decided before the change starts, not debated during the incident:
a. The old router's config saved and restorable in one command. b. The old router left powered and cabled through the soak period, not pulled the moment the new one comes up. c. A written rollback trigger: if sessions flap for more than a few minutes, or more than a set number of centers report issues, roll back, no meeting required.
The FAA says the network stabilized once L3Harris blocked the two locations. That's the rollback trigger you want written down in advance: isolate the site you just touched. The ground stop lasted more than four hours. In my opinion a rollback in the first fifteen minutes might have kept this local instead of national. But nobody's first rollback should be during a real incident on a live safety-critical network, so rehearse it. The backout plan belongs in the change record before the work starts.
5. Risk communicated up the chain (Change Enablement). Before the window opens, the change goes up the chain with four things attached: what is being changed, the blast radius, the risk assessment, and the rollback plan. Here the blast radius was the whole country, since work at one remote site took down communications at 17 of 22 centers. Someone senior should have acknowledged that before the work, not discovered it during the outage.
"Routine maintenance" is the tell. Anything touching a live national safety system is a Normal change, not a Standard one: documented risk assessment, change authority review (a CAB in many shops), backout plan attached. Change Enablement also covers scheduling. According to Duffy, the Amarillo router was replaced weeks before El Dorado, and the El Dorado work triggered it. So every change has to be checked against the changes already made nearby, not just against the old router it replaces. One site, verify it clean, then the next only works if "clean" includes the sites you've already done. That is the ITIL guiding principle "progress iteratively with feedback" applied to a router swap.
6. The supplier held to the same standard (Supplier Management). Duffy said of L3Harris, "They own this network." Whoever owns it, the contract has to carry the change discipline: the supplier follows the same change process, submits to joint change review, and is measured on it. Outsourcing the work doesn't outsource the accountability.
7. Designed for the failure (Availability Management). Ask what happens when this fails, and design the answer in. For a router cutover that means control-plane policing, which rate-limits routing protocol traffic so one flooding router can't overwhelm its neighbors, plus route flap dampening and, for OSPF, LSA throttling. All three are standard features. I can't see the FAA's configs, but a flood that reached 17 centers suggests none of them was limiting it.
During the Outage
8. Find it in minutes (Monitoring and Event Management). The FAA grounded every flight because it could not locate the cause of the overload. Event correlation fixes that: a change in progress at site X plus an anomaly starting at site X in the same minute equals the answer. With it, engineers look at El Dorado, the site worked on that afternoon, and the routers it talks to. Without it, they hunt. This is the biggest reducer of time-to-restore in the whole incident.
9. Run the incident like an incident (Incident Management). A major incident procedure means predefined severity levels, a single incident commander, a war room, and a comms cadence. The ground stop decision gets made with information, and restoration is coordinated across 22 centers instead of improvised.
10. Contain the flood (Capacity and Performance Management). A thousand connection attempts every few seconds overwhelmed router control planes, which makes this a capacity event. The controls from item 7 are what stop one router's flood from becoming 17 centers' outage.
11. A timed way back (Service Continuity Management). The ground stop was, in effect, the continuity response: move to a safe state. What was missing, as far as I can tell, is a rehearsed recovery path with recovery time objectives. More than four hours to restore suggests the plan was the ground stop itself.
After
12. Find the real cause, then the pattern (Problem Management). Problem Management does the root cause analysis and writes the known error record. The proactive half matters more. Reuters noted prior FAA telecom failures: errant fiber cuts, accidental deletion of files, a melted component. And on November 19, 2009, a Harris technician incorrectly configured an FTI router at Los Angeles Center and 129 air traffic facilities lost telecommunications, according to the DOT Inspector General. That was the same kind of change failing the same way, 17 years earlier. In my opinion those were warning signs of one underlying problem, aging infrastructure changed under pressure, and Problem Management tracks that as a trend instead of one-offs.
13. Measure it, then prove the fix (Service Level Management and Continual Improvement). The FAA's telecom service has availability targets, and one router taking down 17 of 22 centers is a design failure against them. Service Level Management forces that architecture conversation. Continual Improvement makes the lesson stick: every incident feeds an improvement register with owned actions and verified outcomes, so the fix gets done and checked, not just filed.
Where the Money Should Go
Secretary Duffy used the outage to ask Congress for $50 billion more to modernize air traffic control, on top of the $12.5 billion Congress approved in 2025. The money is probably needed. But the copper-to-fiber upgrade that caused this outage is the modernization, and new fiber installed with the same procedures can fail the same way. If I were advising on how it gets spent, I would fund the lab and the config record first. They are the cheapest items on this list, and they catch our leading hypothesis before it reaches a live site.
Need help building a change process that sticks? Email us at Hello At PatriotConsultingTech.com
Sources and references:
- Part one of this series, with the reported facts and the hypotheses (here): /blogs/2026/11/15/faa-ground-stop-how-one-router-grounded-america
- FAA, Nationwide Ground Stop: Questions and Answers (cancellations, L3Harris blocked the two locations): https://www.faa.gov/newsroom/nationwide-ground-stop-questions-and-answers
- Reuters, Nov 14, 2026 (prior FAA telecom failures): https://www.reuters.com/business/aerospace-defense/us-aviation-shutdown-blamed-l3-harris-installed-router-2026-11-14/
- Reuters, Nov 15, 2026 ("thousands of times before without issue"): https://www.reuters.com/world/us/us-air-travel-disruptions-continue-into-third-day-2026-11-15/
- CBS, Face the Nation transcript with Secretary Duffy, Nov 15, 2026 (Amarillo work done weeks ago): https://www.cbsnews.com/news/sean-duffy-transportation-secretary-face-the-nation-transcript-11-15-2026/
- DOT Office of Inspector General, report CC-2010-012 (2009 outage): https://www.oig.dot.gov/sites/default/files/WEB%20FILE2_FTI%20%20Nov%2019%20outage-6-18-10_Final%20Signature%28a%29%20%282%29.pdf
- 70 percent of the conversion completed, "They own this network": Al Jazeera, Nov 15, 2026 (https://www.aljazeera.com/news/2026/11/15/what-forced-the-us-to-ground-all-planes-a-router-switch-we-now-know)
- $50 billion more on top of $12.5 billion: USA Today, Nov 14, 2026 (https://www.usatoday.com/story/news/politics/2026/11/14/democrats-oversight-congress-investigation-nationwide-ground-stop-trump/92781114007/) and AFP via TechXplore (https://techxplore.com/news/2026-11-copper-wiring-flight-disruption-trump.html)
- ITIL 4 practice names: PeopleCert (https://www.peoplecert.org/pages/ITIL4-practices) and PeopleCert training-material updates (https://www.peoplecert.org/Partners/support/Articles/learning-resources-updates/_Local/official-training-material-updates)
- ITIL 4 guiding principles: PeopleCert presentation (https://www.peoplecert.org/news-and-announcements/2024/-/media/2048efb812304872960038ec981a791a.ashx)
- ITIL (Version 5) keeps the guiding principles and largely the same 34 practices: PeopleCert FAQ (https://www.peoplecert.org/help-and-support/faq-itil)
About the author. Joe Stocker is the Founder and Chief Technical Officer of Patriot Consulting, a former Microsoft Security MVP (2020-2026), and author of the book "Securing Microsoft 365." In his spare time Joe volunteers with several Microsoft programs including Microsoft's Defending Democracy (AccountGuard) program, Microsoft Tech for Social Impact, and the Microsoft Software and Systems Academy (MSSA), which serves military service members desiring to transition into the civilian workspace.
Note: The author created this article with assistance from AI. Learn more