Reference

Copper to Fiber: 2000s-Era Network Debt and Short-Term Paired Fixes

Nov 14, 2026 · Yoonsun Lee

The helm: one manifest per flow steers four duties. Sources on the left (Shopify webhooks; ERP and WMS with EDI and batch files; machines such as radar, VMS, IoT, LiDAR and trackers; shoppers and agents; design surfaces, outbound only unless allowed) flow into the helm, a control layer with one manifest per flow, versioned in git, validated, canaried and rolled back by pointer. It sets four duties. Data direction: inbound, outbound or both per flow, undeclared directions refused, nothing sent out read back. Orchestration: request, event and scheduled paths; load, validate, canary, switch; idempotent jobs; reads serve the last good copy. Security: one key per flow and device, signed messages with size and type limits, consent checked before personal data moves, agents propose and people approve. Scaling: a partition key per flow, bounded queues with backpressure, a memory budget per task, more instances rather than a bigger machine. Outputs on the right: Xano as the system of record, the edge cache with preloaded reads and A/B slots, channels (Merchant API, Meta, Amazon, Shopify), operations screens and the R2 archive. The helm steers three paths: the request path with Node preload at the edge, the event path with a queue feeding Rust and Tokio tasks, and the scheduled path with Xano background tasks.

Related knowledge base: Global Unit Data Loads — Data Type Management · ISO data in UTF-8: space, sea, air and LiDAR · Hyperscaler vs Distributed

The challenge: tech debt that worked before data got big

Most of the systems in this article weren't badly built. The copper circuits between FAA facilities, the HBO GO sign-in path, the way Windows machines took security updates and the nightly ERP batch were all sound designs for the data they were built to carry. Each one rests on an assumption about how much data arrives, how fast, and how much of it at the same moment. Those assumptions were true when the systems went in. They stopped being true as data got big, and nothing in the system announced it.

That's the kind of tech debt this article is about: a sizing assumption that expired while the system kept working.

SystemBuilt forAssumption that held thenWhat grewHow the debt showed
FAA telecom linksradar and voice between facilities, early 2000sa fixed, modest data rate per circuit; a backup was a second circuitmore radar sources, ADS-B once a second from every aircraft, digital voice, longer routes after facilities were consolidated90-second radar and radio blackouts at Newark in 2025; the backup failed with the primary
HBO GO sign-in (2010s)subscribers watching through their cable accountviewers arrive spread over the eveningmillions arriving in the same minute for one premierethe Game of Thrones season 4 premiere outage, 2014
Windows security updatesa fleet patched on a schedulean update is small, tested, and a bad one hurts a few machinesone vendor's content file pushed to millions of machines within an hourthe CrowdStrike outage, 2024: airlines grounded worldwide
Nightly ERP batchdaily stock and order filesthe business can wait until tomorrowmarketplaces, ad feeds and agents that expect changes within minutesstale prices and stock in feeds, oversells, disapproved products

Three things make this debt hard to see:

How the problem scales

The same expired assumption shows up at four levels. Each level sits on the one below it, so a platform that fails at level 1 can't be made safe at level 4.

LevelWhat breaksToo much data in the form ofRecent exampleWhat it means for a BAU commerce platform
1. Fundamental loada component that can't take the volume, rate or simultaneityone big file, one blocking request, one undersized linkNewark copper links (2025); HBO GO sign-in (2014); the nightly ERP batchif a single job, link or database can't take peak load, nothing above it matters
2. E-commerce data streamsthe always-on flows between store, ERP, WMS, marketplaces and ad platformswebhook storms after a bulk edit, product and inventory updates at peak, feed rebuilds, conversion eventsBlack Friday / Cyber Monday peaks; a bulk price change that fires one webhook per variantbacklogs, rate limits (HTTP 429), retries that multiply the load, duplicate orders, stale stock and prices in feeds
3. Hyperscalers that breaka provider's shared region, control plane or configuration systema configuration or data file that grows or changes and is pushed everywhere at onceAWS us-east-1, October 20, 2025 (a DNS race in DynamoDB's automation); Azure Front Door, October 29, 2025 (a configuration change); Cloudflare, November 18, 2025 (a bot-management feature file doubled in size and passed a hard limit in the proxy); Google Cloud, June 12, 2025 (a policy change with blank fields crashed Service Control)one provider's fault stops thousands of unrelated stores at the same moment; your own load was fine
4. Mob AI on BAU platformsthe platform's front door and its business logicvery large numbers of AI agents and crawlers acting at once: scraping catalogs, comparing prices, testing carts and checkouts, trying credentials, injecting instructions into reviews and product textAI crawler and agent traffic growing faster than human traffic; agentic checkout protocols (UCP, ACP) moving purchases to machinesrequest rates set by machines, not people; security controls that are themselves load; attacks that look like ordinary shopping

Why the order matters:

So the challenge isn't only replacing old equipment. It's finding every place, from the wire up to the provider and the front door, where a design still assumes the data is small, slow, spread out or sent by people, and changing how that part takes data in before the next peak, outage or agent wave finds it. The rest of this article follows that through: the FAA example, why new cable alone doesn't fix it, short-term pairings that cover the gap, and why chunking, concurrency and Rust belong in the same budget as the new wire.

This article is the background for the air row of Global Unit Data Loads. That spec says air positions come from at least two providers and that gaps are marked, not filled. This article explains why.

Overview

US air traffic control still moves radar and radio data between facilities over telecom lines, and many of them are copper circuits from the early 2000s or older. The radio part of radar works. The weak link is the wired leg that carries the processed data from the radar site to the room where controllers work. When that leg fails, controllers lose the picture even though the radar is still running.

The long fix is a full rebuild. It takes years, so the useful question for anyone running or depending on this kind of network is what can be paired with the old system now, so that one failure doesn't become an outage.

The fiber migration itself is a 20-year-old problem: it was scoped around 2007, the year HBO on Broadband launched. The same pattern shows up in commerce stacks from the same era: a 2000s ERP, a T1 or MPLS line, a nightly batch file. New links get laid in, but the system behind them is the same, so the risk stays.

A 20-year problem

Moving these networks to fiber isn't a new project. It has been on the table for about twenty years, the same span in which consumer video moved from cable boxes to broadband.

YearAviationBroadband and streaming
early 2000sFAA Telecommunications Infrastructure (FTI) contract replaces older leased circuits, but much of the last mile stays on carrier copperDSL and cable modems become the home connection
2007NextGen modernization program gets under way, promising a digital, satellite-based systemHBO on Broadband launches: premium video delivered over the subscriber's internet line instead of the cable box
2010sNextGen rolls out ADS-B (mandate from 2020), but the facility-to-facility links largely stay as they wereHBO GO and other streaming apps scale; carriers lay fiber to homes and businesses
2025Newark outages over copper linksfiber and 5G are the normal home and mobile connection
2026–2028Brand New Air Traffic Control System replaces copper, radios and radars—

Consumer streaming crossed to fiber because each subscriber's line could be upgraded on its own. The FAA network couldn't move that way: every link feeds a safety-critical system that can't go dark during a cutover, and the endpoints were never replaced. That is why a problem first scoped around 2007 is still causing outages in 2025.

What failed at Newark (2025)

The second outage is the important one: the backup existed, but it didn't survive the same failure as the primary.

Why copper is the problem, and why fiber alone isn't the fix

IssueWhat it means
CapacityThe circuits were sized for an earlier data load. One controller said the wires can't carry the amount of data now sent over them, which leads to shorts and dropouts.
Distance and ageHundreds of miles of copper behave like an old landline network: corrosion, water, heat and construction damage, with each splice a failure point.
Shared pathsA primary and a backup that run through the same conduit, building or carrier fail together, as on May 9.
Old endpointsEven where fiber is installed, the signal is slowed down at the facility because the computers there are too old to take the data at full speed.
Same architectureFAA leadership put it plainly: "We're pulling copper up and putting fiber down, but it's the same data architecture that we've had for 50 years."

The lesson carries over directly: replacing the cable without replacing the endpoints and the data model moves the bottleneck instead of removing it.

The old endpoints: Compaq computers and floppy disks

The "old endpoints" are literal. Reuters reported that the system relies on decades-old Compaq computers, with replacement parts bought on eBay, and the FAA says it will phase floppy disks out of hundreds of facilities by 2028. The FAA hasn't published how much memory these machines have. For scale, typical desktops of the Windows 95 and NT era shipped with about 16–128 MB of RAM, against 8 GB or more in a current phone.

Memory isn't the main reason they can't take the new load, though. Radar track data is small by modern standards. The limits sit in the rest of the stack:

Airports, facilities and units

How big is the replacement in units? Smaller than it sounds for computers, and much larger for everything around them.

CountApproximate number
Public-use airports in the USabout 5,000 (about 19,000–20,000 including private airfields)
Airports with a control towerabout 520: roughly 260 run by the FAA and roughly 260 contract towers
Approach control facilities (TRACONs)about 147
En-route centers (ARTCCs)about 21
Facilities reported to run Compaq computers313, from a single lightly sourced report; unconfirmed
Compaq machines, estimatedlow thousands (313 facilities × a handful each ≈ 1,000–2,000); the FAA hasn't published a count

What the rebuild buys, as reported:

ItemCount
Radios27,625
Network connections (fiber, satellite, wireless)5,170
Radars612
Digital voice switches462
New surface radars44 airports

The computers are a rounding error. At about $5,000 each, 2,000 new machines cost around $10 million, less than 0.1% of the $12.5 billion already funded. The money and the years go to radios, radars, the network and new software, and above all to certifying that software for safety, integrating it with radar, ADS-B and voice, training controllers, and switching each facility over while it stays live around the clock.

Adding RAM wouldn't fix it, which is why the FAA's plan replaces the computers and software along with the copper. It's the clearest case of tech debt that worked before data got big: each part was sound for the load it was certified for, and the load moved on.

From copper to floppy: the two paths

Live copper data doesn't go to a floppy disk. The two are separate paths, and the floppy sits where systems can't talk to each other over a network.

PathRouteSpeedCarries
Liveradar site → digitized track data → telecom circuit (copper or fiber) → modem or serial interface → automation computer → controller's displaycontinuous, secondspositions, tracks, voice
Batchdata arrives at or is produced on one computer → copied to a 1.44 MB floppy → carried by hand to a machine with no network link, or only a floppy drive → loadedmanual, hours or daysweather and other periodic information; typically maps, sector boundaries, frequency tables and software or configuration updates

The batch path is "sneakernet": a person carrying data between systems that can't connect, sometimes on purpose, as an air gap. The FAA hasn't published the workflow for each facility, so the route above is the general pattern, not a documented procedure.

It survives because the receiving machines are certified with a floppy drive as their only input, and because an air gap keeps network faults away from the safety system. It no longer fits because:

The commerce equivalent is the nightly CSV export or the spreadsheet emailed to the warehouse. The fix in both cases is to keep the separation but replace the hand-carried copy with a declared, verified flow.

An air gap that is AI-secure

An air gap is worth keeping for safety systems, and AI raises the bar for what crosses it. Attackers can generate malicious files, convincing documents and injected instructions faster than before, and an AI agent with access on both sides would be a bridge across the gap. A modern gap is one-way, verified and closed to agents:

ControlWhat it doesWhy it matters for AI
One-way transfera data diode or unidirectional gateway, enforced in hardware, carries data into the safe side; nothing can travel backan agent or malware on the outside can't reach in to read, steer or exfiltrate
Allow-listed formats onlyonly declared data formats cross (for example a weather grid, a sector map, a signed update package); executables, macros, scripts and free-form documents are refusedAI-generated payloads hide in formats with active content; a fixed format has nowhere to hide them
Content disarm and reconstructioneach file is parsed and rebuilt from its data fields only, discarding everything elsestrips hidden content without needing to recognize it, so new AI-made variants are handled the same as old ones
Signed manifestevery transfer carries a manifest of files, sizes and hashes, signed with a key held by a named person or a hardware module, never by an AI agentan agent can propose a change but can't authorize it
Deterministic validationschema, range and size checks run in a memory-safe parser (Rust) on the receiving side; a failure quarantines the transferrejects oversized, malformed or out-of-range data before it's loaded, the failure mode behind the 2024 and 2025 outages
Text treated as datafree-text fields (notes, NOTAM text, descriptions) are length-limited, stripped of control characters and never executed or passed to a model as instructionsblocks prompt injection riding inside routine data
A/B load with human sign-offthe transfer loads into an inactive slot, is checked, and is switched live only after a named person approves; the previous slot stays for rollbacka bad transfer never reaches the live system, and recovery takes one switch
No agents on the safe sideno AI agent holds credentials on the safe side; any model used there is a fixed, hash-pinned version with no network access and no ability to change its own inputsa compromised or confused model can't widen its own reach
Nothing flows back to traindata from the safe side isn't used to train or tune outside models; any reporting out goes through its own one-way path with redactionthe safe side's data can't leak through a model
Full audit trailevery transfer, check, approval and switch is logged with who, what and the hashesan incident can be traced to one transfer and one approver

This is the helm applied to an air gap: one declared flow, inbound only, with its own keys, size limits, validation and staged switch. People approve, machines verify, and AI may propose but never carry or authorize anything across.

Three outages from three industries share the same shape. In each, a component built for an earlier load took the new load as one large, blocking unit. When it choked, everything behind it stopped.

IncidentWhat arrived all at onceWhere it blockedWhat chunking and concurrency change
HBO GO, Game of Thrones season 4 premiere (April 6, 2014)Millions of viewers signing in at the same minute, a spike HBO put down to "excessive user demand"The sign-in and entitlement path. The video was already delivered in small segments, but the check on who may watch ran as a blocking request against a cold backend.Warm the catalog and entitlements into cache before the spike, answer sign-ins on a non-blocking event loop, and admit users in batches instead of all at once. Field note from the team that did the setup: the original stack was threaded Ruby with Adobe's Ruby-based entitlement system; the lesson became the Node preload strategy.
CrowdStrike update on Microsoft Windows (July 19, 2024)One faulty content update sent to every machine at onceInside the Windows kernel. About 8.5 million devices crashed (Microsoft's estimate), including airline check-in, crew scheduling and dispatch systems. Delta alone cancelled thousands of flights over several days.Ship updates in stages (a small ring first, then wider rings), validate each piece before loading it, and isolate the component so that one bad input can't take the whole host down.
Newark radar and radio link (April 28 and May 9, 2025)More radar and voice data than the copper circuits were sized forThe telecom link, then the facility's old computers, which still slow down data arriving over the new fiberCarry the feed as bounded, numbered messages over diverse paths, with backpressure, so an overload degrades the picture instead of cutting it.

At HBO GO the radar link's role was played by the sign-in check, and at airlines by the kernel. In all three cases the medium got faster or the audience got bigger, but the processing model stayed the same.

Why Rust, threads and chunking belong in the legacy tech-debt budget

Legacy tech debt is usually scoped as old equipment: copper lines, old servers, an old ERP. The incidents above show the other half of the debt, which is the processing model. A system that takes data as one big blocking unit, on one thread or on threads that share locks, fails the same way on fiber as it did on copper. That is the FAA's own point: "We're pulling copper up and putting fiber down, but it's the same data architecture that we've had for 50 years."

So when a migration is budgeted, these items belong on the same line as the new wire, not in a later phase:

Legacy patternHow it fails under too much dataFold-in fixWhere it sits in this stack
Whole-payload processing (one file, one request, one batch)One oversized or bad payload blocks or crashes everything behind itChunking. Split data into bounded pieces, each with a sequence number, size cap and hash, so each piece can be checked, retried and resumed on its own.Cloudflare Queues batches; R2 multipart uploads and range reads; COPC tiles for point clouds; HLS/DASH segments for video
One thread, or threads that share locksOne slow call stalls every user; lock contention collapses throughput at peakConcurrency without shared state. Async tasks on a work-stealing thread pool (Rust and Tokio), where the compiler rejects data races. Many isolated single-threaded instances where that suits better.Tokio in Cloudflare Containers for heavy parsing; Workers isolates for the edge; Xano tasks for the system of record
No backpressureThe sender keeps pushing until the link or the process falls over, as the copper circuits didBounded channels and load shedding. When input exceeds capacity, slow the sender, queue up to a limit, and drop the lowest-priority work first.Queue limits and dead-letter queues; per-tenant rate limits at the edge
Garbage-collection pauses and unpredictable memoryLatency spikes at exactly the moment of peak loadPredictable memory. Rust has no garbage collector, so memory use per message is known in advance.Rust/WASM parsers for large feeds (AIS, LAS, EDI)
All-at-once rolloutOne bad update reaches every machine together (CrowdStrike)Staged delivery. Canary rings, validation before loading, and a fast stopVersioned Worker deploys with gradual rollout; the release gate
Cold start at the spikeThe backend is asked for everything at the busiest moment (HBO GO)Warm before the spike. Preload catalogs, entitlements and keys into cache ahead of a known eventKV and cache warm-up before launches and drops

Two cautions:

The result is a single budget line: the new link, the new processing model, and the staged path to switch from one to the other. Paying only for the link leaves the debt where it was, which is why a problem first scoped around 2007 was still producing outages in 2025.

OTT vs data streams

OTT means over-the-top: video delivered over the public internet, "over the top" of the cable or satellite operator's own network instead of through a set-top box. HBO GO, Netflix, Disney+ and YouTube are OTT services.

OTT and commerce or machine data streams share delivery mechanics, but the data has a different shape.

OTT video streamingCommerce and machine data streams
Directionmostly one-to-many and downstream: the same file goes to millions of viewersmostly many-to-one and upstream: millions of different messages arrive at one system of record
Cachinghighly cacheable; every viewer gets the same segments from the CDNeach message is unique, so it can't be cached on the way in; only outputs (preloaded reads) can
ChunkingHLS or DASH segments of about 2–6 seconds, each in several bitratesqueue batches, multipart uploads, COPC tiles, SenML batches
Under overloadadaptive bitrate lowers quality and the viewer keeps watchingevery record matters, so nothing can be sent at "lower quality"; hold, coalesce or deprioritize instead (backpressure and load shedding)
Order and lossa late segment is skipped; a lost frame is barely noticedorder matters within a partition; a lost or duplicated message is a wrong stock count or a double order, so idempotency keys and gap marking are required
Latencyvideo on demand buffers seconds ahead; live runs at about 2–30 secondsdepends on the source: under a second for radar and operations screens, minutes for feeds, hours for reconciliation
Securityper session: DRM and signed URLsper message: signatures, a key per device, a consent check

What carries over from OTT:

OTT solved the outbound, identical-content problem about twenty years ago. Commerce and machine data is mostly the inbound, unique-message problem, so it borrows OTT's chunking, manifests and edge preload, and adds ordering, idempotency and per-message security.

Chunking compared: OTT, AI, gaming and data streams

Chunking is the shared technique, but each field cuts at a different boundary and protects something different.

OTT videoAIGamingCommerce and machine data
Unita segment of a few seconds, per bitratea passage of a few hundred tokens for retrieval; a batch of requests for inference; a stream of tokens outa world cell or level section; a texture mip level; a level-of-detail mesh; a network snapshot or delta per tickone record change, a queue batch, a file part, a tile
Where it's cutat keyframes, so each segment plays on its ownat meaning (headings, paragraphs), with some overlap so context isn't cut mid-thoughtby distance and visibility around the playerby partition key (tenant, market, product, device) and a size cap
Budget it protectsnetwork bandwidth and the playback bufferthe context window and token cost; GPU memorythe frame budget (16.6 ms at 60 fps) and GPU memoryqueue depth, memory per task, API rate limits
Loaded firstthe next segments at the current bitratethe highest-ranked chunks that the user is allowed to seewhat's in view and near the playerchanges ranked by priority: price and stock before descriptions
Under overloadlower bitratefewer chunks, a smaller model, or a fallback with a flaglower level of detail, lower-resolution textures, fewer far objectscoalesce, hold in the queue, shed low-priority work
Manifestthe .m3u8 or MPD playlistthe retrieval index and its metadatathe asset bundle or world-partition manifestthe helm's flow manifest
Delta updatesnew segments onlynew or changed chunks re-embedded onlypatches shipped as changed chunks onlyhash-gated changes only

Rules all four share:

The game-engine frame budget as the origin of "bind before you allocate" is covered in Hyperscaler vs Distributed; AI weighting and ranking of retrieved rows are in the same article.

The approach: preload, background tasks and bounded memory

Why more hyperscaler VMs won't restore business as usual

A worked example shows how big the growth is. Take a catalog of 10,000 products.

Then: nightly batchNow: always-on streams
Unit of changeone row per productone row per variant, per market, per channel
Records10,000 products10,000 × 5 variants × 10 markets × 5 channels (Shopify, Google, Meta, Amazon, the site) = 2.5 million
Updates per day1 (the nightly file)about 20 (stock moves, price changes, feed refreshes)
Record updates per day10,00050,000,000
Growth—5,000×

Your multipliers will differ, but the shape doesn't: every new market, channel and agent multiplies the others. Scaling virtual machines to match doesn't bring back business as usual, for four reasons:

The answer is to change where and when the work happens. Do it before the request, off the request path, and in bounded memory. Then shrink the 5,000× before it reaches compute.

Three paths, three tools

PathToolWhat it doesMemory and load rule
Request path (a shopper, an agent, a controller's screen)Node preload at the edge: a non-blocking event loop plus data warmed into cache (KV) before the spikeAnswers from precomputed data only: catalog, prices per market, entitlements, consent state. Nothing is computed or fetched from the system of record during the request.Read-only, small objects, cached; a cache miss returns the last good version and queues a refresh, never a blocking call
Event path (webhooks, feeds, scans, telemetry)Rust and Tokio background tasks as queue consumers, in Cloudflare Containers for heavy parsing and Workers for light reshapingParses, validates, reshapes, hashes and diffs each chunk, then writes only the changes onwardStreaming parse (the file is never loaded whole), bounded channels, fixed-size buffers reused between chunks, a memory budget per task (a Worker isolate has 128 MB), no garbage-collection pauses
Scheduled path (reconciliation, rebuilds)Xano background tasks on the system of recordBatch upserts from the queue, nightly and hourly reconciliation against Shopify and the ERP, retries of dead-lettered messages, snapshot for the integrity check, and a rebuild of the preload cacheIdempotent by key; batched writes; runs off-peak where it can; one task per job, so a slow job doesn't hold the others

Shrink the multiplier before it reaches compute

Most of the 5,000× is repetition. Background tasks remove it before anything expensive runs:

In the example above, hash-gating and coalescing typically leave a small fraction of the 50 million updates to send. The exact figure depends on the catalog, so measure it on your own data before sizing anything.

How this ties back

Incremental A/B data loads and horizontal scaling by design

Preload and background tasks only pay off if the system is designed to change a little at a time and to grow by adding instances. The opposite, brute force, is the default in many stacks: every change recompiles, re-serves and re-uploads everything.

Brute force vs built in

Brute force: everything compiles, serves and updatesIncremental A/B loads with horizontal scaling built in
Unit of workthe whole site, catalog or feedone changed record, or one partition
Cost per changeproportional to the whole data setproportional to what changed
At 5,000× the change ratefull rebuilds overlap and never finish; the queue grows without endeach change is small, and more instances absorb more changes
Time to liveminutes to hours per rebuildseconds per change
A bad loadreplaces everything at once (the CrowdStrike and Cloudflare shape)stays in the inactive slot and fails validation before anyone sees it
Rollbackrebuild the previous version, as slowly as the bad oneswitch the pointer back
Peak handlinga bigger machine (vertical scale), with the same lockmore stateless instances working separate partitions (horizontal scale)
Provider dependenceone large job on one provider's VMssmall jobs that can run at the edge, in Xano or in a second region

The A/B data load

The live data sits in two slots, A and B. One serves traffic and the other is loaded.

  1. Load the inactive slot incrementally. Copy forward from the live slot, then apply only the changed records (hash-gated and coalesced, as above).
  2. Validate before anyone sees it. Check record counts against the expected change, the hash of each partition, every value against its type and range, and the size of each file and object against a hard limit. An oversized or empty file is rejected here, which is exactly the check that would have stopped the 2024 and 2025 configuration-file outages.
  3. Canary. Serve the new slot to a small share of traffic, one market or one channel first, and compare errors, prices and stock between the slots.
  4. Switch with one pointer. A single key (for example active_slot in KV) moves from A to B. Readers pick it up on their next request; nothing is rebuilt.
  5. Keep the old slot for rollback. If anything looks wrong, the pointer goes back to A immediately. The next load then goes into A.

The same two slots support A/B testing: serving slot B to a set share of visitors is a canary that stays on, with results compared per slot.

Horizontal scaling built in

Horizontal scaling isn't something added at the peak. It comes from four design choices made at the start:

Where each piece sits in this stack

SurfaceBrute force to avoidIncremental, horizontal approach
Google Merchant APIresubmitting the full product file on every changeproductInputs insert or delete per changed offer, batched; a full file only for the first load
Shopifyre-syncing the whole catalogbulk operations once for the initial load, then per-item mutations driven by webhooks
Static site and Webflow pagesa full site rebuild and publish for each content changeincremental builds and per-item or per-page publishing; data rendered from the edge instead of compiled into pages
Xano (system of record)one task that reprocesses every tablebatched upserts per partition from the queue; scheduled reconciliation per partition
Edge cache (KV)flushing everything after a changeA/B slot keys per partition and an active_slot pointer
Files and 3D assets (R2)overwriting objects in placenew versioned objects plus a pointer switch; the old version kept for rollback
Worker deploysreleasing everywhere at oncegradual rollout by percentage, with the previous version ready

The result is that business as usual survives growth. A change costs what it changes, a bad load never reaches readers, and a peak means more instances working more partitions, not a bigger machine waiting on the same lock.

Machine planning: radar, VMS, IoT, LiDAR, spatial data and serving

Most of the 5,000× doesn't come from people. It comes from machines: radar, cameras, sensors, scanners and trackers that send data all day whether anyone is looking or not. Machine planning means sizing each source before it's connected and deciding, per source, what stays at the site, what is processed in the background, and what is served to whom.

The rates below are orders of magnitude for planning. Measure your own devices before sizing anything.

SourceTypical outputRate per deviceKeep raw whereProcess in the backgroundServe
Radartarget reports and tracks (ASTERIX: category 48 for single-radar plots, 62 for system tracks); the raw radar video stays at the radara terminal radar sweeps about every 4.8 s, an en-route radar about every 12 s; track reports are kilobits to low megabits per secondat the radar site and the facilitytrack fusion with ADS-B and multilateration; gap and disagreement flagslive tracks to operations screens; delayed, leg-level positions to customers and agents
ADS-B and AISposition reportsabout once a second per aircraft; every few seconds to minutes per ship, depending on speedprovider archive; our raw copy in R2reduce tracks to legs; cross-check providerslegs and ETAs, never a single provider's silence as a status
VMS (video management: cameras)continuous H.264/H.265 videoabout 2–8 Mbps per 1080p camera, more for 4Kat the site, on the recorder; never shipped raw to the clouddetection at the site or edge, which turns video into events and short clipsevents, clips on request, counts; never a live stream to agents
IoT sensorssmall readings (SenML in CBOR over MQTT or CoAP)tens to hundreds of bytes per reading, every few seconds to minutes; the volume comes from fleet sizeR2, compressedcoalesce, check ranges, flag cold-chain breachescurrent value plus alerts; history as rollups
LiDARpoint clouds (LAS/LAZ, E57; phone scans as USDZ)a terrestrial scanner captures up to about a million points per second; one scan is gigabytesR2, as LAZ and COPCstreaming parse in Rust, measured dimensions, QA against the spec, a GLB meshdimensions and QA result to Xano; the mesh and COPC tiles to the viewer by range request
Spatial data (KML, GeoJSON, vector and 3D tiles)derived positions, routes, footprintschanges when the sources changeR2, versionedprecompute tiles (for example a single PMTiles file) and route KML per legstatic tiles from R2 through a Worker with range reads; nothing rendered per request

The planning steps

  1. List every machine source with its count, rate per device, message size and cadence.
  2. Compute the steady rate and the peak. Peak is not the average times two. Plan for the reconnect storm: after an outage, every device that buffered data sends it at the same moment.
  3. Decide what never leaves the site. Raw video, raw radar returns and full-resolution point clouds stay local or go to R2 in bulk off-peak. This is the biggest lever in the whole plan.
  4. Size the site buffer for the outage you expect. Buffer = rate × hours offline. If the site must survive 24 hours without a link, the buffer has to hold 24 hours of everything that will be sent later.
  5. Choose a partition key per source: site, device or product. That key drives the queue, the A/B slots and horizontal scaling.
  6. Tier the storage. Hot (current state, in Xano and KV), warm (recent history, rollups), cold (raw archive in R2). Each tier has a retention period.
  7. Budget the serving side separately. Who reads what, how fresh it has to be, and from which preloaded copy. Operations get live data over a socket; customers and agents get delayed, summarized data from the cache.

Worked example: one warehouse in Korea

SourceCountRatePer dayPlan
Cameras (VMS)404 Mbps each, 160 Mbps totalabout 1.7 TBstays on the site recorder; only events and clips are sent (megabytes a day)
Sensors (temperature, humidity, doors)500100 bytes every 10 sabout 430 MBsent as SenML/CBOR batches; coalesced and range-checked in the background
LiDAR scanner1a few scans a day, about 1–2 GB each as LAZa few GBuploaded off-peak to R2; dimensions and QA results to Xano
Trackers on outbound pallets20one fix a minuteunder 10 MBsent through the non-browser chain; legs to customers

Video is more than 99% of the raw volume and none of it needs to leave the site. Without that one decision, the link, the cloud bill and every downstream system are sized for 1.7 TB a day. With it, they're sized for a few gigabytes. Brute force ships everything and scales VMs to match. Machine planning decides first what each machine's data is for.

How it connects

Orchestration: the helm

Preload, background tasks, A/B loads and machine planning are parts. The helm steers them. It's the control layer that decides, for every flow, which way data may move, what runs when, under which keys and limits, and how far it may scale. Without it, each part makes those decisions on its own, and the system is again one big job nobody can steer at the peak.

The helm is declared, not coded into each service: one manifest per flow, versioned in git, validated and rolled out in stages like any other data. Teams on Kubernetes will recognize the idea from Helm charts, where one values file sets replicas, resource limits, network policy and secrets for a release. Here the same idea covers the edge, the queues and Xano.

The four duties

DutyWhat the helm decidesRulesWhere it's enforced
Data directionwhich way each flow may move: inbound to the system of record, outbound to channels, or bothevery flow has a stated direction; an unlisted direction is refused; design surfaces are outbound only unless a flow is explicitly allowed inbound; no output is read back as inputedge routes and Worker bindings; Xano endpoint auth; per-flow allow list
Orchestrationwhat runs, in what order, on which path (request, event or scheduled)request path reads preloaded data only; heavy work is a queued background task; scheduled jobs are idempotent and one task per job; dependencies are explicit (load → validate → canary → switch)Cloudflare Queues and cron triggers; Rust/Tokio consumers; Xano background tasks
Securitywhich identity and key each flow uses, what it may touch, and what's checked on each messageone key per flow and per device, never shared; signatures on every inbound message; consent or legal basis checked before personal data moves; size and type limits checked before parsing; agents propose, people approveedge (mTLS, HMAC, Turnstile, WAF, rate limits); key store; consent layer; integrity tripwire
Scalingthe partition key, queue limits, memory budget and instance range per flowscale by adding instances per partition, never by one bigger machine; bounded queues with backpressure; a memory budget per task; a load-shedding order (what's dropped first)queue settings; Worker and Container limits; Xano task schedule

One manifest per flow

An example for stock and price updates going from Shopify to Google's Merchant API:

flow: offers-to-merchant-api
direction: outbound              # Xano (system of record) → Google; never read back
source: xano.offers              # canonical record, not the Shopify webhook itself
sink: google.merchant.productInputs
path: event                      # queued background task, not on the request path
partition_key: [tenant, feed_label, offer_id]
change_filter:
  hash_fields: [price, sale_price, availability, title, link]
  coalesce_window: 60s           # 20 stock moves in a minute → 1 update
load:
  slots: [A, B]
  validate:
    max_batch_bytes: 5MB
    max_records_per_batch: 1000
    price_currency: required     # ISO 4217
  canary: { share: 5%, by: feed_label }
  switch: kv:offers/active_slot
security:
  credential: merchant-sa-tenant # one service account per tenant and flow
  signing: hmac-sha256           # inbound webhooks to the edge
  personal_data: none            # an offer carries no customer data
scaling:
  queue: offers-merchant
  max_queue_depth: 200000
  consumers: { min: 1, max: 50 }
  memory_budget_mb: 96
  shed_order: [refresh_unchanged, low_priority_markets]
on_failure:
  read: serve_last_good
  write: hold_in_queue           # never drop, never publish an unvalidated batch

Rules for the helm itself

On Kubernetes, the same manifest maps onto Helm chart values: replicas and a HorizontalPodAutoscaler for scaling, resources.limits.memory for the memory budget, NetworkPolicy for data direction, and Secrets per release for keys. The duties are the same whichever platform runs them.

The long-term fix: the Brand New Air Traffic Control System

These figures are as reported by the sources below, through autumn 2026. They haven't been independently checked here, so confirm the latest status before citing them.

TrackWhat's plannedReported status
TelecomAbout 5,170 new high-speed connections over fiber, satellite and wirelessabout 67% of copper replaced by August 2026; target around Q3 2027. A new Philadelphia–New York fiber cable for the Newark facility went online in May 2026.
Equipment27,625 radios, 462 digital voice switches, 612 radars, new surface radars at 44 airports, electronic flight stripsin progress
Data networkA digital network on top of the new links ("Putting fiber in the ground is not enough")software side underfunded; needs more from Congress
FundingJuly 2025 law: $12.5 billion, including $4.75 billion for telecom, $3 billion for radars and $1.9 billion for new en-route centersdown payment on a larger total
OversightGAO (GAO-26-107992, September 2026): no lifecycle cost estimate or master schedule, so there's a risk of disruptions or of relying on old systems for longeroverall deadline December 2028

What the signals are

Radar itself is radio, not wire. The wire problem starts only after the radar has turned the echoes into data.

SourceHow it worksBandCan it be faked or switched off?
Primary radar, terminal (ASR)sends pulses and times the echoS-band, about 2.7–2.9 GHzno; it sees any aircraft
Primary radar, en route (ARSR)the same, over longer rangeL-band, about 1.2–1.4 GHzno
Surface radartracks aircraft and vehicles on runways and taxiwaysX/Ku-band, about 9–16 GHzno
Secondary radar (SSR / Mode S)asks at 1030 MHz, and the transponder answers at 1090 MHz with ID and altitude1030 / 1090 MHzneeds a working transponder
ADS-Bthe aircraft broadcasts its own GNSS position about once a second1090 MHz, or 978 MHz (UAT)GNSS can be jammed or spoofed; needs cooperation
Space-based ADS-Bsatellites receive ADS-B from orbit, including over oceans1090 MHzas ADS-B
Multilateration (MLAT / WAM)ground receivers time the transponder replies to compute a position1090 MHzindependent of the aircraft's GNSS

Short-term paired fixes

The rebuild runs to 2028. Until then, the useful fixes pair a new path or source with the old one, so a single failure stays a degradation, not an outage. None of them is new technology; they're standard for any network that can't fail.

ProblemPaired short-term fixWhy it helpsLong-term replacement
One copper circuit carries the feedAdd a second path on a different medium and route: fiber plus licensed microwave, or plus a satellite link (LEO or GEO)a backhoe, flood or fire can't take out bothfull fiber, satellite and wireless network
Backup shares the primary's routeAudit the physical route of every primary and backup pair: building entry, conduit, carrier and central officethe May 9 failure mode is found before it happensnetwork designed for diversity from the start
Legacy endpoints can't take fiber speedsCarry the old circuits unchanged over the new fiber: TDM over IP pseudowires (SAToP, RFC 4553; CESoPSN, RFC 5086) and media converters at each endthe new link's reliability arrives without replacing the old computers yetnew endpoints and a native IP data network
A dropout leaves controllers blindPair sources: primary radar plus Mode S plus ADS-B plus multilateration, fused at the facilityif one source or link drops, the others still show the trafficthe same fusion over the new network
GNSS jamming or spoofingCross-check ADS-B positions against radar and multilateration; flag disagreementsone faked source is caught by the independent onesthe same, with better monitoring
Failures are found by controllersMonitor circuit health (error rates, latency, failover events) with alarms ahead of an outagea degrading line is swapped before it failsmonitoring built into the new network

Why the phone-style approach is a pair, not a replacement

Phones find themselves the way ADS-B does: from GNSS, plus cell towers and Wi-Fi. Aviation already uses that idea (ADS-B, space-based ADS-B, crowdsourced receiver networks, Remote ID for drones). It can't replace radar, for four reasons:

It also doesn't touch the copper problem. GNSS positions still have to reach the control facility over the same kind of link.

What a downstream data chain should do

Commerce, freight and logistics systems don't run on the FAA's internal network. They read commercial ADS-B aggregators, carrier messages and FAA SWIM feeds. An upstream outage reaches them as silence, gaps or a late burst of data, not as wrong values. The Global Unit Data Loads spec already sets the rules; here is how they map to the same pairing idea.

Upstream riskPaired fix in the data chainSpec rule
One feed provider goes quietTake air positions from at least two providers, with different receiver networksISO data in UTF-8: air row
Late or out-of-order data after an outageOrder by source time, not arrival time; accept late dataNon-browser data chains
Gaps in a trackMark the gap; never interpolate a position or infer arrivalstore-and-forward rule
Silence read as statusA quiet feed never clears an ETA, a delay flag or a shipment statusair legacy network rule
A burst after reconnectionQueue it (Cloudflare Queues), process each message idempotently by device ID, sequence and source timeidempotency rule
Spoofed or wrong positionsCross-check providers; disagreements beyond a set distance are held for reviewsecurity table

The same pattern in 2000s commerce stacks

The FAA case is extreme, but the shape is familiar from the same era of enterprise systems: broadband-era streaming platforms, SAP ECC, nightly EDI batches over leased lines, and T1 or MPLS links to warehouses.

FAACommerce equivalentPaired short-term fix
copper circuit between facilitiesleased line or VPN between ERP and WMSa second path through the edge (Cloudflare) with signed requests
old computers can't take fiber speedECC or the WMS can't take event-rate updatesbuffer at the edge and send at the rate the old system can take; keep its interfaces unchanged
same 50-year data architecturethe same batch file sent over a faster pipea canonical record in a system of record (Xano) beside the ERP, read by every new surface
radar plus ADS-BERP plus Shopify/marketplace eventsreconcile two sources; flag disagreements instead of trusting either one
backup on the same routebackup on the same cloud region or vendora second provider or region for the paths that can't stop

"Pulling copper up and putting fiber down" without changing the data architecture is the same as a lift-and-shift migration: the line is faster, and the bottleneck moves to the endpoint.

Open questions

Sources