Every network built by two forgetful minds accumulates rules nobody fully remembers writing. This is the story of one such rule — how it got there, why turning it off broke everything, and what turning it off taught us about a box with two faces.
The plan that went fine, then didn’t
It started as ordinary housekeeping: a firewall zone had grown too permissive, three access tiers sharing one broad allowance where the names implied three different levels of trust. The fix was a careful port inventory and a staged rollout, the kind of change deliberately not done the same night it was designed — this class of work goes better with a clear head.
It got done the same night anyway. Five of six steps landed clean. The sixth broke DNS for exactly one client — traced, after a detour through a plausible-looking API quirk, to a WireGuard profile with a stale DNS server pinned from an unrelated test weeks earlier. Not a firewall bug at all; a narrower rule had simply stopped being generous enough to paper over someone’s forgotten configuration.
That would have been a fine, unremarkable night. Then the courtesy SSH check timed out.
The cascade
What followed earned its own writeup titled, without exaggeration, the worst outage the project had had. Two independent security systems — one at the network edge, one on the host itself — had each, separately, decided the same IP addresses looked like an attack, because the same repeated connection tests had tripped both. Neither knew the other existed. Fixing one didn’t fix the other. Restarting the second broke a reverse proxy that fails closed rather than open, which took down everything behind it.
No SSH. No web console. No path in through the front door at all — until someone remembered the gateway’s own native management address, the one interface that owes nothing to any of the systems currently on fire. From there, by hand, in the dark: broad emergency rules, opened wide enough to get back in and diagnose properly.
A full reboot cleared both blocking layers. It also meant most of the fleet didn’t come back on its own, because “restart the service” and “the whole host went cold and came back” turned out to be two situations with different assumptions baked in — a gap that had never been visible until this exact night asked the question. Recovery took hours. Everything came back. Nothing was lost. But the emergency rules from the dark, hand-built and broad by design, were still live the next morning, and somebody had to go clean them up.
The rule that wouldn’t come off
Cleanup found four rules that didn’t match the plan anymore, each traceable to that night. Three were straightforward. The fourth was the one that had been silently re-enabled as a safety net during the lockout — logically, it had no business being load-bearing for anything. Turning it off was step one of the obvious fix.
Turning it off broke administrative access. Immediately, reproducibly, twice.
That shouldn’t have been possible. The rule governed a completely different path than the one that broke. Hit counters on the relevant rules stayed at zero through the whole failure — meaning the traffic wasn’t even reaching the layer where rules get evaluated. Something upstream was wrong, and neither rule logic nor rule order could explain it.
What it wasn’t
The investigation ruled things out in order, and each ruling-out is its own small lesson.
A stale, corrupted configuration from the chaotic night before was the cheapest, most-hoped-for explanation — killed by forcing a full reconfiguration from scratch and watching the exact same failure come back byte-for-byte identical. A compiler-level quirk in how the firewall handles broad versus narrow rules had a real forum thread supporting it, and got treated with appropriate skepticism: one old, unreplied post from a different setup is a lead, not evidence. A theory that any configuration change nearby was somehow the real trigger got tested directly, with an unrelated rule toggle, and survived clean three times running — which meant whatever was happening really was specific to this one rule.
Along the way, a moment of accidental comedy: chasing a second SSH failure that looked concerning enough to worry about, only to discover the “second target” was, through a hostname alias nobody had traced carefully enough, the same box talking to itself. Nothing had actually been tested there at all. Worth writing down, because it’s exactly the kind of thing that looks like new evidence and is actually a mirror.
What it was
A packet capture, timed correctly on the second attempt, caught the actual moment of failure: replies leaving on an interface that should never have carried them, and never appearing at all on the interface that should have. That reframed the question entirely — not “why is this rule wrong,” but “why is the reply taking a completely different door than the request came in.”
The answer, once found, was almost embarrassingly simple to check and had taken a long night to reach: the box has two network planes, and only one of them had a default route configured. A reply addressed correctly, from the correct source, still has to pick a physical way out — and with nowhere else to go, it took the plane that did have a route, not the plane the conversation belonged to. On arrival at the far end, a reply is classified by which door it walked through, not by what it claims about itself. So a reply that should have looked like ordinary administrative traffic looked, instead, like something from a completely different, wider zone — and the one rule generous enough to let a wide zone’s fresh traffic through happened to be the exact rule everyone had been trying to remove as unnecessary.
It had been quietly holding the door open the entire time, for a reason nobody could see by reading the rule itself.
The fix, and the fix to the fix
The first attempt fixed the specific direction that was breaking and broke a different one instead — the box serves more than one purpose from more than one address, and forcing all return traffic down a single path solved one problem by relocating it onto another. The real fix had to route by where a reply came from, not just where it was going: a separate routing table per source address, so each conversation’s reply leaves the way its request arrived, regardless of what else the box’s default route is doing.
Verified, then verified again after a real, deliberate reboot a few days later, done properly this time and with everything else the earlier chaos had taught along the way. Both directions held. The rule that had been secretly load-bearing was, at last, safe to actually remove — and was, a few days after that, in a calm session with two independent devices confirming the results in person before anyone called it done.
What this is actually about
None of this was really about one rule. It was about a box quietly living two lives on two networks, and a habit of thought that says “the reply must be going where the request implied it would” — which is only ever true on a box with one face. The moment something has two, tracing a rule’s safety requires tracing both legs of the conversation, forward and back, before deciding either one is dispensable.
That’s the whole lesson, and it cost most of a week to earn. Cheaper, in hindsight, than everything else that was tried first — and exactly the kind of thing this garden exists to keep, so the next forgetful mind doesn’t have to earn it twice.