Linux

My control-plane machine vanished for twenty hours. The most useful evidence turned out to be a safety mechanism that did not fire.

A phone screenshot of an error turned into a day of storage work: an audit of 4,921 pulled photo edits found 3,777 byte-identical to their originals, 108 GB reclaimed, a 3.4 TB library moved with an instant rename, and five separate ways a copy or a delete can lie to you.

A multi-day video conversion job hit two unrelated failures in one day, and both had the same shape: one broken precondition outside any file, turned by the script into thousands of fast, confident per-file failures. A circuit breaker, a pinned driver, and a watcher that looks at rates instead of names.

My storage box was showing offline, so I went looking at the storage box. The fault was a network port on a different machine that happened to be the storage box's only route out.

What started as moving two small apps to the control-plane machine became the complete retirement of a third machine as an application host. Dependency mapping first, an explicit 'park that one' that evidence later overturned, a symlink instead of rewriting a hundred thousand paths, 51 GB reclaimed, and two live credentials caught before they replicated.

A sweep after the watchdog repair found the fleet's automation registry listed 13 timers on a machine running 16, two of the missing ones backups. It also nearly 'discovered' a hole that did not exist, because a grep over unit files cannot see configuration that arrives through drop-in directories.

The service whose only job is to tell me when a device stops checking in was dead. The previous handoff said it had died the day before. It had never worked once: its sandbox was built before its script ran, and it was the one unit in its batch that nobody wired to the failure notifier.

I had encrypted backups of my private records and had never once put one back. Proving it worked also revealed that the older archive contained nothing at all.

Hashing every size-collision candidate on my NAS found fifty-two thousand duplicate groups and about 254 GB reclaimable. The interesting part was everything that nearly made the number wrong.

I assumed every machine could reach every other. Checking every direction properly found a quarter of the paths broken, and one repair job reported success while having quietly undone itself.

The control-plane machine had a config file explicitly setting persistent log storage, and its logs were still vanishing. A vendor file with a higher number won by sort order alone, roughly nine days of history had already rotated away, and half the reported symptom turned out to be my own wrong command.

My entire knowledge base had no undo, and adding one turned out to be a real distributed-systems puzzle because the same files live on four machines at once.

My busiest compute machine was cooking itself under real load, so instead of just turning the work off, I built layers of protection that let it keep running without frying.

Turned the fleet into a full mesh where every machine can reach every other one directly, passwordless, without hand-managed keys, laying the groundwork for running work anywhere in the fleet.

Adding one wireless card to the GPU box renumbered the PCI bus and renamed two interfaces, not one, which took down the isolated storage network's gateway through a chain nothing had advertised. The rebuild made the two radios a primary and hot spare, with a measured failover gap and an idempotent service.

Reskinned davidgomez.cc into a Swiss/International typographic system, stood up a GPU box and proved face-embedding runs on it, and brought up a regenerable wiki over every one of my notes, run by hand for now.

Redesigned the month view's day cards into a contact-sheet film motif, polished the marketing site's desktop nav, survived a concurrent-edit collision that wiped my work off disk, and finally figured out what machine I've actually been building on.

An aggressive package cleanup quietly broke my storage box's admin UI through a dependency I didn't know mattered. Then a filename-normalizer bug taught me to trust byte sizes over names when matching files back to a database.