Kill a worker mid-flight — zero jobs lost

01 Hook

If a worker dies mid-job, the work should still finish — not vanish into the void.

02 Context

Solo systems project: FastAPI control plane, React operator console, Postgres as the broker — no Redis/RabbitMQ required.

03 Problem

Queues fail in production when workers crash, claims race, or DAGs leave orphaned dependents. Extra brokers add ops cost most teams don't want.

04 Insight

Transactional enqueue + FOR UPDATE SKIP LOCKED + heartbeats + a reclaimer beats a second infrastructure hop for many workloads.

05 Exploration

Built claim loops, zombie reclaim, exponential backoff/DLQ, idempotent enqueue, batch DAG APIs, signed webhooks, Prometheus/Grafana, production fail-closed defaults.

06 Decision

Chose Postgres as the source of truth so business writes and job inserts commit together — and proved crash safety with a chaos monkey in CI.

07 Solution

Shipped HydraTask: API + workers + reclaimer + UI, with cancel/retry, multi-tenant keys, and a demo kit for portfolio walkthroughs.

08 Impact

Chaos run: 10,000 jobs, 5 workers, kill every 5s — completed 10,000, dead-letter 0, lost 0. Open repo with CI badge and production checklist.