FaultLab
A PostgreSQL-backed job runner that makes retries, worker crashes, lease recovery, and cancellation visible—with Python and Go workers.
- 2
- Worker runtimes · Python / Go
- 7,500
- Claims in local claim experiment
- At least once
- Execution semantics
What I was trying to solve.
Background jobs are easy to demonstrate when nothing fails. FaultLab is an engineering lab for understanding what happens when workers disappear, requests repeat, and side effects outlive an attempt.
Watch the working system.
In progress14-second recording from the working local console: submit a flaky job, inspect its failed first attempt and successful retry, then view cancelled jobs. Silent recording; these steps provide a text alternative.
Play the console walkthrough 14 seconds
Follow the data.
A simplified processing flow. Each numbered stage feeds the next; storage and retrieval boundaries stay explicit.
- FastAPI validates and accepts jobs
- PostgreSQL stores jobs and attempts
- Workers claim with SKIP LOCKED
- Leases and heartbeats fence attempts
- Registered handlers execute work
- Retry, cancel, or recover expired leases
The constraints shaped the system.
PostgreSQL is the coordination layer: short transactions claim work atomically, while Python and Go processes share a lease protocol. Submission idempotency is separate from business-effect idempotency. Execution is at least once; arbitrary external side effects are not exactly once.
Where the happy path ends.
Cancellation is cooperative, so a handler that ignores it can still succeed. SKIP LOCKED does not guarantee global fairness. The lab has no tenant isolation, arbitrary-code sandbox, or validated backup strategy. Local benchmark observations are not production capacity or service-level guarantees.
What exists today.
- Implemented a job console with attempt history, payloads, lease status, retries, and cooperative cancellation.
- The documented claim experiment covers 15 runs and 7,500 claims with no duplicate or missing claims in that workload; it does not measure completed execution.
- Integration tests exercise child-process crashes after claim and effect commit, real lease expiry, and recovery. Python and Go workers use the same job and attempt schema.
What I would improve.
Continue testing failure boundaries and more representative handlers. Production use would require stronger access controls, isolation, quotas, and backup/restore validation.