Designing document-processing state machines
A pipeline is easier to trust when every transition has an owner, an invariant, and a cleanup path.
UPLOAD, VALIDATING, PROCESSING, READY, and FAILED are product states, not merely backend details. Retrieval should never see half-written data.
Retries must be idempotent, duplicate inputs need stable identities, and a failed stage should remove or quarantine its partial writes before the next attempt.
For ConceptGraph, that means durable job state and an attempt identity, not just a background promise. A restarted process can find unfinished work; fencing prevents an old attempt from committing over a newer one. READY is the boundary between storing a document and allowing it to influence an answer.