Relay
A support-investigation prototype that plans, retrieves evidence, verifies its proposal, and waits for human approval before creating a review ticket.
- 42
- Tests · documented local suite
- 10/10
- Scripted evaluation cases
- Human
- Approval before ticket creation
What I was trying to solve.
A plausible support reply is only one part of an investigation. Relay makes tool choices, supporting evidence, verification, and the boundary between a proposal and an approved action visible.
Follow the data.
A simplified processing flow. Each numbered stage feeds the next; storage and retrieval boundaries stay explicit.
- Issue → persistent SQLite queue
- Worker records investigation plan
- Model selects scoped tools
- BM25 policies + account/service fixtures
- Evidence → plan revision
- Verify exact proposal
- Human approves or rejects
- Idempotent local review ticket
The constraints shaped the system.
A bounded single-agent loop uses Gemini/OpenAI adapters and explicit tools rather than arbitrary code execution. Verification is bound to the exact proposal and is invalidated when evidence or the plan changes. SQLite transactions enforce one local ticket per approved run.
Where the happy path ends.
Demo scenarios use fictional accounts, policies, and incidents. Scripted mode is deterministic; live mode uses model-selected tool calls. Mechanical evidence checks do not establish semantic correctness or prompt-injection resistance. The local server has no production authentication or tenant isolation, and approval creates a local ticket rather than a refund or an external support action.
What exists today.
- Implemented a plan–observe–revise–verify workflow with policy retrieval, account/service tools, review decisions, and inspectable execution traces.
- The documented local suite records 42 passing tests and 10/10 scripted evaluation cases. These are regression results, not live-model accuracy.
- A four-worker deterministic check completed 50/50 jobs with saved traces and no unapproved tickets; it excludes LLM and network latency.
- Persistent queue state, paginated history, reload recovery, retries, and transactional approval make interrupted investigations inspectable.
What I would improve.
Evaluate live tool selection, answer groundedness, and abstention on held-out cases; add authentication and scoped real integrations before considering a public service.