ap / engineering archive
← All field notes

Why vector retrieval alone was insufficient

Semantic similarity finds nearby language. It does not automatically preserve prerequisites, hierarchy, or evidence chains.

Edited · 1 min read · Engineering / systems

A useful learning system has to recover more than a similar paragraph. It needs to understand where a concept sits, what it depends on, and which source supports the answer.

That changed the retrieval design: vector search became a candidate generator, while graph expansion, reranking, and citation checks decided what could reach the final context. The graph is a hypothesis about useful structure—not proof that retrieval improves.

The September 2026 provider-graph experiment is a useful reality check. All three retrieval modes returned the primary expected page for 15 of 17 supported questions. Provider quota limited extraction to eight nodes and seven edges in one document, with no two-hop expansion terms used. Those equal scores cannot establish an advantage for graph traversal. Better coverage and controlled evaluation come before a stronger claim.

See the evaluation and its limitations ↗