ConceptGraph AI
Course-aware GraphRAG that turns PDFs into searchable concept graphs, reranks evidence, and returns page-grounded answers.
- 22
- Questions · retrieval ablation
- 12/17
- Final source coverage · all modes
- 5/5
- Unsupported refusals · ablation
What I was trying to solve.
Flat retrieval loses prerequisite relationships and course structure. ConceptGraph explores how semantic search and explicit concept relationships can work together without sacrificing provenance.
Follow the data.
A simplified processing flow. Each numbered stage feeds the next; storage and retrieval boundaries stay explicit.
- PDF extraction / bounded OCR
- Page-aware chunks → PostgreSQL
- Embeddings → Qdrant
- Concept extraction → Neo4j
- READY-gated vector candidates
- Graph relationship expansion
- Reranking
- Grounded answer
The constraints shaped the system.
Immutable course UUIDs align records across stores. PostgreSQL-backed leases and attempt fencing make a bounded, in-process worker restart-safe without adding a separate queue service. READY gating prevents partially ingested documents from entering retrieval; page and chunk provenance survive expansion and reranking.
Where the happy path ends.
An accepted upload is not a usable document. Partial graph extraction, provider rate limits, interrupted jobs, and cross-store deletion all need explicit states. Resumable section checkpoints, provider failover, attempt fencing, and upload-scoped cleanup make these failure paths inspectable.
Measured, with the caveats attached.
The provider-generated retrieval ablation uses 22 labelled questions across three authored PDFs: 17 supported and five unsupported. It compares vector-only, one-hop, and two-hop retrieval over 66 requests. Answer generation is excluded.
| Metric | Vector only | One hop | Two hop |
|---|---|---|---|
| All sources in raw top five | 16/17 | 17/17 | 17/17 |
| All sources after evidence gate | 12/17 | 12/17 | 12/17 |
| Unsupported refused | 5/5 | 5/5 | 5/5 |
| Mean retrieval | 0.06s | 0.48s | 0.41s |
Graph expansion improved one raw source-coverage case while adding latency. It did not improve the final evidence-gated result. Two-hop terms were used in only three questions; this small fixture does not establish general graph superiority or answer faithfulness.
A subsequent offline evidence-selection check across 10, 50, and 100 authored PDFs improved completeness from 29/35 to 31/35 while retaining 10/10 unsupported refusals. A harder challenge set still exposed missing evidence. These are retrieval checks, not live answer-quality scores.
What exists today.
- A 15-question baseline recovered the expected source page in the top five for 8/10 supported questions and refused 5/5 unsupported questions.
- A 22-question retrieval-only ablation recovered all required sources in raw top-five results for 16/17 supported questions with vector search and 17/17 with graph expansion. Final evidence-gated coverage remained 12/17 in every mode.
- An offline evidence-selection check across 10, 50, and 100 authored PDFs improved source completeness from 29/35 to 31/35 while retaining 10/10 unsupported refusals. A separate harder set exposed remaining misses; these are not generated-answer accuracy scores.
- A shared, read-only reviewer demo exposes pre-uploaded course material. Reviewer authorization gates uploads and paid AI operations; this is not a multi-tenant SaaS.
What I would improve.
Expand graph coverage and the evaluation set, measure repeated-run variance, and improve multi-source coverage after evidence filtering. Retrieval hits alone do not establish answer faithfulness.