Route the Question Before You Retrieve: Cypher, BM25 and Vectors

Illustrative scenario: a fictional composite written to make the point concrete. It does not describe a real organisation, and its figures are not reported outcomes.

Three people ask an internal assistant three questions in the same morning.

Which services depend on the authentication API? That is a question about relationships, and the answer is a list that is either complete or wrong. What does error E-4471 mean? That is a question about an exact token that appears in a handful of documents and nowhere else. Why do customers keep failing to sign in? That is a question about meaning, and the relevant documents may never use the words “sign in”.

One retriever does not serve all three well. The enterprise answer is not a bigger model on top; it is a small decision made before retrieval, followed by a fusion step, with the entitlement filter inside the query rather than after it.

Three retrievers, three jobs

Pure vector search became the default because it demonstrates well. Its weakness is the exact-match case, which is also the case a support engineer cares about most.

BM25 in two parameters

BM25 scores a document for a query by combining how often each query term appears in it, how rare the term is across the collection, and how long the document is. Two parameters shape the behaviour, and both are commonly left at defaults.

• k1 controls how quickly extra occurrences of a term stop adding score. A low value means the second and third mention matter little; a high value lets repetition keep counting.
• b controls length normalisation, from none (0) to full (1). Raise it when document lengths vary wildly and long documents are winning by volume; lower it for short fields such as titles, where length says little.

Commonly cited defaults are around 1.2 for k1 and 0.75 for b; check what your engine actually ships with. Treat them as a starting point. Tune them against queries you have labelled from your own collection, never by feel, and change one at a time. For a long-form reference, see Robertson and Zaragoza’s survey in the sources.

Fusing the results: Reciprocal Rank Fusion

When two or three retrievers each return a ranked list, the lists are on incompatible scales. A BM25 score and a cosine similarity cannot be added. Reciprocal Rank Fusion avoids the problem by ignoring scores and using only rank:

score(d) = sum over retrievers of 1 / (k + rank(d))

A document that appears near the top of several lists beats one that tops a single list. The constant k damps the influence of the very top positions; the original paper used 60. It needs no training and no score calibration, which is why it is the usual first choice. Its limitation is the mirror image of its strength: it cannot tell you that one retriever deserves more trust for this particular question. That is the router’s job.

Put context in the chunk before you index it

A chunk lifted out of a document often loses what made it meaningful (“revenue grew 3 per cent over the previous quarter”, of which company, and which quarter?). Anthropic’s Contextual Retrieval prepends a short, model-written description of where the chunk sits in its document before indexing, for both the embeddings and the BM25 index.

On their evaluation, which averaged four domains (codebases, fiction, ArXiv papers and science papers) and measured the top-20 chunk failure rate, the baseline was 5.7 per cent. Contextual embeddings alone brought it to 3.7 per cent, a 35 per cent reduction. Adding contextual BM25 brought it to 2.9 per cent, a 49 per cent reduction. Adding reranking as well reached 1.9 per cent, a 67 per cent reduction. The authors put the one-off cost of generating the context at about US$1.02 per million document tokens with prompt caching, under their stated assumptions about chunk and document size. They also advise testing on your own data, and so should you: these are one vendor’s results on one set of corpora.

A router, in code

The router should be boring. Rules first, because they are cheap, fast and auditable; a classifier only for what the rules cannot place. The sketch below shows the part that is easy to get wrong: the graph route runs fixed, parameterised queries, not Cypher written by a model.

import re

ERROR_CODE = re.compile(r"\b[A-Z]{1,4}-\d{3,6}\b")          # E-4471, ORD-10293
DEPENDS = re.compile(r"\b(depend(s|ed)? on|owned by|approved by)\b", re.I)

# Fixed, parameterised, read-only. A model never writes these strings.
GRAPH_QUERIES = {
    "dependents": (
        "MATCH (s:Service)-[:DEPENDS_ON]->(:Service {name: $name}) "
        "RETURN s.name AS service ORDER BY service LIMIT $limit"
    ),
}
FULLTEXT = (
    "CALL db.index.fulltext.queryNodes($index, $q) "
    "YIELD node, score RETURN node.id AS id, score LIMIT $k"
)

def route(question: str, entities: dict) -> list[str]:
    """Return the retrievers to run, in order of trust for this question."""
    routes = []
    if DEPENDS.search(question) and "service" in entities:
        routes.append("graph:dependents")
    if ERROR_CODE.search(question):
        routes.append("bm25")           # exact tokens: keyword first
    routes.append("vector")             # always available as the fallback
    return routes

Three notes on it. The graph route only fires when an entity was recognised and a fixed query exists for the intent; otherwise it is simply not used, and the question falls through to the other retrievers. The database account behind it should be read-only, so a mistake in routing cannot become a write. And the FULLTEXT call follows the form in Neo4j’s manual, whose examples use literal values; confirm parameter binding on the version you run. That manual describes a full-text index as Apache Lucene under the hood and describes the score as how well the entry matches the query; it does not name BM25, so confirm the scoring behaviour on the version you run before you tune k1 or b against it.

I have not run this sketch against a live database. Treat it as a shape to adapt, not a component to deploy.

Enforce entitlements inside the query

The most expensive retrieval mistake is not a poor ranking. It is retrieving across everything and removing what the caller may not see afterwards.

Suppose you ask for the ten nearest chunks and then drop those the caller is not entitled to read. If most of the closest chunks belong to someone else’s documents, you are left with two, or none, and the model answers confidently from what remains. The answer is incomplete in a way nothing reports. Applying the entitlement filter as part of the search, so the ten you get are ten the caller may see, is slower to build and the right default. The same rule holds for every retriever in the router: the graph query takes the caller’s scope as a parameter, the keyword index is partitioned or filtered, and the vector search filters before it ranks.

What to measure

• Recall at k, per route. If the BM25 route never finds the error-code documents, the router is sending the wrong questions there or the index is wrong.
• Router accuracy on a labelled set of real questions, including the ones that should fall through.
• Empty and thin answers after filtering. A rise means you are filtering after retrieval.
• Cost per successful answer, not per call. Routing saves money only if the cheaper route still resolves the question. A cheaper retriever that fails twice is the expensive one.

Limits

The retrieval figures above come from Anthropic’s post and the fusion constant from Cormack’s paper; the BM25 defaults are commonly cited values, not figures measured here. The Contextual Retrieval numbers are one vendor’s results on its own four corpora. The router is a sketch, not a tested component. Larger cost-reduction claims circulate in secondary articles; I could not trace them to a primary source, so they are not repeated here. If you have measured your own, publish them with the method.

Sources and further reading

The sources listed above are the basis for the figures and behaviours described in this article, with the BM25 defaults as noted under Limits.