Keyword search missed a fact my OpenClaw agents already knew
My OpenClaw agents couldn't find a fact that was already in their memory files. I asked who handled technical writing; the keyword search missed the line saying Quill owned the journal pipeline.
The system I'd built used keyword search, a grep-style backend called QMD that indexed markdown files by term frequency and returned the top-5 results. It worked. But it had a consistent blind spot: queries where the phrasing didn't match the exact words in the file. Ask "who handles technical writing?" and you'd get nothing back, even though "Quill owns the journal pipeline" was sitting in AGENTS.md. The words didn't match. The context was invisible.
That sent me into a March 2026 comparison of 7 retrieval backends. The vector conditions used the same Voyage voyage-3-lite embeddings, with QMD keyword search as the comparison. Chroma failed during setup; it never reached the query benchmark.
The one that didn't make it past setup
Chroma was first on the list. Within 20 minutes, an import failed in my Python 3.14.3 environment: AttributeError: module 'pydantic' has no attribute 'BaseSettings'. Chroma was out before it ran a single query. The failure identified a dependency compatibility problem in that environment; the error alone didn't establish which package version caused it.
The Qdrant breaking change I found the hard way
Qdrant is where I went deepest. The client version I was using no longer had client.search(); I needed client.query_points(). I spent 40 minutes assuming the problem was in my embedding pipeline before checking the changelog.
Once it was running: Qdrant's hit rate across 40 test queries was 97.5%, compared to 87.5% for the keyword backend. Qdrant almost never returned zero useful results; keyword search missed on 5 queries entirely.
But here's where it got complicated.
Higher hit rate, lower mean score
The benchmark used 489 chunks from 58 files, scored on a 0/0.5/1.0 rubric.
Qdrant hit rate: 97.5%. QMD hit rate: 87.5%. Qdrant wins on coverage.
Qdrant mean chunk score: 0.367. QMD mean chunk score: 0.445. Keyword wins on precision.
When keyword search hits, it lands on the exact source more often. On "model" and "lesson" queries, grep finds what you want. On "person" queries, grep fails completely. Qdrant gets those right every time: 100% hit rate on person queries vs 50% for keyword.
The practical conclusion: the two backends are not competing. They have genuinely different error modes.
Weaviate's sweet spot was not where I expected it
Weaviate has a hybrid search mode that blends dense vector retrieval (semantic) with BM25 keyword retrieval. The alpha parameter controls the blend: alpha=0.0 is pure BM25, alpha=1.0 is pure dense vector, alpha=0.5 is the default.
These Weaviate scores came from the alpha sweep reported in my notes; the available methodology does not establish that they are directly comparable with the Qdrant/QMD chunk scores above. The default wasn't the best setting for my test suite. At alpha=0.5, the average score was 0.9700. At alpha=0.1, with BM25 doing most of the work and semantic matching contributing a smaller signal, it was 0.9940: a 0.0240 difference on a 0–1 scale.
The BM25 component at low alpha provides the precision; the semantic component fills in what keyword matching misses. Tuning this parameter mattered more than I expected. Try alpha=0.1 before you accept 0.5 as the right setting for your corpus.
voyage_numpy: the baseline that held up
The brute-force cosine similarity baseline using Voyage embeddings directly scored between the two. Holding embeddings constant let me compare retrieval and tuning choices without changing the embedding model at the same time.
The combination that actually won
I ran a hybrid condition: Qdrant dense (top-10 results) interleaved with QMD keyword (top-10 results), deduplicated on content fingerprint, top-5 injected as context.
Hit rate: 97.5%, matching Qdrant's coverage. Mean score: 0.440, nearly matching QMD's precision. The hybrid eliminated most of both backends' individual failure modes.
Then I added an LLM reranker. After the hybrid merge, I ran the top-10 candidates through a scoring pass using gpt-4o-mini and selected the top-5 by judge score. On the 10 hardest queries, where the heuristic scorer and the LLM judge most disagreed, the reranked hybrid scored a mean of 0.516 by LLM judge. That beat pure Qdrant (0.383) and pure keyword (0.200) on the same query set.
There's a cost: one extra model call per memory query. For real-time agent retrieval, that adds latency. For async background queries, it's worth it. And there's a token budget bonus: dropping from top-5 to top-3 injection achieved 40.7% token reduction with only a 5 percentage point hit rate drop. That's a meaningful context window saving.
What I'd do differently
Check your Python version against the database's dependency tree before writing a single line of retrieval code. The Chroma dependency failure in my Python 3.14.3 environment cost me an hour.
Pin your Qdrant client version. The client.search() removal cost me 40 minutes diagnosing the wrong problem.
Run the Weaviate alpha sweep before deploying. In my test suite, alpha=0.1 scored better than 0.5; that is a reason to test the blend on your own corpus, not a universal setting.
For this corpus, Qdrant covered queries that keyword search missed, while QMD was more precise when the words matched. Combining them and reranking the candidates gave me the strongest reported result on the 10 hardest queries. That smaller reranking test and its extra model-call cost are the boundary of the result.