Hybrid Search Relevance — a Per-Query Analysis

The search-lab results table says hybrid retrieval wins on BEIR SciFact: 0.6896 nDCG@10 for BM25 + dense fused with reciprocal-rank fusion, against 0.6537 for BM25 alone. But an aggregate can hide as much as it shows. This notebook opens the per-query evidence behind each headline claim: which queries each retrieval arm wins, where fusion rescues, and why reranking helps a weak first stage but hurts a strong one.

Everything here reads from committed run artifacts in runs/ — per-query rankings, metrics, and latencies written by search-lab eval --save-run. No Elasticsearch or Postgres needed to rerun this notebook:

uv sync --extra datasets --extra notebook

(The datasets extra is used only to show query/document text for examples; the metrics come entirely from the artifacts.) To regenerate the artifacts themselves from live backends, see docs/runbook.md.

The setup: BEIR SciFact — 5,183 scientific abstracts, 300 test queries, expert relevance judgments. Two retrieval arms (Elasticsearch BM25; MiniLM sentence embeddings under HNSW), RRF fusion, an optional cross-encoder second stage, and the same harness pointed at Postgres as a comparison backend. Metrics via ir_measures, so numbers are comparable to published baselines.

import json
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd

RUNS_DIR = Path("../runs/beir-scifact-test")
runs = {p.stem: json.loads(p.read_text()) for p in sorted(RUNS_DIR.glob("*.json"))}

ORDER = [
    "elastic-lexical",
    "elastic-lexical-rerank",
    "elastic-semantic",
    "elastic-hybrid",
    "elastic-hybrid-rerank",
    "postgres-lexical",
    "postgres-semantic",
    "postgres-hybrid",
]
assert sorted(runs) == sorted(ORDER), sorted(runs)
runs["elastic-lexical"]["meta"]

{'dataset': 'beir/scifact/test',
 'backend': 'elastic',
 'mode': 'lexical',
 'rerank': False,
 'k': 100,
 'embedding_model': None,
 'rerank_model': None,
 'timestamp': '2026-07-07T03:50:09+00:00'}

1. The scoreboard, recomputed from the artifacts

First, prove the artifacts carry the same numbers as RESULTS.md. Two things to notice before going per-query: BM25 at 0.6537 matches the published BEIR baseline (which validates the whole harness), and the two semantic rows — Elasticsearch and Postgres — are identical to four decimals. Both facts get unpacked below.

agg = pd.DataFrame({name: runs[name]["aggregate"] for name in ORDER}).T
agg.round(4)

	nDCG@10	R@100	RR	p50_ms	p95_ms
elastic-lexical	0.6537	0.8846	0.6236	21.9442	50.9763
elastic-lexical-rerank	0.6743	0.8846	0.6539	2086.5960	3733.0052
elastic-semantic	0.6484	0.9250	0.6123	32.6490	71.5942
elastic-hybrid	0.6896	0.9543	0.6612	53.6217	89.4236
elastic-hybrid-rerank	0.6862	0.9543	0.6611	2890.9765	6291.0728
postgres-lexical	0.2975	0.7459	0.2603	21.1724	57.6173
postgres-semantic	0.6484	0.9250	0.6123	29.4284	63.7381
postgres-hybrid	0.5763	0.9260	0.5487	58.0743	117.9013

Every aggregate above is the mean of 300 per-query values. The rest of this notebook works with those distributions directly — a 300 × 8 frame of per-query nDCG@10, one column per backend/mode.

def metric_series(name: str, metric: str = "nDCG@10") -> pd.Series:
    return pd.Series(
        {q["query_id"]: q["metrics"][metric] for q in runs[name]["queries"]}, name=name
    )


ndcg = pd.DataFrame({name: metric_series(name) for name in ORDER})
ndcg.describe().loc[["mean", "std", "50%"]].round(4)

	elastic-lexical	elastic-lexical-rerank	elastic-semantic	elastic-hybrid	elastic-hybrid-rerank	postgres-lexical	postgres-semantic	postgres-hybrid
mean	0.6537	0.6743	0.6484	0.6896	0.6862	0.2975	0.6484	0.5763
std	0.3991	0.4021	0.3969	0.3866	0.3900	0.3635	0.3969	0.4134
50%	1.0000	1.0000	0.7816	1.0000	1.0000	0.0000	0.7816	0.6309

2. Lexical and semantic retrieval fail on different queries

This is the fact that makes hybrid search work at all. If BM25 and the dense arm succeeded and failed on the same queries, fusing them would be pointless — you'd average two copies of the same signal. The scatter below shows they don't: the mass off the diagonal is queries where one arm succeeds and the other whiffs entirely.

lex = ndcg["elastic-lexical"]
sem = ndcg["elastic-semantic"]

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(lex, sem, alpha=0.3, s=18)
ax.plot([0, 1], [0, 1], lw=1, ls="--", color="gray")
ax.set_xlabel("BM25 nDCG@10")
ax.set_ylabel("dense nDCG@10")
ax.set_title("Per-query nDCG@10 — one dot per query")
plt.show()

delta = lex - sem
print(f"BM25 wins by > 0.2 on {(delta > 0.2).sum()} queries")
print(f"dense wins by > 0.2 on {(delta < -0.2).sum()} queries")
print(f"roughly tied (|diff| <= 0.2) on {(delta.abs() <= 0.2).sum()} queries")

BM25 wins by > 0.2 on 61 queries
dense wins by > 0.2 on 59 queries
roughly tied (|diff| <= 0.2) on 180 queries

What do those off-diagonal queries look like? Below, the two most extreme in each direction, with each arm's top-3 retrieved documents (* = judged relevant in the qrels). The pattern is the textbook one: the dense arm rescues queries phrased differently from the relevant abstract (paraphrase, no shared vocabulary), while BM25 wins when the query hinges on exact rare terminology that a small general-purpose encoder blurs away.

from search_lab import datasets

ds = datasets.load("beir/scifact/test")
texts = {q["query_id"]: q["text"] for q in runs["elastic-lexical"]["queries"]}


def query_row(name: str, qid: str) -> dict:
    return next(q for q in runs[name]["queries"] if q["query_id"] == qid)


def show(qid: str, arms: tuple[str, str] = ("elastic-lexical", "elastic-semantic")) -> None:
    print(f"\nQ{qid}: {texts[qid]}")
    relevant = {d for d, r in ds.qrels.get(qid, {}).items() if r > 0}
    for name in arms:
        q = query_row(name, qid)
        print(f"  {name:<20} nDCG@10 = {q['metrics']['nDCG@10']:.2f}")
        for doc_id, _score in q["results"][:3]:
            mark = "*" if doc_id in relevant else " "
            title = ds.corpus[doc_id].get("title", "")[:72]
            print(f"    {mark} {doc_id:>10}  {title}")


print("--- queries the dense arm rescues ---")
for qid in delta.sort_values().index[:2]:
    show(qid)

print("\n--- queries only BM25 gets ---")
for qid in delta.sort_values().index[-2:]:
    show(qid)

--- queries the dense arm rescues ---

Q1382: aPKCz causes tumour enhancement by affecting glutamine metabolism.
  elastic-lexical      nDCG@10 = 0.00
         3831884  Glutamine supports pancreatic cancer growth through a Kras-regulated met
         4138659  Macropinocytosis of protein is an amino acid supply route in Ras-transfo
         4423401  Succinate is an inflammatory signal that induces IL-1β through HIF-1α
  elastic-semantic     nDCG@10 = 1.00
    *   17755060  Control of Nutrient Stress-Induced Metabolic Reprogramming by PKCζ in Tu
         3831884  Glutamine supports pancreatic cancer growth through a Kras-regulated met
        22914228  Proteomic analysis reveals Warburg effect and anomalous metabolism of gl

Q1279: The treatment of cancer patients with co-IR blockade precipitates adverse autoimmune events.
  elastic-lexical      nDCG@10 = 0.00
        11254040  Adverse Events Associated With the Treatment of Multidrug-Resistant Tube
         3825750  Aliskiren combined with losartan in type 2 diabetes and nephropathy.
         1454773  In vitro characterization of the anti-PD-1 antibody nivolumab, BMS-93655
  elastic-semantic     nDCG@10 = 1.00
    *   11335781  Is autoimmunity the Achilles' heel of cancer immunotherapy?
        22635278  Immunotherapy of metastatic kidney cancer.
        10162553  Ablation and regeneration of tolerance-inducing medullary thymic epithel

--- queries only BM25 gets ---

Q1024: Recurrent mutations occur frequently within CTCF anchor sites adjacent to oncogenes.
  elastic-lexical      nDCG@10 = 1.00
    *    5373138  3D Chromosome Regulatory Landscape of Human Pluripotent Cells.
        24995939  Specific sites in the C terminus of CTCF interact with the SA2 subunit o
         8494570  Characterization of constitutive CTCF/cohesin loci: a possible role in e
  elastic-semantic     nDCG@10 = 0.00
         8494570  Characterization of constitutive CTCF/cohesin loci: a possible role in e
        36713289  Genome architecture, rearrangements and genomic disorders.
         4926049  TRF2 Recruits RTEL1 to Telomeres in S Phase to Promote T-Loop Unwinding

Q1363: Venules have a thinner or absent smooth layer compared to arterioles.
  elastic-lexical      nDCG@10 = 1.00
    *    8290953  Scaffold-based three-dimensional human fibroblast culture provides a str
         1122279  Endothelium-mediated relaxation of porcine collateral-dependent arteriol
        19140422  Evaluation of human papillomavirus testing in primary screening for cerv
  elastic-semantic     nDCG@10 = 0.00
        25175997  Pulmonary vascular lesions in end-stage idiopathic pulmonary fibrosis: H
         2593298  Vascular endothelial cadherin controls VEGFR-2 internalization and signa
           79447  Arteriolar function in visceral adipose tissue is impaired in human obes

[INFO] Opening /Users/e/.ir_datasets/beir/scifact/docs.pklz4/bin with direct file access

3. RRF fusion earns its win against a punishing baseline

Reciprocal-rank fusion sums 1/(60 + rank) across the two arms' ranked lists — no score normalization, no tuned weights. The fair per-query question is a hard one: compare the fused list not to an arm but to the better arm for that specific query — a per-query oracle you couldn't actually build, since you don't know in advance which arm will win. Against that oracle, hybrid matches or beats on ~77% of queries and loses only small amounts on the rest; and it comfortably beats either individual arm on aggregate. It also lifts recall — pulling relevant documents into the pool that BM25 alone never retrieved.

best_arm = ndcg[["elastic-lexical", "elastic-semantic"]].max(axis=1)
gain = ndcg["elastic-hybrid"] - best_arm

fig, ax = plt.subplots(figsize=(7, 3.5))
ax.hist(gain, bins=40)
ax.axvline(0, color="gray", lw=1)
ax.set_title("Hybrid minus best single arm, per query")
ax.set_xlabel("nDCG@10 difference")
plt.show()

print(f"hybrid >= best single arm on {(gain >= 0).sum()}/{len(gain)} queries")
print(f"strictly better on {(gain > 0).sum()}, strictly worse on {(gain < 0).sum()}")

r100_lex = metric_series("elastic-lexical", "R@100")
r100_hyb = metric_series("elastic-hybrid", "R@100")
rescued = ((r100_lex < 1) & (r100_hyb > r100_lex)).sum()
print(f"queries where hybrid raises Recall@100 over BM25 alone: {rescued}")

hybrid >= best single arm on 230/300 queries
strictly better on 24, strictly worse on 70
queries where hybrid raises Recall@100 over BM25 alone: 26

4. Reranking is conditional — it helps the weak and hurts the strong

The cross-encoder rescores the first stage's candidate pool, reading query and document together. The aggregate finding was the interesting one: +2.1 nDCG points over BM25, −0.3 over hybrid. The per-query delta distributions show why. Over BM25, the reranker fixes real ranking mistakes (a fat right tail). Over hybrid, the first stage has already used two independent signals to order the pool — the reranker mostly reshuffles an already-good ordering, and its occasional confident mistakes now have nothing to offset them.

The operational cost matters too: reranking 100 candidates per query with a cross-encoder on CPU dominates end-to-end latency (see the -rerank rows' p50 in the scoreboard). A second stage must earn that — per workload, by measurement.

pairs = [
    ("elastic-lexical", "elastic-lexical-rerank"),
    ("elastic-hybrid", "elastic-hybrid-rerank"),
]
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5), sharex=True, sharey=True)
for ax, (base, reranked) in zip(axes, pairs, strict=True):
    d = ndcg[reranked] - ndcg[base]
    ax.hist(d, bins=40)
    ax.axvline(0, color="gray", lw=1)
    ax.set_title(f"rerank over {base.split('-')[1]} (mean {d.mean():+.4f})")
    ax.set_xlabel("nDCG@10 difference")
plt.show()

5. Elasticsearch vs Postgres: the gap is BM25 quality, not speed

Same corpus, same MiniLM vectors, same harness — only the backend changes. Three per-query facts settle the comparison:

1. The dense arms are the same system. Identical L2-normalized vectors under HNSW return the same neighbors whether pgvector or Lucene stores them.
2. The lexical arms are not. Postgres full-text search (tsvector) ranks with ts_rank_cd, not BM25 — no document-length normalization, no term saturation — and it loses badly and almost uniformly.
3. Latency doesn't decide it. First-stage p50s are comparable across backends; you don't pick Elasticsearch here for speed. And the weak FTS arm actively poisons Postgres's hybrid (0.5763 vs 0.6484 for its own dense arm alone) — fusing in a bad signal is worse than not fusing at all.

dense_gap = (ndcg["elastic-semantic"] - ndcg["postgres-semantic"]).abs()
print(f"max per-query |nDCG@10 diff| between the two dense arms: {dense_gap.max():.6f}")

lex_gap = ndcg["elastic-lexical"] - ndcg["postgres-lexical"]
print(
    f"BM25 beats Postgres FTS on {(lex_gap > 0).sum()} queries; FTS wins on {(lex_gap < 0).sum()}"
)

latency = agg.loc[[n for n in ORDER if "rerank" not in n], ["p50_ms", "p95_ms"]]
latency.plot.bar(figsize=(8, 3.5), rot=30, title="Client-side latency (ms), first-stage modes")
plt.tight_layout()
plt.show()

max per-query |nDCG@10 diff| between the two dense arms: 0.000000
BM25 beats Postgres FTS on 185 queries; FTS wins on 16

6. Takeaways

• Default to hybrid RRF on Elasticsearch. The arms are complementary (section 2), fusion beats either arm alone and holds up against a per-query oracle (section 3), and the +3.6 nDCG / +7 recall points over BM25 come with a p50 still under 60 ms.
• Reach for Elasticsearch because of BM25, not speed. If Postgres is already on the stack, its dense arm is exactly as good — it's the lexical arm that isn't (section 5).
• Treat reranking as a per-workload experiment. It lifts weak first stages and taxes strong ones (section 4), at a latency cost of seconds per query on CPU; never assume the lift.
• Aggregates hide the mechanism. Every claim above was visible only per-query — which is the argument for saving run artifacts, not just metric rows.

What's next is queued in docs/experiments.md: a stronger encoder for the dense arm (e5/bge), a learned-sparse arm (SPLADE/ELSER) that keeps lexical grounding, real BM25 for Postgres via ParadeDB, and generalizing across more BEIR datasets. Each is a hypothesis; each gets a row in the table.