My research — retrieval-augmented AI systems

Systems that retrieve documents, passages, or knowledge at query time to answer questions, verify claims, and make decisions.

I'm known for Explainable IR

Foundational methods for understanding why retrieval and ranking models decide what they decide.

Algorithms I introduced are now standard tools in the field.

But I have a split personality

15+ years alongside explainability: scaling and efficiency of retrieval — inside and alongside industry.

Algorithms running in production at companies you've heard of.

What goes wrong in production RAG today

Three failure modes — all three show up in this single session.

I'll walk through each one with a concrete example.

Failure 2 — The drift problem

Modern systems try to fix recall with query reformulations.

Generate several rephrasings of the original query. Retrieve for each. Merge.

The intuition: more queries, more recall.

The reality: most reformulations drift.

Reformulating "good hotels in Tokyo"

Concatenate or rank-fuse all of them, and noise dominates signal.

Failure 3 — The cost wall

The obvious response to failures 1 and 2: use a stronger model.

A reasoning-model reranker reads the document, the query, the constraints — and judges relevance like a human would.

It works. It's also prohibitively expensive.

So you're back to the recall ceiling — gated by whatever the cheap retriever surfaces.

What these failures share

All three come from the same root cause:

The system makes one-shot decisions.

The retriever decides once. Reformulation happens once. The reranker reads what it's given.

No stage's signal is fed back to revise the stages before it.

The principle

Feedback from later stages can — and should — drive decisions in earlier stages.

The reranker's relevance estimates can re-prioritize the retriever's candidates.
The downstream generator's confidence can score upstream evidence.
The cost of judgments can budget the depth of search.

The reranker is a better estimator of relevance

Use the re-ranking signal as feedback.

When the LLM gives many semantically different answers, the evidence is bad.
When it gives consistent answers, the evidence is good.
Uncertainty itself becomes the feedback signal.

Different queries need different signals

"Marriott London Bridge" — exact match. Lexical / BM25 wins.
Neighborhood signals just add noise.

"romantic weekend getaway near Amsterdam" — vague. Lexical fails.
Embedding similarity and document neighborhood matter.

"kid-friendly hotel with pool, walking distance to Sagrada Familia, under €200" — multi-constraint. No single signal is enough.

The standard pipeline picks one weighting of these signals at training time.

That weighting is wrong for most queries.

What if the system picked the weighting per query, online, from reranker feedback?

Algorithm 3 — ORE — Online Relevance Estimation

QUAM and SUNAR pick the next batch using a fixed affinity structure.

ORE goes further: it learns online, per query, which signals matter for this query.

A linear stochastic bandit over the candidate pool. Each document is an arm; features are simple, well-known relevance factors; rewards are the expensive ranker's scores.

But signals aren't the only thing we can pick per query

ORE asks: given this query, which relevance signals should I trust?

Now ask the same question one level up.

ReformIR — picking reformulations per query

Failure 2: more reformulations → more drift.

Remember the Tokyo example? Four reformulations, one of them useful, three of them drifty.

The fix isn't fewer reformulations. It's picking which ones to trust, per query, online, from reranker feedback.

ReformIR — same idea as ORE, one level up

ORE puts relevance signals in the bandit's feature vector.

ReformIR puts reformulations in the bandit's feature vector.

The reranker always scores against the original query, anchoring the system against drift.

Drifty reformulations get downweighted. Useful ones get upweighted.

This is the paradigm shift

Stop fixing the system at training time.

Bandits give us the language for all of it

Same principle, different decisions, query by query:

• Which query formulation to issue → ReformIR
• Which documents to score → ORE, QUAM, SUNAR
• Which judgments to spend budget on → all of the above

Not "how do I train a better retriever" — but "how do I deploy compute optimally, query by query?"

The science: subset selection under uncertainty

Every algorithm I just showed reduces to the same question:

Given a large pool and a tiny budget, which items deserve the expensive call?

This is top-m arms identification in stochastic linear bandits.

The bottleneck: the candidate pool is exponentially large. Classical bandit algorithms compare every arm against every other — infeasible at retrieval scale.

Ways in which you can use explanations

EXPLORA — Choosing Explanations

You have a pool of N explanations. In-context learning needs a k-subset — the right k demonstrations decide whether the LLM reasons correctly or not.

The brute-force search: evaluate all subsets with the LLM.
At N = 100, k = 4, that's 3.9 million LLM calls.

Again EXPLORA frames this as bandit arm selection.

Each k-subset is an arm. An LLM call is a pull. The arms are exponentially many — classical bandit algorithms that compare every pair are infeasible.

CASE — what EXPLORA leaves open

EXPLORA explores efficiently but never answers a fundamental question:

Without confidence information, EXPLORA can't eliminate arms — it keeps spending budget on subsets that are clearly suboptimal.

Explain-and-predict isn't always perfect

The classical recipe — extract a rationale, then predict from it — is faithful by construction. But faithfulness is not free.

Recent work shows that the joint optimization can succeed without leaking the label or the feature itself into the rationale — and that the resulting predictors are competitive with their non-interpretable counterparts.

RAG faithfulness isn't perfect either

LLMs can produce a fluent answer, cite the right passage, and still be unfaithful — the citation explains the answer post hoc rather than supporting how the answer was actually derived.

We call this post-rationalization: the model decides, then dresses the decision in a plausible attribution.

A retrieval system that:

• generates its own data it needs to improve on
• evaluates itself in a way that aligns with deployment
• optimizes itself, query by query, over time

Personas — generating behavior, not labels

Most synthetic-data work generates labels:
"Is document D relevant to query Q? Yes / No."

That's not how users behave.

We generate personas:

• the budget traveler vs the luxury traveler
• the planner vs the last-minute booker
• the comparison shopper vs the decisive buyer

Each persona issues queries, reformulates, clicks, abandons —
the way users actually do.

The system creates the data it needs to improve.

Static benchmarks → continuous self-evaluation.

I work with a small number of companies a year on exactly this.
If any of is interesting for you — let's talk after.

Thats it !!!

First-generation RAG — where it breaks

These systems work. But they break in systematic, reproducible ways.

I've seen the same three failures across search engines, fact-checkers, and recommendation systems.

CASE — what EXPLORA leaves open

EXPLORA explores efficiently but never answers a fundamental question:

Without confidence information, EXPLORA can't eliminate arms — it keeps spending budget on subsets that are clearly suboptimal.

CASE (ICML 2025) adds gap-index theory:

• Tracks a shortlist of challenger arms — subsets still statistically competitive with the current best
• Arms with a large performance gap below the leader are eliminated early
• The gap-index bounds the expected number of pulls needed to certify the best arm
• Algorithm terminates with a provable optimality guarantee

Same idea as EXPLORA, with convergence certificates. Fewer queries. Broader tasks.

EXPLORA — learning to score subsets

A scoring function σ(α, S) predicts the quality of subset S.
α is unknown — the parameters we learn from LLM feedback.

The loop:

1. Sample a batch of candidate subsets from the pool
2. Query the LLM on a handful — observe accuracy as reward
3. Fit α by minimizing prediction loss over observed rewards
4. Use σ(α, ·) to guide which subsets to evaluate next
5. Repeat until budget exhausted

No confidence bounds. No elimination. Explores by loss minimization alone.

Results on AquaRat · FinQA · GSM8K · TabMwp:
Uses ~11% of the LLM calls required by prior SOTA · +12.24% accuracy

Algorithm 1 — QUAM

Failure 1: the recall ceiling — the relevant doc never made it into the top-1000.

QUAM asks the reranker to teach the retriever:

1. Rerank a small batch
2. For high-scoring docs, look up their learned-affinity neighbors
3. Score neighbors by set affinity — proximity to the set of already-relevant docs
4. Send the top neighbors to the reranker next
5. Repeat until budget exhausted

The learned affinity graph encodes co-relevance, not just textual similarity.