My research — retrieval-augmented AI systems

Systems that retrieve documents, passages, or knowledge at query time to answer questions, verify claims, and make decisions.

I have built many retrieval systems…

…and, I have broken just as many.

I'm known for Explainable IR

Foundational methods for understanding why retrieval and ranking models decide what they decide.

Algorithms I introduced are now standard tools in the field.

But I have a split personality

15+ years alongside explainability: scaling and efficiency of retrieval — inside and alongside industry.

Algorithms running in production at companies you've heard of.

What goes wrong in production RAG today

Three failure modes — all three show up in this single session.

I'll walk through each one with a concrete example.

A natural solution: query reformulations

Generate several rephrasings of the original query. Retrieve for each. Merge.

A natural solution: query reformulations

Generate several rephrasings of the original query. Retrieve for each. Merge.

Failure 2

Why not use a stronger model?

The obvious response to failures 1 and 2: use a stronger model.

A reasoning-model reranker reads the document, the query, the constraints — and judges relevance like a human would.

Why not use a stronger model?

The obvious response to failures 1 and 2: use a stronger model.

A reasoning-model reranker reads the document, the query, the constraints — and judges relevance like a human would.

It works. It's also prohibitively expensive.

Failure 3

The principle

Feedback from later stages can — and should — drive decisions in earlier stages.

The reranker's relevance estimates can re-prioritize the retriever's candidates.
The downstream generator's confidence can score upstream evidence.
The cost of judgments can budget the depth of search.

Use the re-ranking signal as feedback.

Re-ranking changes the order — and we throw the signal away

When the LLM gives many semantically different answers, the evidence is bad.
When it gives consistent answers, the evidence is good.
Uncertainty itself becomes the feedback signal.

Different queries need different signals

"Marriott London Bridge" — exact match. Lexical / BM25 wins.
Neighborhood signals just add noise.

"romantic weekend getaway near Amsterdam" — vague. Lexical fails.
Embedding similarity and document neighborhood matter.

"kid-friendly hotel with pool, walking distance to Sagrada Familia, under €200" — multi-constraint. No single signal is enough.

The standard pipeline picks one weighting of these signals at training time.

That weighting is wrong for most queries.

What if the system picked the weighting per query, online, from reranker feedback?

ORE — Online Relevance Estimation

QUAM and SUNAR pick the next batch using a fixed affinity structure.

ORE goes further: it learns online, per query, which signals matter for this query.

A linear stochastic bandit over the candidate pool. Each document is an arm; features are simple, well-known relevance factors; rewards are the expensive ranker's scores.

But signals aren't the only thing we can pick per query

ORE asks: given this query, which relevance signals should I trust?

Now ask the same question one level up.

ReformIR — picking reformulations per query

Failure 2: more reformulations → more drift.

Remember the Tokyo example? Four reformulations, one of them useful, three of them drifty.

The fix isn't fewer reformulations. It's picking which ones to trust, per query, online, from reranker feedback.

ReformIR — same idea as ORE, one level up

ORE puts relevance signals in the bandit's feature vector.

ReformIR puts reformulations in the bandit's feature vector.

The reranker always scores against the original query, anchoring the system against drift.

Drifty reformulations get downweighted. Useful ones get upweighted.

This is the paradigm shift

Stop fixing the system at training time.

Bandits give us the language for all of it

Same principle, different decisions, query by query:

• Which query formulation to issue → ReformIR
• Which documents to score → ORE, QUAM, SUNAR
• Which judgments to spend budget on → all of the above

Not "how do I train a better retriever" — but "how do I deploy compute optimally, query by query?"

The science: subset selection under uncertainty

Every algorithm I just showed reduces to the same question:

Given a large pool and a tiny budget, which items deserve the expensive call?

This is top-m arms identification in stochastic linear bandits. Specifically disposable bandits.

The bottleneck: the candidate pool is exponentially large. Classical bandit algorithms compare every arm against every other — infeasible at retrieval scale.

EXPLORA — Choosing Explanations

You have a pool of N explanations. In-context learning needs a k-subset — the right k demonstrations decide whether the LLM reasons correctly or not.

The brute-force search: evaluate all subsets with the LLM.
At N = 100, k = 4, that's 3.9 million LLM calls.

Again EXPLORA frames this as bandit arm selection.

Each k-subset is an arm. An LLM call is a pull. The arms are exponentially many — classical bandit algorithms that compare every pair are infeasible.

CASE — what EXPLORA leaves open

EXPLORA explores efficiently but never answers a fundamental question:

Without confidence information, EXPLORA can't eliminate arms — it keeps spending budget on subsets that are clearly suboptimal.

Explain-and-predict isn't always perfect

The classical recipe — extract a rationale, then predict from it — is faithful by construction. But faithfulness is not free.

Recent work shows that the joint optimization can succeed without leaking the label or the feature itself into the rationale — and that the resulting predictors are competitive with their non-interpretable counterparts.

RAG faithfulness isn't perfect either

LLMs can produce a fluent answer, cite the right passage, and still be unfaithful — the citation explains the answer post hoc rather than supporting how the answer was actually derived.

We call this post-rationalization: the model decides, then dresses the decision in a plausible attribution.

A retrieval system that:

• generates its own data it needs to improve on
• evaluates itself in a way that aligns with deployment
• optimizes itself, query by query, over time

Personas — generating behavior, not labels

Most synthetic-data work generates labels:
"Is document D relevant to query Q? Yes / No."

That's not how users behave.

We generate personas:

• the budget traveler vs the luxury traveler
• the planner vs the last-minute booker
• the comparison shopper vs the decisive buyer

Each persona issues queries, reformulates, clicks, abandons —
the way users actually do.

The system creates the data it needs to improve.

Static benchmarks → continuous self-evaluation.

We've closed the recall gap

The bandit framing — across QUAM, SUNAR, ORE, ReformIR — lets us recover the documents that classical cascades were leaving on the floor.

The gap we closed → the world we're opening

The gap we closed

Bounded-recall → feedback-aware retrieval

The world we're opening

AutoIR — systems that learn on their own

I work with a small number of companies a year on exactly this.
If any of is interesting for you — let's talk after.

Thats it !!!