The consumer asks “what’s the efficient date of this coverage?”. Retrieval returns 5 candidate line-windows and passes all of them to the LLM in a single immediate. The LLM reads all 5 to extract the identical date the primary candidate already had. Chunks two to 5 have been signatures, footnotes, and a paragraph about historic dates. Paid for nothing. Ship the top-1 first, ask the LLM if that’s sufficient, and cease when it says sure: sequential feeding cuts the token price by 80 % on this class of query. The remainder of the article catalogues the place sequential wins, the place a single-call over all Okay is the precise default, and the way the query parser dispatches between the 2.
This text is a companion to Enterprise Doc Intelligence, the sequence whose philosophy is specified by Amplify the Knowledgeable, the sequence that builds enterprise RAG from 4 bricks (doc parsing, query parsing, retrieval, era). It sits between Article 8 (era) and Article 9 (upgrading the mini-RAG) and develops one particular resolution: how do you feed the top-Okay retrieved candidates into the era brick?. The reply most pipelines have is “all Okay directly”. This text catalogues the second regime (sequential, top-1 first) and exhibits when each wins.
The naive baseline this text pushes again on
Naive RAG ships batch by default. Retrieval returns top-5, the LLM will get all 5, the reply comes again. It really works, and on onerous questions (comparability, itemizing) it’s the proper alternative. However the silent price is paid on each different query: the straightforward factual ones the place the top-1 already had the reply. This text walks the second regime and the dispatch that picks per query.
📓 The runnable pocket book for this text is on GitHub: doc-intel/notebooks-vol1. It sends the identical questions via each batch and sequential era, prints the per-question token price and the 2 sufficiency booleans (answer_found, complete_answer_found) that cease the loop, and reproduces the dispatch desk by yourself machine.
1. Two regimes for feeding the top-Okay to era
Retrieval palms era an ordered top-Okay. There are two methods to feed it in, and so they price very in a different way.
1.1 The mounted top-Okay pipeline and what it wastes
The default sample throughout most RAG tutorials seems like this:
top_k = retrieval(query, ok=5)
reply = era(query, top_k)
It runs in two steps for each query. Token price is roughly generation_cost(query + 5 chunks of context). Latency is roughly retrieval + one LLM name. And the LLM fortunately processes the 5 chunks even when the top-1 was already adequate and chunks 2..5 added zero info.
Concretely: the consumer asks “what’s the efficient date of this coverage?”. Retrieval returns 5 line-windows the place the key phrase “efficient” seems. The primary one is the reply (“efficient from January 1, 2026”). Chunks 2..5 are signatures, footnotes, and a paragraph about historic efficient dates of previous insurance policies. The LLM reads all 5 to extract the identical date the primary one already had. On a corpus of 1 doc, the price is rounding error. On a corpus of 50k insurance policies, that is actual cash per thirty days.
1.2 Sequential: top-1 first, sufficiency predicate, escalate if wanted
The sequential regime treats the Okay candidates as an ordered listing and asks the era brick to validate sufficiency at every step:
for i, candidate in enumerate(top_k):
reply = era(query, [candidate])
if reply.answer_found and reply.complete_answer_found:
break
The sufficiency sign lives contained in the typed contract Article 8A launched. The AnswerWithEvidence schema exposes answer_found (was the query’s info current on this candidate?) and complete_answer_found (was all the reply current, not a fraction?). The loop reads these fields, not a customized heuristic.
Within the efficient date instance above: era runs as soon as on the top-1 candidate, the LLM returns answer_found = True, complete_answer_found = True, the loop exits. Tokens spent: generation_cost(query + 1 chunk of context). That’s 1/5 of the batch price for this query, on a pipeline that handles 1000’s of comparable lookups per day.
1.3 Batch: ship all Okay directly, let the LLM arbitrate
Batch mode retains the default behaviour: one LLM name sees all Okay candidates and produces one typed reply. Its case is constructed on three query varieties the place sequential breaks down:
Itemizing questions: “listing all exclusions on this contract”. The reply is each matching candidate, not the primary. Sequential would cease at top-1 (one exclusion discovered) and miss the opposite 4. Batch is the one right mode.
Comparability questions: “is the premium greater than the earlier yr’s?”. The reply requires each candidates (this yr + final yr) in the identical name so the LLM can examine them. Sequential would extract each independently and lose the be part of.
Tight-score retrieval: when the top-Okay candidates’ relevance scores are inside 5% of one another, retrieval can’t reliably promote top-1 to the highest. Batch lets the LLM arbitrate with the complete proof seen.
Price evaluation: batch all the time pays the generation_cost(query + Okay chunks of context) as soon as. Sequential pays generation_cost(query + 1 chunk of context) within the straightforward case (top-1 adequate) and generation_cost(query + 1 chunk) × Okay within the worst case (each candidate inadequate). On a typical enterprise corpus with Okay = 5, sequential is cheaper on common for factual lookups (~80% of typical visitors) and costlier for itemizing / comparability (~20%).
2. The dispatch resolution: per query, not per pipeline
The clear structure doesn’t decide batch or sequential globally. It picks per query, utilizing the parsed question_df row from brick 2. Query form, decomposition sample, and intent drive the selection:
The dispatcher reads question_df.answer_shape and question_df.decomposition and routes. Naive RAG has no option to make this distinction as a result of it has no parsed query to learn from.
The identical routing desk holds throughout sectors and professions. Totally different domains carry the identical form patterns and the identical sequential / batch resolution flows out of them:
In each row, the sequential column is a single typed worth (Quantity, Date, Boolean) and advantages from the top-1 cease. The batch column is an inventory or a comparability and desires all Okay candidates seen directly. The dispatcher’s desk covers all 5 sectors with the identical logic.
3. The sufficiency sign
Sequential mode leans on one factor: the era brick reporting whether or not the candidate it simply learn was sufficient. That sign, and the principles that cease the loop, reside right here.
3.1 The place the sufficiency sign lives within the typed contract
Sequential mode solely works if the era brick can self-report whether or not the candidate it simply noticed contained the reply. The typed contract from Article 8A is what makes this doable:
class AnswerWithEvidence(BaseModel):
worth: Any
proof: listing[Span]
answer_found: bool
complete_answer_found: bool
confidence: float = Subject(ge=0, le=1)
caveats: listing[str] = []
The sequential loop reads answer_found and complete_answer_found, not a confidence float or a customized heuristic. The clear separation between the 2 booleans (from Sample 4 of Article 8ter) is what makes the loop deterministic. Discovered + incomplete says “proceed to the subsequent candidate”; discovered + full says “cease”; not discovered says “proceed or quit at Okay”.
A confidence float would drive a threshold (e.g. “cease at 0.8”), and that threshold drifts mannequin to mannequin. Two booleans don’t drift.
3.2 Bounded iteration: even sequential has to cease
The sequential loop has three exits, not one:
Sufficiency: answer_found and complete_answer_found on a candidate. Cease, ship the reply.
Exhaustion: each one of many Okay candidates seen, none adequate. Cease, return answer_found = False (a first-class reply the validator passes via).
Price range: a token or time finances set by the dispatcher. Helpful when the corpus is giant and a runaway sequential loop would blow the cap. Similar form because the bounded iteration M4 (loop engineering) names.
Naive sequential implementations skip the third exit and burn tokens without end on edge instances. The sequence model units a finances upfront and the dispatcher logs it on each name.
4. Price, and the place the sequence stops
Two regimes, two price profiles, and one boundary the sequence doesn’t cross.
4.1 A concrete price comparability on a hundred-question batch
To make the trade-off actual, here’s a back-of-envelope for an enterprise insurance coverage Q&A workload:
100 questions / day, Okay = 5, common chunk measurement = 600 tokens.
Batch baseline: 100 × generation_cost(query + 5×600 tokens) = ~330k enter tokens per day for era.
Sequential (80% top-1 adequate): 80 × generation_cost(query + 600) + 20 × generation_cost(query + 5×600) (worst case for the 20% advanced questions) = ~115k enter tokens per day.
The ratio is 65% saving on enter tokens for era on this workload. The precise ratio will depend on the easy-question proportion and the chunk measurement; the precept is that sequential is a budget default as soon as the typed contract is in place. Batch is reserved for the query varieties that want it.
4.2 The agentic temptation, and the place the sequence stops
A pure subsequent step is to let the LLM resolve between batch and sequential per query (agentic dispatch). The sequence stops in need of this. The dispatcher in Part 2 is deterministic-dispatcher (one of many three approaches catalogued in Article 6C), not LLM-decides. The reason being the identical one which holds throughout the sequence: audit. The identical query on the identical day should route the identical approach; an LLM that re-plans the dispatch per name can’t give that assure.
In case you want a stronger agentic loop (the LLM picks which candidates to have a look at, in what order, with what scope), the larger article on adaptive RAG loops is the precise place. This text retains the scope to the deterministic sequential vs batch resolution pushed by the parsed query.
5. The choice belongs to the parser, not the LLM
The mounted top-Okay + batch pipeline is the precise default for the query varieties the place each candidate issues. It’s the flawed default for the factual lookups that make up most enterprise visitors. The sequence provides two items: the typed sufficiency sign (answer_found, complete_answer_found) from Article 8A, and the dispatch desk from Part 2. Collectively they let the era brick cease after the primary candidate when that’s sufficient, and course of all Okay when the query sort requires it. The choice is made by the parser, not by the LLM, which retains the audit path intact.
6. Additional studying and sources
The sequential / batch resolution sits on the boundary between retrieval and era. The articles that body either side:
