{"id":975,"date":"2026-06-13T17:00:00","date_gmt":"2026-06-13T17:00:00","guid":{"rendered":"https:\/\/futurenews24.com\/index.php\/2026\/06\/13\/larger-context-windows-dont-fix-rag-so-i-built-a-system-that-does\/"},"modified":"2026-06-14T08:59:34","modified_gmt":"2026-06-14T08:59:34","slug":"larger-context-windows-dont-fix-rag-so-i-built-a-system-that-does","status":"publish","type":"post","link":"https:\/\/futurenews24.com\/index.php\/2026\/06\/13\/larger-context-windows-dont-fix-rag-so-i-built-a-system-that-does\/","title":{"rendered":"Bigger Context Home windows Don\u2019t Repair RAG \u2014 So I Constructed a System That Does"},"content":{"rendered":"
\n<\/p>\n
\n
TL;DR<\/h2>\n
I constructed a dataset Q&A system and trusted a RAG reply that was lower than half-correct.<\/p>\n
I measured this throughout 7 question sorts and 5 context sizes on 100,000 rows.<\/p>\n
queries away from RAG totally.<\/p>\n
I Trusted the Incorrect Quantity<\/h2>\n
Final month I used to be heads-down constructing a brand new function for EmiTechLogic. Learners can now add their very own messy CSV information and ask questions in plain English about their knowledge. Sounded good for RAG, so I went all in \u2014 embeddings, retrieval, nice-looking responses.<\/p>\n
The primary few demos seemed superb. Clear tables, assured numbers, skilled formatting. I really began trusting the system in our inner testing.<\/p>\n
Then I picked one quantity to double-check.<\/p>\n
Actual grocery spend within the dataset: $1,140,033.24.\u00a0\u00a0<\/p>\n
The mannequin gave me a gorgeous breakdown by class. It seemed legit. I added up the numbers it returned.<\/p>\n
It was lower than half.<\/p>\n
I sat there staring on the display screen considering \u201cthis will\u2019t be proper.\u201d So I did what any engineer would do. I elevated the context window. 4k\u2026 16k\u2026 32k\u2026 128k tokens. Every time the reply bought longer, extra detailed, and extra confidently flawed.<\/p>\n
That\u2019s when it lastly clicked. This wasn\u2019t a retrieval concern. I used to be asking a retrieval system to carry out heavy computation on knowledge it had solely partially seen. And as an alternative of claiming it was uncertain or lacking data, the mannequin was producing polished, structured solutions that seemed right.<\/p>\n
Why RAG Can not Combination<\/h2>\n
The RAG pipeline doesn\u2019t really perceive structured knowledge. All it does is take every CSV row and flatten it into plain textual content. That\u2019s it. To the mannequin, a row appears to be like one thing like this:<\/p>\n
“2019-01-01 grocery_pos 107.23 F NC Jennifer Banks …”<\/p>\n
For a question like \u201cWhat’s the whole spend by class?\u201d, the RAG pipeline does this:<\/p>\n
1. Tokenise: [“total”, “spend”, “category”] \n2. Rating all 100,000 rows by key phrase overlap \n3. Return the top-N rows as serialised plain textual content \n4. Ask the LLM to sum and group from that textual content<\/p>\n
Step 4 is the place the system fails. The LLM isn’t operating a SUM. It’s pattern-matching numbers from a textual content blob and producing a response that mimics an aggregation.<\/p>\n
Fashions wrestle with numerical precision at scale [1], however the true concern is the presentation. The mannequin offers you an in depth breakdown throughout all classes. It is a basic entice. The output appears to be like skilled. It mimics the construction of an actual report so nicely that your mind assumes the content material is legitimate. You don’t have any solution to confirm that 92% of your knowledge is lacking.<\/p>\n
RAG is a retrieval instrument. It isn’t a calculation engine. Retrieval finds related fragments. Computation requires a full dataset scan. If you use RAG for math, you get a flawed reply that appears authoritative. That distinction is important. A partial reply indicators that knowledge is lacking. A whole-looking flawed reply simply indicators false confidence.<\/p>\n
\n
Full code: https:\/\/github.com\/Emmimal\/context-window-engine\/<\/p>\n<\/blockquote>\n
The Benchmark: Two Pipelines, Similar Question<\/h2>\n
To measure this exactly, I constructed a benchmark that runs two pipelines facet by facet for each question.<\/p>\n
The primary pipeline is a RAG simulation. It fashions what a naive vector pipeline passes to an LLM at 5 context sizes. I examined 5 context sizes, starting from 5 rows as much as 8,000. That scales from 325 tokens to 500,000. For every dimension, I tracked three metrics: how a lot knowledge the LLM sees, what sum it computes from that particular slice, and whether or not a reader may really spot the error.<\/p>\n
The second pipeline is a semantic engine that executes the identical question as a deterministic full-scan over all 100,000 rows and returns the precise right reply.<\/p>\nArchitectural comparability of question processing workflows, contrasting text-based RAG Simulation retrieval with structured knowledge aggregation in a Semantic Engine. Picture by Creator.<\/figcaption><\/figure>\n
The simulation doesn’t reproduce precise LLM outputs. What it preserves is the important thing structural property: a partial slice of information fed right into a system that returns a full-form reply. That’s the property that causes the issue, and that’s what the benchmark measures.<\/p>\n
I selected seven question sorts to cowl each aggregation sample a structured knowledge system is more likely to encounter:<\/p>\nQueryOperationWhy it breaks RAGTotal spend by categorySUM + GROUP BYRequires summing all rows throughout 14 groupsHighest common transaction by categoryAVG + GROUP BYAverage modifications with each lacking rowTotal spent on grocery_posSUM + categorical filterFilter requires seeing all matching rowsHow many feminine clients transactedCOUNT + filterCount is meaningless on a partial scanTotal spend the place quantity > $500SUM + numeric comparisonThreshold logic requires full dataState with lowest whole spendingMIN + GROUP BY throughout 50 groupsMinimum can solely be discovered with all teams presentPercentage of transactions which can be fraudulentCOUNT + ratioRatio is undefined on a partial denominator<\/figure>\n
These queries usually are not distinctive or advanced. They’re the usual questions any analyst asks when taking a look at a brand new dataset. That’s precisely why this failure is so important.<\/p>\n
Error Observability Collapse<\/h2>\n
Right here is the total benchmark output for the question that began all of this. I’m exhibiting it in full as a result of the numbers make the issue not possible to dismiss.<\/p>\n
GROUND TRUTH (Semantic Engine) \nSUM(amt) GROUP BY class \u2192 14 teams \n #1 grocery_pos 1,140,033.24 \n #2 shopping_net 773,527.93 \n #3 shopping_pos 725,766.14 \n #4 gas_transport 648,804.24 \n #5 dwelling 556,526.53 \nLatency: 100.47ms | Rows scanned: 100,000<\/p>\n
RAG SIMULATION \u2014 what the LLM receives at every context dimension<\/p>\n
Context Rows Protection Partial sum Error detectable? \ntiny (~325 tokens) 5 0.0050% 197.73 EASY \nsmall (~3K tokens) 50 0.0500% 2,003.56 MODERATE \nmedium (~32K tokens) 500 0.5000% 31,023.21 HARD \nmassive (~130K tokens) 2,000 2.0000% 140,093.16 VERY HARD \nxlarge (~520K tokens) 8,000 8.0000% 569,368.22 NEAR IMPOSSIBLE<\/p>\n
I stared at these outcomes for some time. Essentially the most troubling half wasn\u2019t simply that the solutions had been flawed, it was how a lot tougher the errors turned to identify because the context window grew.<\/p>\n
At 8,000 rows the error was nonetheless over 50%, but the response seemed like an expert report. You\u2019d have to manually confirm the numbers to note one thing was off. That\u2019s what I began calling Error Observability Collapse. The extra context I gave the mannequin, the extra convincing \u2014 however no more correct \u2014 the output turned.<\/p>\n
The \u201cPartial sum\u201d column reveals the overall if the LLM added each quantity worth within the rows it really retrieved. The \u201cError detectable?\u201d column scores how doubtless a human reader is to identify a mistake.<\/p>\n
With 5 rows, the partial sum is 197.73. The right whole is 1,140,033.24. It’s apparent. The output is brief, the numbers are flawed, and the lacking knowledge is obvious. The error is immediate.<\/p>\n
At 8,000 rows, the partial sum hits 569,368.22. The LLM has now seen all 14 classes. It generates a 1,500-word report with particular figures and assured language. The error is 50%, however it’s hidden inside authoritative, well-structured prose. With out an exterior reference, a reader has no solution to catch it.<\/p>\n
That is the sample that held throughout all seven queries:<\/p>\nContext WindowRowsDataset CoverageResponse LengthError Detectable?~325 tokens50.005%~50 wordsYES \u2014 clearly a guess~3K tokens500.050%~150 wordsMAYBE~32K tokens5000.500%~400 wordsHARD~130K tokens2,0002.000%~800 wordsVERY HARD~520K tokens8,0008.000%~1,500 wordsNEAR IMPOSSIBLESemantic Engine100,000100%<200msN\/A \u2014 precise<\/figure>\n
I referred to as this Error Observability Collapse. As context grows, confidence scales with it. Correctness doesn’t.<\/p>\nThe phantasm of context: How bigger context home windows in RAG and LLM programs enhance person confidence and reduce error detectability with out enhancing precise accuracy. Picture by Creator. <\/figcaption><\/figure>\n
The failure modes are uneven, which makes them harmful:<\/p>\n
A flawed RAG reply appears to be like right. It’s formatted, particular, and assured. A failed computation throws an specific error. It’s seen.<\/p>\n
One failure is silent. The opposite is loud. As context home windows attain hundreds of thousands of tokens, the silent failure turns into tougher to detect [4]. The system doesn’t get safer because it scales. It simply will get extra convincing.<\/p>\n
The Semantic Engine: Proof That the Right Reply Is Quick<\/h2>\n
Earlier than I totally understood the issue, I had already thrown collectively a easy semantic engine out of frustration. I simply needed the right reply at the very least as soon as.<\/p>\n
The method turned out to be easy: parse the question into correct operations and run a single go over your entire dataset. No embeddings, no retrieval, no guessing.<\/p>\n
Right here\u2019s what that appears like in follow:<\/p>\n
The logic is straightforward. Take a question like \u201cWhat’s the whole spend by class?\u201d. The engine maps this to a direct operation: SUM(amt) GROUP BY class. It processes the total 100,000-row set in a single go. It accumulates grouped totals. There is no such thing as a retrieval. No inference. No partial scanning. It visits each row as soon as and returns the precise outcome.<\/p>\n
This proves that the right reply isn’t costly. Benchmark queries completed below 200ms. Pattern dimension: 100,000 rows. Aggregation is trivial. The failure occurs once you route these queries to a system constructed to misconceive them.<\/p>\n
from context_window_engine import compute_ground_truth, load_csv<\/p>\n
Engine helps SUM, AVG, COUNT, MIN, MAX. Handles categorical and numeric filters. Consists of GROUP BY and ratios. Zero exterior dependencies. Each operation runs as a deterministic operate over the total checklist.<\/p>\n
The engine itself isn\u2019t the product. It\u2019s the proof: the right reply is reachable below a second. No inference required. The actual problem is routing queries there reliably. <\/p>\n
The Repair Is Not Higher Retrieval<\/h2>\n
Cease making an attempt to enhance retrieval. If a question wants 100% of the information, an 8% pattern fails. The repair is eradicating retrieval from the loop.<\/p>\n
We’d like a classification layer. It sits earlier than the pipeline and makes one binary name: computation or lookup?<\/p>\n
The distinction is obvious. \u201cWhole spend by class\u201d calls for a full scan. \u201cDiscover transactions from Jennifer Banks\u201d is a straightforward lookup. Normal RAG forces each down the identical path. That’s the design flaw.<\/p>\n
A QueryRouter fixes this. It inspects each incoming question and routes it to the right path earlier than a single retrieval begins.<\/p>\nIntent-based question routing structure, separating analytical calculation intents from semantic data retrieval pipelines. Picture by Auhor.<\/figcaption><\/figure>\n
The classifier makes use of three sign tiers, prioritized. Tier 1: aggregation verbs\u2014whole, what number of, common, lowest, share. These demand full-dataset computation. Tier 2: numeric comparability\u2014larger than 500, above $1,000, at the very least. These suggest filter-then-aggregate, not possible for RAG. Tier 3: retrieval indicators\u2014discover, present me, checklist, fetch. These point out lookups the place semantic similarity works.<\/p>\nTierSignalExamplesRoute1Aggregation verbtotal, what number of, common, lowest, percentageCOMPUTATION2Numeric comparisongreater than 500, above $1,000, at leastCOMPUTATION3Retrieval signalfind, present me, checklist, fetchRETRIEVAL0No matchambiguousCOMPUTATION \u2014 safer default<\/figure>\n
Default to COMPUTATION if no tier matches. That is deliberate. Failure modes are uneven: a flawed RAG reply on an aggregation is silently flawed. A computation engine that may\u2019t parse a question throws an error. When doubtful, fail loudly.<\/p>\n
outcome = router.route(“Discover transactions from Jennifer Banks”) \n# outcome.routed_to \u2192 “RETRIEVAL” \n# outcome.reply.secure \u2192 True \u2014 RAG is acceptable<\/p>\n
Routing the Full Benchmark<\/h2>\n
I ran 9 queries by means of the router to confirm efficiency throughout each sorts: seven aggregation queries destined for the semantic engine, and two lookup queries for RAG.<\/p>\n
Each route was right. The seven aggregation queries hit the full-scan engine and returned precise outcomes. The 2 lookup queries accurately triggered the RAG path. Have a look at the output: excessive confidence scores, right sample matching, and latency below 130ms\u2014even with the 100,000-row scan.<\/p>\n
[4] \u2713 COMPUTATION “What number of transactions had been made by feminine clients?” \n Tier 1 | matched=’What number of’ | confidence=0.97 \n 54,641.00 (90.45ms | 100,000 rows | precise)<\/p>\n
[5] \u2713 COMPUTATION “What’s the whole spend the place quantity is bigger than 500?” \n Tier 1 | matched=’whole’ | confidence=0.97 \n 1,274,269.60 (91.65ms | 100,000 rows | precise)<\/p>\n
[6] \u2713 COMPUTATION “Which state has the bottom whole spending?” \n Tier 1 | matched=’lowest’ | confidence=0.97 \n lowest RI 2,125.60 (109.05ms | 100,000 rows | precise)<\/p>\n
[8] \u2713 RETRIEVAL “Discover transactions from Jennifer Banks” \n Tier 3 | matched=’Discover’ | confidence=0.85 \n RAG is acceptable \u2014 no aggregation required<\/p>\n
[9] \u2713 RETRIEVAL “Present me a pattern transaction from Texas” \n Tier 3 | matched=’Present me’ | confidence=0.85 \n RAG is acceptable \u2014 no aggregation required<\/p>\n
Routing accuracy: 9\/9<\/p>\n
9\/9 right. Error Observability Collapse is not possible if aggregation queries by no means attain RAG.<\/p>\n
The Take a look at Suite<\/h2>\n
The benchmark verifies 9 particular queries. The check suite ensures reliability throughout a broader vary: edge circumstances, malformed inputs, lacking knowledge, and customary manufacturing failure factors. <\/p>\n
The engine suite has 87 checks throughout 10 lessons. It covers float parsing with greenback indicators, commas, and scientific notation; all 5 aggregation features below regular circumstances and with empty inputs; all 5 numeric filter operators; full GROUP BY aggregation with categorical and numeric filters mixed; RAG simulation protection metrics at every context dimension; and edge circumstances together with empty datasets, rows with lacking column values, and single-row inputs.<\/p>\n
The router suite has 72 checks throughout 5 lessons. It covers all three tier patterns, together with edge circumstances like all-caps queries and really lengthy queries; pure language to typed operation parsing for each supported question kind; routing and execution correctness towards all seven benchmark queries; and a distinction suite that verifies router solutions match impartial ground-truth computation \u2014 guaranteeing the router doesn’t introduce any deviation from the engine\u2019s personal output.<\/p>\n
Run the engine checks by typing python house -m house unittest house test_engine house -v. This executes the 87 checks within the suite.<\/p>\n
Run the router checks by typing python house -m house unittest house test_router house -v. This executes the 72 checks within the suite.<\/p>\n
All 159 go on Python 3.9+ with zero exterior dependencies.<\/p>\n
Sincere Limitations<\/h2>\n
This resolution isn\u2019t good. It solely works on single CSV information proper now. Actual manufacturing datasets are often messy with a number of tables that want becoming a member of \u2014 I intentionally saved the scope small as a result of I needed one thing that really labored end-to-end first.<\/p>\n
The router can also be nonetheless fairly fundamental (regex-based). I attempted a small LLM-based classifier early on however it was inconsistent and added latency, so I went again to the straightforward method. Typically the boring resolution wins.<\/p>\n
I additionally simulated the RAG responses as an alternative of hitting actual APIs for the benchmark. The patterns maintain up, however your mileage with GPT-4o or Claude 3.5 will fluctuate barely.<\/p>\n
CSV format required. The engine hundreds structured knowledge straight from CSV information. Database connections, Parquet information, and different tabular codecs usually are not supported presently.<\/p>\n
What This Modifications<\/h2>\n
Including a routing layer prices nearly nothing. Classifying a question towards 65 regex patterns takes simply microseconds. The semantic engine provides lower than 200ms to scan a 100,000 row dataset. The whole overhead is smaller than a single embedding name.<\/p>\n
What you get in return is a deterministic reply for each aggregation question. Each whole, each depend, and each share now comes from a full scan as an alternative of a assured approximation primarily based on 8 p.c of the information. RAG retains dealing with what it’s really good at: retrieving particular data, surfacing related passages, and answering lookup questions the place semantic similarity is the proper instrument for the job.<\/p>\n
RAG isn’t damaged. It’s simply being requested to compute, and it can not try this.The harmful half isn’t that it fails. It’s that it fails convincingly. And no quantity of context modifications that.<\/p>\n
You may strive typing it out like this:<\/p>\n
To start out, clone the repository utilizing git clone adopted by the URL https:\/\/github.com\/Emmimal\/context-window-engine\/. As soon as that finishes, transfer into the listing by typing cd context-window-engine. Lastly, launch the mission by operating python demo.py in your terminal.<\/p>\n
References<\/h2>\n
[1] Levy, M., Jacoby, A., & Goldberg, Y. (2024). Similar process, extra tokens: The impression of enter size on the reasoning efficiency of huge language fashions. In Proceedings of the 62nd Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers), pages 15339\u201315353, Bangkok, Thailand. Affiliation for Computational Linguistics.https:\/\/doi.org\/10.18653\/v1\/2024.acl-long.818<\/p>\n
[2] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K\u00fcttler, H., Lewis, M., Yih, W.-t., Rockt\u00e4schel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented technology for knowledge-intensive NLP duties. Advances in Neural Data Processing Methods, 33, 9459\u20139474. https:\/\/doi.org\/10.48550\/arXiv.2005.11401<\/p>\n
[3] Gao, Y., Xiong, Y., Gao, X., Jia, Ok., Pan, J., Bi, Y., Dai, Y., Solar, J.,Guo, Q., Wang, M., & Wang, H. (2023). Retrieval-augmented technology for giant language fashions: A survey. arXiv preprint arXiv:2312.10997.https:\/\/doi.org\/10.48550\/arXiv.2312.10997<\/p>\n
[4] Liu, N. F., Lin, Ok., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Misplaced within the center: How language fashions use lengthy contexts. Transactions of the Affiliation for Computational Linguistics, 12, 157\u2013173. https:\/\/doi.org\/10.1162\/tacl_a_00638<\/p>\n
[5] Koshorek, O., Granot, N., Alloni, A., Admati, S., Hendel, R., Weiss, I., Arazi, A., Cohen, S.-N., & Belinkov, Y. (2025). Structured RAG for answering aggregative questions. arXiv preprint arXiv:2511.08505.https:\/\/doi.org\/10.48550\/arXiv.2511.08505<\/p>\n
Disclosure<\/h2>\n
All benchmark numbers are from precise runs on Python 3.12.6, Home windows 11, CPU solely, no GPU. The benchmark makes use of the Credit score Card Transactions Fraud Detection dataset (Kartik Gajjar, Kaggle, 2020), an artificial dataset generated utilizing the Sparkov transaction simulator created by Brandon Harris, licensed CC0 (Public Area), accessible at kaggle.com\/datasets\/kartik2112\/fraud-detection. The RAG baseline simulates retrieval and fashions confidence indicators \u2014 no actual LLM API calls are made. No exterior API keys are required to breed any outcome on this article. All code described right here was written and examined by me.<\/p>\n<\/div>\n
![\"Diagram](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/Two-Pipelines-Same-Query-1024x615.png\")
![\"Flowchart](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/Error-Observability-Collapse-1024x556.png\")
![\"Architectural](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/06\/QueryRouter-1024x687.png\")