Experiments and outcomes
We evaluated agentic RAG on FramesQA, which relies on the FRAMES paper. An instance multi-hop query is:
“Of the highest two most watched tv season finales (as of June 2024), which finale ran the longest in size and by how a lot?”
The RAG system must carry out a number of steps to reach on the right reply. First, it has to establish that the 2 most watched finales are from the reveals M*A*S*H and Cheers. Then, it has to seek out their operating instances, and calculate the size distinction. In lots of RAG settings (Vanilla RAG or agentic RAG with out adequate context), we may find yourself in a scenario the place the mannequin says one thing like:
“Regardless of a number of scans, I discovered no express runtimes for M*A*S*H or Cheers. The paperwork present viewership information, however not the length in minutes or hours.”
This doesn’t reply the query.
Fortuitously, our agentic RAG can remedy this by first looking for the TV reveals, then utilizing the Question Rewriter and Adequate Context Agent to have a focused seek for the run time of M*A*S*H or Cheers. Then, Gemini can simply decide which finale ran the longest in size and by how a lot:
“The M*A*S*H finale ran for 150 minutes, making it the longest of the highest two. It was 52 minutes longer than the Cheers finale, which ran for roughly 98 minutes.”
We ran an experiment to check this potential at scale (FramesQA has 824 queries together with a corpus containing 2,676 PDF paperwork). Within the “Vanilla” RAG setting, we use Google’s RAG Engine (which has a sophisticated retrieval engine, LLM parser, and re-ranker). We in contrast this with our agentic RAG in two settings. Within the single-corpus setting, we retrieve from the FramesQA paperwork. Within the cross-corpus setting, we additionally embody three different distracting datasets, the place the Planner Agent should decide the place to retrieve from. This cross-corpus setting mimics use instances the place firms have databases managed by separate groups. We compute accuracy through the use of an LLM-as-a-judge to check the system responses to the bottom reality solutions within the dataset.
Within the cross-corpus setting, our system practically matches its single-corpus accuracy. Even when the Planner Agent should choose the proper corpus out of 4 prospects, we efficiently route the search queries and reply 90.1% of questions accurately. Additionally, the latency of each single- and cross-corpus variations is about the identical (inside 3% on common). This demonstrates that our Agentic RAG system can cause over a number of, unrelated information sources, which opens up prospects for extra versatile retrieval eventualities.
