Agentic RAG: Let the Agent Search

software we construct is a RAG app.

The recipe is easy: chunk, embed, retrieve, then reply.

It seems to be clear on paper. However when you apply it to actual instances, issues get messy in a short time: similarity search finds related wordings however not essentially helpful chunks, the appropriate proof by no means exhibits up within the retrieved context because it ranks too low, or necessary context could also be break up throughout chunk boundaries.

With inadequate context, the LLM has little room to get better.

So, how about we make retrieval iterative? What if the mannequin can search, learn, determine whether or not it has sufficient proof, and search once more when wanted? Most likely we don’t even want the vector embeddings within the first place.

That’s the premise of agentic RAG.

On this publish, we’ll construct a mini agentic RAG workflow with the OpenAI Brokers SDK. We’ll study how the agent iteratively searches, reads, and grounds its reply.

On the finish, we’ll take a step again and briefly talk about the concerns for constructing a sensible agentic RAG resolution.

1. Case Research: Answering a Coverage Query with Agentic RAG

For our case examine, we’ll construct a coverage RAG agent over an organization coverage doc assortment.

1.1 Curating The Doc Assortment

Right here, I created six artificial firm coverage docs. They’re all markdown recordsdata. Every one has a title, an efficient date, a brief abstract, and the coverage textual content.

To be life like, these docs cowl 6 frequent firm coverage areas:

approval_matrix.md, containing approval ranges for frequent enterprise journey selections, efficient on July 1, 2025.

conference_guidelines.md, containing guidelines for attending exterior occasions, efficient on Might 15, 2025.

faq.md, containing casual solutions to frequent journey questions, efficient on September 1, 2025.

policy_updates_2026.md, containing updates to lodging, convention journey, and approval timing for 2026, efficient on January 1, 2026.

remote_work_policy.md, containing guidelines for distant work, efficient on February 1, 2026.

travel_policy.md, containing commonplace journey reserving guidelines for flights, lodging, meals, and transportation, efficient on March 1, 2025.

We made it intentional that the reply to a coverage query might not reside in a single doc. This enables us to see the specified agentic habits.

Yow will discover the complete artificial paperwork and the agentic RAG implementation pocket book right here.

1.2 Defining The Agent

Subsequent, we configure the agent. For that, we use the OpenAI Brokers SDK.

At a excessive degree, the agent is simply this:

# pip set up openai-agents
from brokers import Agent

agent = Agent(
title=”Coverage analysis assistant”,
directions=INSTRUCTIONS,
mannequin=”gpt-5.4″,
instruments=[list_docs, search_docs, read_doc],
)

Two components we have to undergo: the agent instruction, and the instruments it has entry to.

First, the instruction. That is the place we outline the specified search habits:

# Be aware: This instruction is iterated with AI
INSTRUCTIONS = “””
[Role]
You’re a cautious inner coverage analysis assistant.

[Research behavior]
Reply worker coverage questions utilizing the doc instruments.
Discover sufficient related proof to assist the reply.
Preserve conclusions grounded within the coverage paperwork.

[Expected output]
Give a direct reply first.
Then briefly clarify the proof.
Cite the doc filenames used for every necessary declare.
“””.strip()

For this case examine, we require that the agent can solely contact the docs through three pre-defined instruments:

The primary one is a instrument that provides the agent a fast overview of what paperwork exist:

@function_tool
def list_docs() -> checklist[dict]:
“””Listing accessible coverage paperwork with out returning their physique textual content.”””
return [
{
“doc_name”: doc[“doc_name”],
“title”: doc[“title”],
“efficient”: doc[“effective”],
“abstract”: doc[“summary”],
}
for doc in docs.values()
]

The second instrument is a keyword-search instrument. We hold it easy right here: every doc is break up into paragraph chunks, and every question is matched towards these chunks by token overlap:

@function_tool
def search_docs(question: str) -> checklist[dict]:
“””Search coverage paperwork and return the highest three brief snippets.”””
query_tokens = tokenize(question)
scored = []

for chunk in chunks:
rating = len(query_tokens & chunk[“tokens”])
if rating:
scored.append((rating, chunk))

scored.type(key=lambda merchandise: merchandise[0], reverse=True)

outcomes = []
for rating, chunk in scored[:3]:
snippet = chunk[“text”].substitute(“n”, ” “)
if len(snippet) > 420:
snippet = snippet[:417].rstrip() + “…”
outcomes.append({
“doc_name”: chunk[“doc_name”],
“title”: chunk[“title”],
“part”: chunk[“section”],
“snippet”: snippet,
“rating”: spherical(rating, 2),
})

return outcomes

The final instrument is what permits the agent to open one doc by filename:

@function_tool
def read_doc(doc_name: str) -> str:
“””Learn one coverage doc by filename.”””
if doc_name not in docs:
legitimate = “, “.be part of(sorted(docs))
return f”Unknown doc: {doc_name}. Legitimate paperwork: {legitimate}”

return docs[doc_name][“text”]

That’s the complete RAG agent.

1.3 Working One Coverage Query

Now we check the agent with one concrete query:

We run the agent with:

from brokers import Runner

end result = await Runner.run(agent, PROMPT, max_turns=12)

The agent produced the appropriate reply: sure, the worker can e book the official convention resort if there’s a sensible enterprise purpose. It received that info from conference_guidelines.md.

For the approval half, the agent first recognized that approval is required because the resort is above the traditional cap. Then it gave the corresponding approval situations. The agent used travel_policy.md, approval_matrix.md, and policy_updates_2026.md to assist its reply, which is strictly what we’d anticipate.

The extra attention-grabbing half is the hint, from which we are able to learn the way the agent thinks. We will present the hint within the following manner:

for merchandise in end result.new_items:
print(kind(merchandise).__name__, merchandise)

end result.new_items comprises the intermediate instrument calls and power outputs produced by the agent. In my run, I can see that the agent first known as search_docs() with key phrases like convention resort, resort cap, approval, and Berlin. Then, it known as list_docs() to examine the accessible coverage paperwork. After that, it opened the related recordsdata with read_doc(). Solely then did it produce the ultimate reply.

That is precisely the agentic loop we needed to see.

3. What to Resolve Earlier than Constructing Agentic RAG

The case examine we simply went by way of solely scratched the floor. To essentially construct a sensible agentic RAG resolution, primarily based on my expertise, I recommend you reply the next 5 questions:

Q1: How a lot freedom ought to the agent have?

One frequent choice is strictly what we now have achieved within the earlier case examine: we uncovered a few fastidiously curated instruments, and the agent is barely allowed to make use of these instruments to do the investigation. That is easy when it comes to controlling, testing, and auditing.

However we are able to additionally give the agent broader entry, resembling shell and file system. This manner, the agent can instantly run scripts to look and examine recordsdata, and possibly even do additional knowledge processing to generate helpful artifacts, all by itself.

This sample may be far more highly effective, however it additionally will increase threat and makes habits tougher to foretell.

So for many RAG purposes, I’d begin with curated instruments first, and solely add shell/file-system entry when the duty complexity justifies it.

Q2: Ought to the agent search uncooked textual content solely?

Most RAG initiatives would possibly begin with plain textual content like PDFs, wiki pages, manuals, and many others. That’s effective.

However in follow, we are able to typically make retrieval simpler by deriving a information layer on high of the uncooked texts.

These derived information artifacts may be doc metadata, summaries, cross-document hyperlinks, or we are able to go additional and implement a correct information graph.

These derived information artifacts assist the agent navigate the corpus, whereas the uncooked texts stay because the supply of fact.

Q3: Will we nonetheless want embeddings?

Agentic RAG doesn’t essentially imply embeddings are gone.

Vector embeddings are nonetheless an environment friendly option to discover semantically related texts, and it typically outperforms a pure key phrase search technique.

In agentic RAG, what modified primarily is that the retrieval turns into an “motion” the agent can take. Below this framing, “motion” can nonetheless be powered by an embedding-based retriever, a keyword-based one, or perhaps a hybrid one.

So embeddings can nonetheless be helpful. They’re only one potential option to energy the agent’s search instrument.

This fall: Ought to one agent deal with every part?

The best agentic RAG setup is only one agent that does the search, learn, and reply.

However as the duty will get extra advanced, you would possibly need to break up the work amongst a number of brokers. Extra concretely, you would possibly have to undertake a multi-agent technique.

You may break up the work by position. For instance, the planner-retriever-writer break up, the place the planner decides what proof is required, the retriever collects it, and the author produces the ultimate reply through the use of the collected proof.

You may as well break up by supply kind, the place every agent is supplied with custom-made instruments and focuses on one particular kind of supply.

Simply be mindful: A multi-agent setup provides coordination complexity, and there’s no assure that it’s going to carry out higher than a single-agent setup. Empirical testing is essential.

Q5: Ought to we at all times use agentic RAG?

Possibly not at all times.

Simply because agentic RAG turns into a classy matter doesn’t essentially imply it is best to at all times default to it.

Agentic RAG provides extra flexibility, however that comes with prices. That value will not be solely about latency or token value, but in addition much less predictable agent habits.

At all times begin easy, then add agentic loops when the query truly wants iterative retrieval.