On this article, you’ll find out how an agent’s strategy to managing state — stateless or stateful — shapes each its implementation and the deployment structure constructed round it.
Subjects we are going to cowl embody:
What separates stateless from stateful brokers, and the tradeoffs every design imposes on scaling.
The right way to implement a stateless agent that relies upon solely on the consumer to produce dialog historical past.
The right way to implement a stateful agent that manages its personal reminiscence via a database layer.
Introduction
A earlier article laid out a complete architectural roadmap for AI agent deployment, inspecting the infrastructure wanted to convey brokers into manufacturing settings.
As a follow-up, we now flip to a basic, sensible query that needs to be answered earlier than any load balancer is configured: the place does the agent’s reminiscence reside? Brokers could deal with their state (the context gained to this point and the dialog historical past) in numerous methods, and this code-level determination can considerably influence all the deployment structure.
This text breaks down the 2 major paradigms for dealing with an agent’s state: stateless and stateful design. A simplified model of a real-world implementation, utilizing open language fashions served via the quick Groq API, will illustrate these concepts in apply.
Preliminary Setup
If that is the primary time you might be utilizing language fashions from Groq in a Python program, you’ll want to put in the required library: pip set up groq.
After that, we import it and set our Groq API key within the code under:
import os
from groq import Groq
# Get an API key in https://console.groq.com/keys and set it right here
os.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
# Initializing the consumer
consumer = Groq()
# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Immediate
MODEL_ID = “llama-3.1-8b-instant”
import os
from groq import Groq
# Get an API key in https://console.groq.com/keys and set it right here
os.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
# Initializing the consumer
consumer = Groq()
# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Immediate
MODEL_ID = “llama-3.1-8b-instant”
An essential setup determination right here is the selection of a selected mannequin. llama-3.1-8b-instant is a extremely cost-efficient mannequin that’s, on the time of writing, generously supported on Groq’s 2026 free tier: it permits as much as 14,400 requests per day. That makes it a perfect alternative for illustrating the stateless and stateful agent paradigms under.
Stateless Brokers: Hearth and Neglect
Stateless brokers deal with every request as fully remoted and unbiased. The agent reads the person immediate, invokes the LLM inference engine, and delivers the output. As soon as that execution cycle ends, every little thing is forgotten.
The Tradeoff
Architectures based mostly on stateless brokers might be scaled horizontally with outstanding ease. Since no person reminiscence is saved on a backend server, incoming requests might be forwarded to any accessible occasion. There’s, nevertheless, an essential limitation in multi-turn conversations: the frontend should re-send the entire dialog historical past alongside each new request. In consequence, the context window grows with a snowballing impact, shortly driving up token utilization.
Illustrative Instance
This runnable code illustrates, via a fundamental situation, how a stateless agent sometimes interacts with a Groq language mannequin.
First, we outline a stateless_agent perform that emulates an agent’s interplay with our chosen mannequin. Importantly, no state or reminiscence of the dialog is saved internally. As an alternative, the earlier dialog historical past can optionally be handed in as a parameter and appended to the present immediate. The API name to the Groq mannequin takes place in consumer.chat.completions.create().
def stateless_agent(immediate: str, provided_history: listing = None) -> str:
“””
The agent depends fully on the consumer to supply context.
It retains no data from previous interactions in native reminiscence.
“””
# Initializing with a system immediate
messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# Appending no matter historical past the consumer offered
if provided_history:
messages.lengthen(provided_history)
# Appending the brand new immediate
messages.append({“function”: “person”, “content material”: immediate})
# The LLM processes all the chain of messages
response = consumer.chat.completions.create(
mannequin=MODEL_ID,
messages=messages,
max_tokens=100
)
return response.selections[0].message.content material.strip()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def stateless_agent(immediate: str, provided_history: listing = None) -> str:
“”“
The agent depends fully on the consumer to supply context.
It retains no data from previous interactions in native reminiscence.
““”
# Initializing with a system immediate
messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# Appending no matter historical past the consumer offered
if provided_history:
messages.lengthen(provided_history)
# Appending the brand new immediate
messages.append({“function”: “person”, “content material”: immediate})
# The LLM processes all the chain of messages
response = consumer.chat.completions.create(
mannequin=MODEL_ID,
messages=messages,
max_tokens=100
)
return response.selections[0].message.content material.strip()
To know the constraints of a stateless agent, we simulate a easy user-model dialog via it:
# — Testing the Stateless Agent —
print(“— Flip 1 —“)
prompt_1 = “Hello, my identify is Alice and I’m studying about API infrastructure.”
response_1 = stateless_agent(prompt_1)
print(f”Agent: {response_1}”)
print(“n— Flip 2 (With out Shopper Context) —“)
# The agent fails right here as a result of it retained no reminiscence of Flip 1
prompt_2 = “What’s my identify and what am I studying about?”
response_2 = stateless_agent(prompt_2)
print(f”Agent: {response_2}”)
print(“n— Flip 2 (With Shopper Context) —“)
# The frontend MUST inject the historical past into the payload for the agent to succeed
frontend_payload = [
{“role”: “user”, “content”: prompt_1},
{“role”: “assistant”, “content”: response_1}
]
response_3 = stateless_agent(prompt_2, provided_history=frontend_payload)
print(f”Agent: {response_3}”)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# — Testing the Stateless Agent —
print(“— Flip 1 —“)
prompt_1 = “Hello, my identify is Alice and I’m studying about API infrastructure.”
response_1 = stateless_agent(prompt_1)
print(f“Agent: {response_1}”)
print(“n— Flip 2 (With out Shopper Context) —“)
# The agent fails right here as a result of it retained no reminiscence of Flip 1
prompt_2 = “What’s my identify and what am I studying about?”
response_2 = stateless_agent(prompt_2)
print(f“Agent: {response_2}”)
print(“n— Flip 2 (With Shopper Context) —“)
# The frontend MUST inject the historical past into the payload for the agent to succeed
frontend_payload = [
{“role”: “user”, “content”: prompt_1},
{“role”: “assistant”, “content”: response_1}
]
response_3 = stateless_agent(prompt_2, provided_history=frontend_payload)
print(f“Agent: {response_3}”)
Output:
— Flip 1 —
Agent: Hey Alice, good to fulfill you. Studying about API infrastructure is usually a fascinating and rewarding subject. What particular elements of API infrastructure would you wish to discover or focus on? Are you searching for data on API administration, safety, deployment, or one thing else?
— Flip 2 (With out Shopper Context) —
Agent: Sadly, I haven’t got any details about you, together with your identify. Our dialog simply began, so I am right here that can assist you with any questions or matters you’d wish to find out about. Please be at liberty to share your identify and a subject you are concerned about studying about.
— Flip 2 (With Shopper Context) —
Agent: Your identify is Alice, and you might be studying about API infrastructure.
—– Flip 1 —–
Agent: Hey Alice, good to meet you. Studying about API infrastructure can be a fascinating and rewarding subject. What particular elements of API infrastructure would you like to discover or focus on? Are you trying for data on API administration, safety, deployment, or one thing else?
—– Flip 2 (With out Shopper Context) —–
Agent: Sadly, I don‘t have any details about you, together with your identify. Our dialog simply began, so I’m right here to assist you with any questions or matters you‘d wish to find out about. Please be at liberty to share your identify and a subject you’re in studying about.
—– Flip 2 (With Shopper Context) —–
Agent: Your identify is Alice, and you are studying about API infrastructure.
The implementation is straightforward, however with out a consumer or frontend that sends the total dialog historical past to the agent on each flip, the agent’s LLM lacks the context it must reply sure questions correctly.
Stateful Brokers: Context-driven Continuity
Underneath this strategy, the agent takes on the reminiscence burden itself. The consumer, in the meantime, solely must ship the latest person immediate along with a singular identifier, usually related to the present session. The agent then retrieves the session historical past or context from a database and appends the brand new message to it. As soon as the LLM inference has been processed, the agent updates the context within the database.
The Tradeoff
It is a a lot neater expertise from the consumer facet. It additionally facilitates advanced and asynchronous workflows during which brokers could must pause their execution and anticipate instruments, app responses, or human approval. However it all comes with a value: scaling this answer turns into a lot tougher, beginning with the necessity for a persistent database layer within the structure. In infrastructures that scale horizontally, methods reminiscent of centralized reminiscence caching with Redis might also turn into essential to keep away from “localized amnesia”, the place a session’s historical past is stranded on the only occasion that occurred to serve the sooner turns.
Illustrative Instance
We illustrate the fundamental concepts behind a stateful agent by incorporating a “persistent” database layer. For simplicity, we use a tiny SQLite database. The secret’s to have the agent handle its personal dialog reminiscence as a substitute of relying on a frontend to supply it externally:
import sqlite3
import json
# Initializing an in-memory SQLite database for pocket book testing
conn = sqlite3.join(‘:reminiscence:’)
cursor = conn.cursor()
cursor.execute(”’CREATE TABLE IF NOT EXISTS agent_memory (session_id TEXT PRIMARY KEY, historical past TEXT)”’)
conn.commit()
def stateful_agent(session_id: str, new_prompt: str) -> str:
“””
The agent manages its personal state utilizing a database.
The consumer solely sends the brand new immediate and their session ID.
“””
# 1. Retrieving current state from the database
cursor.execute(“SELECT historical past FROM agent_memory WHERE session_id=?”, (session_id,))
row = cursor.fetchone()
if row:
conversation_history = json.hundreds(row[0])
else:
# Initializing with system immediate for brand spanking new classes
conversation_history = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# 2. Appending the brand new person immediate
conversation_history.append({“function”: “person”, “content material”: new_prompt})
# 3. Processing the LLM name utilizing the retrieved historical past
response = consumer.chat.completions.create(
mannequin=MODEL_ID,
messages=conversation_history,
max_tokens=100
).selections[0].message.content material.strip()
# 4. Updating the state with the assistant’s reply
conversation_history.append({“function”: “assistant”, “content material”: response})
# 5. Saving the brand new state again to the database
cursor.execute(”’
INSERT INTO agent_memory (session_id, historical past)
VALUES (?, ?)
ON CONFLICT(session_id) DO UPDATE SET historical past=excluded.historical past
”’, (session_id, json.dumps(conversation_history)))
conn.commit()
return response
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import sqlite3
import json
# Initializing an in-memory SQLite database for pocket book testing
conn = sqlite3.join(‘:reminiscence:’)
cursor = conn.cursor()
cursor.execute(”‘CREATE TABLE IF NOT EXISTS agent_memory (session_id TEXT PRIMARY KEY, historical past TEXT)’”)
conn.commit()
def stateful_agent(session_id: str, new_prompt: str) -> str:
“”“
The agent manages its personal state utilizing a database.
The consumer solely sends the brand new immediate and their session ID.
““”
# 1. Retrieving current state from the database
cursor.execute(“SELECT historical past FROM agent_memory WHERE session_id=?”, (session_id,))
row = cursor.fetchone()
if row:
conversation_history = json.hundreds(row[0])
else:
# Initializing with system immediate for brand spanking new classes
conversation_history = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# 2. Appending the brand new person immediate
conversation_history.append({“function”: “person”, “content material”: new_prompt})
# 3. Processing the LLM name utilizing the retrieved historical past
response = consumer.chat.completions.create(
mannequin=MODEL_ID,
messages=conversation_history,
max_tokens=100
).selections[0].message.content material.strip()
# 4. Updating the state with the assistant’s reply
conversation_history.append({“function”: “assistant”, “content material”: response})
# 5. Saving the brand new state again to the database
cursor.execute(”‘
INSERT INTO agent_memory (session_id, historical past)
VALUES (?, ?)
ON CONFLICT(session_id) DO UPDATE SET historical past=excluded.historical past
‘”, (session_id, json.dumps(conversation_history)))
conn.commit()
return response
Discover how the session identifier is used to question the related data from previous interactions within the dialog at hand.
Now let’s strive all of it in a dialog just like the earlier one, however this time with the person asking the agent to recall the person’s personal identify:
# — Testing the Stateful Agent —
print(“— Flip 1 —“)
print(f”Agent: {stateful_agent(‘user_123’, ‘Hello, I’m Bob and I need to scale my AI app.’)}”)
print(“n— Flip 2 —“)
# Discover how the consumer NO LONGER sends the context payload. Simply the session ID.
print(f”Agent: {stateful_agent(‘user_123’, ‘What was my identify once more?’)}”)
# — Testing the Stateful Agent —
print(“— Flip 1 —“)
print(f“Agent: {stateful_agent(‘user_123’, ‘Hello, I’m Bob and I need to scale my AI app.’)}”)
print(“n— Flip 2 —“)
# Discover how the consumer NO LONGER sends the context payload. Simply the session ID.
print(f“Agent: {stateful_agent(‘user_123’, ‘What was my identify once more?’)}”)
Output:
— Flip 1 —
Agent: Hey Bob, scaling an AI app is usually a advanced course of. Might I ask:
1. What kind of AI know-how is your app constructed on? (e.g., machine studying, pure language processing, laptop imaginative and prescient)
2. Are you utilizing any cloud providers like AWS, Google Cloud, or Azure?
3. What are your scalability targets (e.g., enhance person depend, scale back latency, enhance response occasions)?
This data will assist me higher perceive your necessities and supply more practical help.
— Flip 2 —
Agent: Your identify is Bob.
—– Flip 1 —–
Agent: Hey Bob, scaling an AI app can be a advanced course of. Might I ask:
1. What kind of AI know-how is your app constructed on? (e.g., machine studying, pure language processing, laptop imaginative and prescient)
2. Are you utilizing any cloud providers like AWS, Google Cloud, or Azure?
3. What are your scalability targets (e.g., enhance person depend, scale back latency, enhance response occasions)?
This data will assist me higher perceive your necessities and present extra efficient help.
—– Flip 2 —–
Agent: Your identify is Bob.
This instance is, after all, a great distance from a scaled-up manufacturing structure, nevertheless it serves to make clear the important thing distinction between how stateful and stateless brokers work.
Wrapping Up: The Tradeoffs
The selection between a stateful and a stateless architectural design boils right down to correctly matching the infrastructure to the workflow:
Stateless brokers are most well-liked in easy pipelines oriented to very particular duties, like textual content extraction, summarization, or single-turn classification chatbots. They preserve the structure light-weight, which is often sufficient in such use instances, avoiding database bottlenecks and permitting seamless horizontal scaling.
Stateful brokers make rather more sense if we intend to develop long-running assistants, coding assistants, or multi-turn bots in purposes like customer support. As a result of the agent owns the historical past, the consumer payload stays small on each flip, and the dialog might be trimmed or summarized server-side as a substitute of being resent in full because it grows.
