On this article, you’ll find out how an agent\u2019s strategy to managing state \u2014 stateless or stateful \u2014 shapes each its implementation and the deployment structure constructed round it.<\/p>\n
Subjects we are going to cowl embody:<\/p>\n
What separates stateless from stateful brokers, and the tradeoffs every design imposes on scaling.
\nThe right way to implement a stateless agent that relies upon solely on the consumer to produce dialog historical past.
\nThe right way to implement a stateful agent that manages its personal reminiscence via a database layer.<\/p>\n
![\"Stateful](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2026\/07\/mlm-stateful-vs-stateless-agent-design-tradeoffs-for-scalable-agentic-systems-feature.png\")
<\/p>\n
Introduction<\/h2>\n
A earlier article laid out a complete architectural roadmap for AI agent deployment, inspecting the infrastructure wanted to convey brokers into manufacturing settings.<\/p>\n
As a follow-up, we now flip to a basic, sensible query that needs to be answered earlier than any load balancer is configured: the place does the agent\u2019s reminiscence reside? Brokers could deal with their state (the context gained to this point and the dialog historical past) in numerous methods, and this code-level determination can considerably influence all the deployment structure.<\/p>\n
This text breaks down the 2 major paradigms for dealing with an agent\u2019s state: stateless and stateful design. A simplified model of a real-world implementation, utilizing open language fashions served via the quick Groq API, will illustrate these concepts in apply.<\/p>\n
Preliminary Setup<\/h2>\n
If that is the primary time you might be utilizing language fashions from Groq in a Python program, you\u2019ll want to put in the required library: pip set up groq.<\/p>\n
After that, we import it and set our Groq API key within the code under:<\/p>\n
\nimport os
\nfrom groq import Groq
\n
\n# Get an API key in https:\/\/console.groq.com\/keys and set it right here
\nos.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
\n
\n# Initializing the consumer
\nconsumer = Groq()
\n
\n# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Immediate
\nMODEL_ID = “llama-3.1-8b-instant”<\/p>\n
import <\/span>os<\/span><\/p>\n from <\/span>groq <\/span>import <\/span>Groq<\/span><\/p>\n \u00a0<\/p>\n # Get an API key in https:\/\/console.groq.com\/keys and set it right here<\/span><\/p>\n os<\/span>.<\/span>environ<\/span>[<\/span>“GROQ_API_KEY”<\/span>]<\/span> <\/span>=<\/span> <\/span>“PASTE_YOUR_GROQ_API_KEY_HERE”<\/span><\/p>\n \u00a0<\/p>\n # Initializing the consumer<\/span><\/p>\n consumer<\/span> <\/span>=<\/span> <\/span>Groq<\/span>(<\/span>)<\/span><\/p>\n \u00a0<\/p>\n # Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Immediate<\/span><\/p>\n MODEL_ID<\/span> <\/span>=<\/span> <\/span>“llama-3.1-8b-instant”<\/span><\/p>\n<\/div><\/div><\/div>\n An essential setup determination right here is the selection of a selected mannequin. llama-3.1-8b-instant is a extremely cost-efficient mannequin that’s, on the time of writing, generously supported on Groq\u2019s 2026 free tier: it permits as much as 14,400 requests per day. That makes it a perfect alternative for illustrating the stateless and stateful agent paradigms under.<\/p>\n Stateless brokers deal with every request as fully remoted and unbiased. The agent reads the person immediate, invokes the LLM inference engine, and delivers the output. As soon as that execution cycle ends, every little thing is forgotten.<\/p>\n Architectures based mostly on stateless brokers might be scaled horizontally with outstanding ease. Since no person reminiscence is saved on a backend server, incoming requests might be forwarded to any accessible occasion. There’s, nevertheless, an essential limitation in multi-turn conversations: the frontend should re-send the entire dialog historical past alongside each new request. In consequence, the context window grows with a snowballing impact, shortly driving up token utilization.<\/p>\n This runnable code illustrates, via a fundamental situation, how a stateless agent sometimes interacts with a Groq language mannequin. <\/p>\n First, we outline a stateless_agent perform that emulates an agent\u2019s interplay with our chosen mannequin. Importantly, no state or reminiscence of the dialog is saved internally. As an alternative, the earlier dialog historical past can optionally be handed in as a parameter and appended to the present immediate. The API name to the Groq mannequin takes place in consumer.chat.completions.create().<\/p>\n \ndef stateless_agent(immediate: str, provided_history: listing = None) -> str:
1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n 22<\/p>\n 23<\/p>\n<\/div>\n def <\/span>stateless_agent<\/span>(<\/span>immediate<\/span>:<\/span> <\/span>str<\/span>,<\/span> <\/span>provided_history<\/span>:<\/span> <\/span>listing<\/span> <\/span>=<\/span> <\/span>None<\/span>)<\/span> <\/span>-><\/span> <\/span>str<\/span>:<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0The agent depends fully on the consumer to supply context.<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0It retains no data from previous interactions in native reminiscence.<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span># Initializing with a system immediate<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>messages<\/span> <\/span>=<\/span> <\/span>[<\/span>{<\/span>“role”<\/span>:<\/span> <\/span>“system”<\/span>,<\/span> <\/span>“content”<\/span>:<\/span> <\/span>“You are a helpful, concise assistant.”<\/span>}<\/span>]<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span># Appending no matter historical past the consumer offered<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>provided_history<\/span>:<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>messages<\/span>.<\/span>lengthen<\/span>(<\/span>provided_history<\/span>)<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span># Appending the brand new immediate<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>messages<\/span>.<\/span>append<\/span>(<\/span>{<\/span>“function”<\/span>:<\/span> <\/span>“person”<\/span>,<\/span> <\/span>“content material”<\/span>:<\/span> <\/span>immediate<\/span>}<\/span>)<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span># The LLM processes all the chain of messages<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>response<\/span> <\/span>=<\/span> <\/span>consumer<\/span>.<\/span>chat<\/span>.<\/span>completions<\/span>.<\/span>create<\/span>(<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>mannequin<\/span>=<\/span>MODEL_ID<\/span>,<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>messages<\/span>=<\/span>messages<\/span>,<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>max_tokens<\/span>=<\/span>100<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n \u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>response<\/span>.<\/span>selections<\/span>[<\/span>0<\/span>]<\/span>.<\/span>message<\/span>.<\/span>content material<\/span>.<\/span>strip<\/span>(<\/span>)<\/span><\/p>\n<\/div><\/div><\/div>\n To know the constraints of a stateless agent, we simulate a easy user-model dialog via it:<\/p>\n \n# — Testing the Stateless Agent —
1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n<\/div>\n # — Testing the Stateless Agent —<\/span><\/p>\n \u00a0<\/p>\n print<\/span>(<\/span>“— Flip 1 —“<\/span>)<\/span><\/p>\n prompt_1<\/span> <\/span>=<\/span> <\/span>“Hello, my identify is Alice and I’m studying about API infrastructure.”<\/span><\/p>\n response_1<\/span> <\/span>=<\/span> <\/span>stateless_agent<\/span>(<\/span>prompt_1<\/span>)<\/span><\/p>\n print<\/span>(<\/span>f<\/span>“Agent: {response_1}”<\/span>)<\/span><\/p>\n \u00a0<\/p>\n print<\/span>(<\/span>“n— Flip 2 (With out Shopper Context) —“<\/span>)<\/span><\/p>\n # The agent fails right here as a result of it retained no reminiscence of Flip 1<\/span><\/p>\n prompt_2<\/span> <\/span>=<\/span> <\/span>“What’s my identify and what am I studying about?”<\/span><\/p>\n response_2<\/span>Stateless Brokers: Hearth and Neglect<\/h2>\n
The Tradeoff<\/h3>\n
Illustrative Instance<\/h3>\n
\n “””
\n The agent depends fully on the consumer to supply context.
\n It retains no data from previous interactions in native reminiscence.
\n “””
\n # Initializing with a system immediate
\n messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
\n
\n # Appending no matter historical past the consumer offered
\n if provided_history:
\n messages.lengthen(provided_history)
\n
\n # Appending the brand new immediate
\n messages.append({“function”: “person”, “content material”: immediate})
\n
\n # The LLM processes all the chain of messages
\n response = consumer.chat.completions.create(
\n mannequin=MODEL_ID,
\n messages=messages,
\n max_tokens=100
\n )
\n
\n return response.selections[0].message.content material.strip()<\/p>\n
\n
\nprint(“— Flip 1 —“)
\nprompt_1 = “Hello, my identify is Alice and I’m studying about API infrastructure.”
\nresponse_1 = stateless_agent(prompt_1)
\nprint(f”Agent: {response_1}”)
\n
\nprint(“n— Flip 2 (With out Shopper Context) —“)
\n# The agent fails right here as a result of it retained no reminiscence of Flip 1
\nprompt_2 = “What’s my identify and what am I studying about?”
\nresponse_2 = stateless_agent(prompt_2)
\nprint(f”Agent: {response_2}”)
\n
\nprint(“n— Flip 2 (With Shopper Context) —“)
\n# The frontend MUST inject the historical past into the payload for the agent to succeed
\nfrontend_payload = [
\n {“role”: “user”, “content”: prompt_1},
\n {“role”: “assistant”, “content”: response_1}
\n]
\nresponse_3 = stateless_agent(prompt_2, provided_history=frontend_payload)
\nprint(f”Agent: {response_3}”)<\/p>\n