Immediately, we’re asserting the supply of the Gemma 4 household on Amazon Bedrock. Constructed by Google DeepMind and launched underneath the Apache 2.0 license, Gemma 4 is a household of open-weight fashions designed with a give attention to intelligence-per-parameter throughout a broad vary of deployment eventualities. The household consists of three instruction-tuned variants: Gemma 4 31B, Gemma 4 26B-A4B, and Gemma 4 E2B. These cowl dense and mixture-of-experts (MoE) architectures, the place solely a fraction of the mannequin’s parameters activate per request. The variants supply built-in reasoning, native operate calling, and multimodal enter throughout textual content and picture.
Impartial benchmarks replicate Gemma 4’s intelligence-per-parameter focus: Synthetic Evaluation stories an Intelligence Index of 39 for Gemma 4 31B, properly above the median of 15 within the 4B–40B open-weights class.
Organizations adopting open-weight basis fashions (FMs) for manufacturing face a relentless trade-off: entry the main fashions, however with out compromising on information safety, regulatory alignment, or operational management. Amazon Bedrock removes that trade-off. It provides you main open-weight FMs via a completely managed service, with inference operating fully on infrastructure operated by AWS and the safety and privateness controls you anticipate from Amazon Bedrock.
On this publish, we stroll via how one can get began with Gemma 4 fashions on Amazon Bedrock. We cowl the capabilities supported by these fashions, the service tiers obtainable, how on-demand inference scales to deal with your workloads, and the totally different APIs you should use to entry them. With these fashions, you’ll be able to construct multimodal brokers, light-weight purposes, doc understanding pipelines, and software program engineering workflows on Amazon Bedrock. Your prompts and completions usually are not used to coach any fashions, and your content material just isn’t shared with third events.
Key capabilities of Gemma 4
The Gemma 4 household on Amazon Bedrock spans a 2.3B-effective-parameter compact mannequin as much as a 30.7B-parameter dense mannequin, supplying you with a selection of variants to match totally different value and latency profiles. All variants assist a built-in reasoning mode, native operate calling for agentic workflows, multimodal enter that mixes textual content and picture, and out-of-the-box assist for over 35 languages with pre-training throughout 140+. As a result of the fashions are open-weight, you’ll be able to independently consider the mannequin structure and coaching methodology, benchmark by yourself workloads, and fine-tune on proprietary information when customization is required. You’ll be able to entry the fashions via a completely managed AWS service with out provisioning infrastructure, internet hosting mannequin weights, or working inference stacks your self.anno
For the newest checklist of supported fashions, consult with the Amazon Bedrock mannequin catalog.
The Gemma 4 household on Amazon Bedrock at a look
The household consists of three instruction-tuned variants optimized for various value and latency profiles. The next desk summarizes the important thing specs for every mannequin on Amazon Bedrock:
Gemma 4 31B
Gemma 4 26B-A4B
Gemma 4 E2B
Mannequin ID
google.gemma-4-31b
google.gemma-4-26b-a4b
google.gemma-4-e2b
Structure
Dense
Combination-of-Consultants
Dense (PLE)
Whole / Lively parameters
30.7B
25.2B / 3.8B lively
5.1B complete / 2.3B efficient
Context window
256K tokens
256K tokens
128K tokens
Modalities
Textual content, picture
Textual content, picture
Textual content, picture
Reasoning mode
Sure
Sure
Sure
Perform calling
Native
Native
Native
Service tiers
Customary, Precedence, Flex
Customary, Precedence, Flex
Customary, Precedence, Flex
Selecting a variant
Choose the variant that greatest matches your workload’s efficiency and price necessities. The next desk supplies steering on which mannequin to decide on primarily based in your use case:
In case your workload is…
Select
Why
Reasoning-heavy or coding-heavy with a single dense mannequin
Gemma 4 31B
Largest dense variant within the household; sturdy reasoning and coding efficiency with a 256K context window.
Price-sensitive at excessive throughput, with data breadth necessities
Gemma 4 26B-A4B
MoE design means inference value and latency nearer to a 4B dense mannequin whereas retaining the data capability of a a lot bigger one.
Latency-sensitive, on-device-style, or multimodal classification
Gemma 4 E2B
Smallest, quickest variant; suited to lowest-cost or fastest-response multimodal workloads. Set reasoning_effort=excessive for this variant (see Allow reasoning mode).
Throughout the household, Gemma 4 fashions share a standard interface: system prompts, structured device calling, picture enter, and a considering mode that may be toggled per request. You’ll be able to develop an utility towards the API floor as soon as and change between variants primarily based on the price and latency profile that matches the workload.
Structure highlights
All Gemma 4 variants use a hybrid consideration design that interleaves native and international consideration, supporting lengthy contexts as much as 256K tokens on 31B and 26B-A4B whereas retaining a small reminiscence footprint. The 26B-A4B variant is a mixture-of-experts mannequin: 25.2B complete parameters however solely 3.8B lively per token, giving roughly 4B-class value and latency with the data capability of a bigger mannequin. The E2B variant makes use of Per-Layer Embeddings (PLE) to maintain its efficient parameter depend (2.3B of 5.1B complete) small, decreasing reminiscence and compute value. For structure particulars, consult with the Gemma 4 mannequin card.
Word: Use the bedrock-mantle endpoint for Gemma 4 fashions.
Reasoning mode
Gemma 4 features a built-in reasoning mode. When enabled, the mannequin emits its inner thought course of earlier than producing the ultimate reply. On the bedrock-mantle endpoint, you allow reasoning via the Responses API reasoning parameter, and the thought course of is returned as a separate reasoning merchandise alongside the ultimate reply (see Allow reasoning mode within the walkthrough).
In multi-turn conversations, ship again solely the ultimate solutions from earlier turns, not their reasoning gadgets. Replaying prior reasoning again to the mannequin can degrade its responses. You’ll be able to nonetheless preserve the reasoning in your personal logs or audit path. Strip it from the historical past you ship on the subsequent flip.
Accessing Gemma 4 fashions on Amazon Bedrock
You entry Gemma 4 fashions on Amazon Bedrock via the bedrock-mantle endpoint, the OpenAI-compatible API purpose-built for the next-generation inference engine for Amazon Bedrock. Its endpoint URL is https://bedrock-mantle.{area}.api.aws/openai/v1, and it exposes the Chat Completions and Responses APIs.
It’s useful to maintain the endpoint and the engine distinct: the engine is the underlying serving infrastructure—designed with Mannequin Deployment Account isolation and 0 operator entry—and the endpoint is the HTTPS API floor you name. The bedrock-mantle endpoint is the general public API for that next-generation inference engine and exposes its full characteristic set. For a deeper have a look at the underlying inference engine, consult with Exploring the zero operator entry design of Amazon Bedrock’s next-generation inference engine.
The bedrock-mantle endpoint makes use of the identical interface because the OpenAI Python and TypeScript SDKs, so groups already on these SDKs can change to Gemma 4 fashions on Amazon Bedrock by updating solely the bottom URL and mannequin ID. It additionally helps Amazon Bedrock API keys, initiatives, and client-side device calling.
Get began with Gemma 4 household fashions on Amazon Bedrock
Full the next steps to start out utilizing Gemma 4 on Amazon Bedrock.
Stipulations
To make use of Gemma 4 fashions, you want an AWS account with permissions to run inference on the bedrock-mantle endpoint. The best strategy to grant these is to connect the AWS managed coverage AmazonBedrockMantleInferenceAccess to your AWS Identification and Entry Administration (IAM) principal. It grants learn and inference-creation entry on Mantle—the permissions the examples on this publish want. These embody bedrock-mantle:CreateInference (which authorizes the Chat Completions and Responses inference calls) and bedrock-mantle:CallWithBearerToken (which authorizes calling the endpoint with an Amazon Bedrock API key). For particulars on creating and managing API keys, consult with Amazon Bedrock API keys.
In case you additionally must handle initiatives, fine-tuning, customized fashions, and extra, connect AmazonBedrockMantleFullAccess as a substitute, which grants the total bedrock-mantle motion set.
Console playground
The Amazon Bedrock console features a chat/textual content playground the place you’ll be able to shortly check fashions with out writing any code. Comply with these steps to load a Gemma 4 mannequin and put together it for interplay:
Navigate to the Amazon Bedrock console.
From the left menu, select Take a look at playgrounds.
Select the Chat/Textual content playground.
Select Choose mannequin within the upper-left nook of the playground.
From the class checklist, select Google.
Select a Gemma 4 mannequin.
Select Apply to load the mannequin.
Confirm that the mannequin loaded efficiently: the mannequin identify seems within the playground header, and the chat interface is prepared for enter.
To exhibit Gemma 4 31B’s reasoning and code technology capabilities, attempt the next immediate within the playground: “Design a Python microservice that exposes a REST API for managing a job queue. Embody error dealing with, enter validation, and write unit exams. Clarify your design choices.”
Name the bedrock-mantle endpoint from the OpenAI SDK
The next instance makes use of the OpenAI Python SDK as a consumer library to name the bedrock-mantle endpoint. When utilizing the OpenAI SDK, you want an Amazon Bedrock API key. For manufacturing workloads, use short-term API keys. They expire routinely (most 12 hours) and inherit the permissions of the IAM function that generated them. If you’re already utilizing native AWS credentials and don’t have an API key, the aws-bedrock-token-generator bundle generates a short-term bearer token from these credentials.
If you’re migrating an current utility that already makes use of the OpenAI SDK format with a unique mannequin, the migration sometimes requires updating solely the bottom URL and the mannequin ID. To manage permissions for producing and utilizing API keys, consult with Management permissions for producing and utilizing Amazon Bedrock API keys.
Word: The code examples on this publish learn the API key from an atmosphere variable for demonstration solely. In manufacturing, retailer and retrieve credentials from a managed secrets and techniques service akin to AWS Secrets and techniques Supervisor or AWS Techniques Supervisor Parameter Retailer moderately than atmosphere variables.
Multimodal enter (picture)
As a result of Gemma 4 helps picture enter throughout all variants, the identical Chat Completions API works for imaginative and prescient duties. The bedrock-mantle endpoint accepts photographs as inline base64-encoded information URLs or as Amazon Easy Storage Service (Amazon S3) URLs (s3://); arbitrary public https:// picture URLs usually are not supported. The next instance reads an area picture file, encodes it as a base64 information URL, and consists of it within the message content material alongside the textual content immediate:
Alternatively, you’ll be able to reference a picture saved in Amazon S3 by passing an s3:// URL rather than the info URL (for instance, {“url”: “s3://my-bucket/chart.png”}). This avoids inlining massive photographs within the request physique. For greatest outcomes, place picture content material earlier than the textual content within the immediate, which matches Google DeepMind’s really helpful ordering for Gemma 4 multimodal inputs.
Streaming responses
For chat and agent use circumstances the place you need to floor tokens to the person as they’re generated, set stream=True. The response turns into an iterator of incremental delta occasions:
Streaming makes use of the identical bedrock-mantle:CreateInference permission as a non-streaming name—the IAM coverage proven earlier already covers it.
Software calling
Gemma 4 helps native operate calling for agentic workflows. The next instance reveals an entire tool-calling loop: defining a device, receiving a device name from the mannequin, executing the operate, and passing the consequence again.
Responses API
Along with Chat Completions, the bedrock-mantle endpoint helps the OpenAI Responses API, which makes use of a single enter discipline as a substitute of a messages checklist and returns a top-level output_text for the generated response. It is a good match for single-turn technology and for purposes already constructed towards the Responses interface:
The Responses API makes use of the identical Amazon Bedrock API key and base URL because the Chat Completions examples; solely the tactic and response form differ. Use Chat Completions if you want multi-turn message historical past or client-side tool-calling loops, and the Responses API for streamlined single-turn technology or if you need the mannequin’s reasoning (lined subsequent).
Allow reasoning mode
Gemma 4 can produce an specific thought course of earlier than its last reply, which is helpful for advanced multi-step duties at the price of extra latency and token utilization. On the bedrock-mantle endpoint, you allow this via the Responses API by setting the reasoning parameter. The mannequin returns the thought course of as a separate reasoning merchandise within the output, alongside the ultimate reply in output_text:
The hassle worth (low, medium, or excessive) controls how a lot the mannequin causes earlier than answering. For multi-turn conversations, move solely the ultimate solutions again as historical past—exclude prior reasoning gadgets from the enter on the subsequent flip.
For Gemma 4 E2B, we advocate setting reasoning_effort to excessive. The smallest variant tends to motive extensively by default, and a excessive reasoning effort retains that considering within the devoted reasoning channel—enhancing output high quality and stopping reasoning textual content from leaking into the ultimate reply.
Really useful sampling parameters
On the bedrock-mantle endpoint, sampling is managed by temperature and top_p. For Gemma 4, we advocate temperature=1.0 and top_p=0.95, which work properly for each reasoning and non-reasoning modes.
Clear up
On-demand inference incurs fees solely when invoked, so there isn’t a infrastructure to tear down. To keep away from unintended fees:
In case you generated short-term Amazon Bedrock API keys for testing, the keys expire routinely (most 12 hours). To revoke earlier, delete the API key within the console. Deleting an API key instantly revokes entry for all purposes utilizing that key, so ensure no lively purposes rely upon it earlier than you delete it.
In case you opted in to the Precedence tier for testing, change again to Customary for non-latency-sensitive visitors by eradicating the service_tier parameter out of your invocations.
For pricing particulars by tier and mannequin, consult with Amazon Bedrock pricing.
Service tiers
Amazon Bedrock gives a number of service tiers to match totally different workload necessities:
Tier
Greatest for
Traits
Precedence
Mission-critical, customer-facing workflows that want the quickest response instances
As much as 25% higher output tokens per second (OTPS) latency in comparison with Customary. Prioritized forward of Customary and Flex requests. Premium over commonplace on-demand pricing. No upfront reservation or dedication.
Customary
On a regular basis AI duties akin to content material technology, textual content evaluation, and routine doc processing
Constant efficiency at commonplace on-demand pricing. Default tier when no tier is specified. No dedication.
Flex
Workloads that may tolerate longer processing instances, akin to mannequin evaluations, content material summarization, and agentic workflows
Discounted pricing relative to Customary. Larger latency, particularly throughout peak visitors, as a result of Flex requests are processed after Customary. No dedication.
For the newest tier availability per mannequin, consult with the Amazon Bedrock mannequin catalog.
Scaling on-demand inference
Whenever you invoke Gemma 4 fashions on Amazon Bedrock, requests use on-demand inference (Customary tier) by default, the place you pay per token with out reserving capability. On-demand throughput is shared and allotted per AWS Area, so during times of excessive regional demand a request is perhaps briefly queued or throttled. Designing for that is essential for purposes that must scale reliably in manufacturing.
On the bedrock-mantle endpoint, there isn’t a requests-per-minute (RPM) quota. Inference is ruled by per-model, per-Area token-based quotas—separate input-tokens-per-minute and output-tokens-per-minute limits. Gemma 4 and different open fashions don’t presently have per-account token quotas printed within the Service Quotas console. Their throughput is ruled by inner service capability, so use retry logic with exponential backoff to deal with transient throttling. Cached enter tokens learn via immediate caching don’t depend towards the input-token quota. For particulars, see Quotas for the bedrock-mantle endpoint.
Amazon Bedrock surfaces two HTTP error codes that point out {that a} request can’t be served:
Error
What it means
What to do
HTTP 429
A token-per-minute quota for the mannequin has been exceeded.
Scale back the submission charge and retry with exponential backoff; request a quota enhance via AWS Help when you constantly hit the restrict.
HTTP 503
Regional capability for the mannequin is underneath strain.
Retry with exponential backoff for infrequent responses; scale back the submission charge for sustained ones.
The excellence between these two responses issues when troubleshooting manufacturing visitors. A 429 signifies {that a} token-per-minute quota has been exceeded and is greatest addressed by lowering your submission charge or requesting a quota enhance via AWS Help. A 503 signifies that regional capability for the mannequin is underneath strain. Following the steering within the subsequent part to ramp steadily permits purposes to proceed working on on-demand inference with out disruption.
Deal with one-off 503 responses
Some on-demand inference requests may see occasional 503 responses when the mannequin is in excessive demand. The really helpful strategy to deal with them is exponential backoff with jitter and a bounded retry depend. The OpenAI SDK helps this via its built-in max_retries setting, which retries failed requests with exponential backoff. The next instance configures the consumer used with the bedrock-mantle endpoint:
Backoff and retry is the really helpful strategy to deal with these transient responses on the bedrock-mantle endpoint.
If 503 responses develop into sustained, retries alone received’t resolve the problem as a result of the efficient request charge is exceeding obtainable capability for the mannequin. In that case, contemplate routing latency-sensitive visitors to the Precedence tier, which receives preferential processing forward of Customary and Flex requests during times of excessive demand.
Deal with steep visitors ramps
When utilizing Customary tier on-demand inference, your utility’s incoming visitors ought to align with how the mannequin’s regional capability scales. Sudden, massive jumps in request charge usually tend to set off 503s than gradual will increase that the system can accommodate. Everytime you enhance the request charge towards Gemma 4 fashions, scale up in measured increments moderately than stepping straight to a brand new goal quantity. The really helpful ramp process is as follows:
Begin at your goal request charge.
In case you obtain 503 responses, scale back the speed by 50% and proceed lowering till requests are succeeding constantly.
Maintain at that regular state for quarter-hour.
Improve the speed by 50% and maintain for an additional quarter-hour.
Repeat till you attain your goal quantity.
As a labored instance: in case your goal is 2,000 RPM and also you encounter 503s, scale back to 1,000 RPM, then to 500 if errors persist. As soon as 500 RPM is regular for quarter-hour, scale to 750, then 1,125, and so forth. Skipping the 15-minute maintain turns every step-up right into a contemporary load check.
Select the Precedence tier for latency-sensitive workloads
Past reactive use throughout sustained 503s, the Precedence tier is usually a helpful lever to scale back occurrences of 503 whereas persevering with to make use of on-demand inference. Precedence delivers as much as 25% higher output tokens per second in comparison with Customary, and there’s no upfront reservation or dedication. Functions decide in by setting the service_tier parameter to precedence on every invocation, and tiers might be blended throughout the identical utility. Buyer-facing prompts, real-time brokers, and different person interactions the place response time instantly impacts expertise are good candidates for Precedence. For background and batch-style work, Customary or Flex is often the precise selection and avoids paying the Precedence premium on requests that wouldn’t profit from it.
Extra greatest practices for manufacturing scale
A handful of practices assist preserve inference workloads operating easily at scale:
Unfold massive workloads throughout a number of minutes, moderately than firing them in tight bursts, to scale back strain on regional capability.
When migrating manufacturing visitors to a brand new Gemma 4 mannequin model, use characteristic flags to ramp the proportion of visitors steadily as a substitute of chopping over .
Route asynchronous work—mannequin evaluations, content material summarization, agentic backfills—to the Flex tier for improved price-performance.
For workloads with out information residency necessities, distribute throughout a number of Areas to enhance resilience throughout regional demand spikes.
For workloads anticipated to develop, plan headroom for 2 to 3 instances the anticipated peak as a buffer for visitors surges.
For full steering, consult with Scaling and throughput greatest practices within the Amazon Bedrock Person Information.
Scale back latency with implicit immediate caching
Gemma 4 fashions on Amazon Bedrock assist implicit immediate caching, which is routinely enabled. Consecutive requests that share a standard immediate prefix may lead to a cache hit, permitting the mannequin to reuse the cached inner state as a substitute of recomputing it. Cache hits scale back inference latency on the matching tokens, with no code adjustments or cache markers required.
Implicit immediate caching is accessible throughout all on-demand service tiers (Customary, Precedence, and Flex), so purposes can reap the benefits of it no matter how their visitors is routed. Cache hits usually are not at all times current on each request, however they’re widespread in workloads with secure prefixes akin to multi-turn brokers, retrieval-augmented technology, and long-context evaluation, the place system prompts, device definitions, or supply paperwork are reused throughout requests. In case you place static content material on the entrance of the immediate and dynamic content material on the finish, purposes can reap the benefits of caching when it happens.
Pricing and availability
At launch, Gemma 4 fashions can be found in 4 AWS Areas: US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Frankfurt). For the newest checklist, consult with the Amazon Bedrock mannequin catalog.
Pricing is per token and varies by mannequin and repair tier. For present charges, consult with Amazon Bedrock pricing.
Conclusion
On this publish, we explored the Gemma 4 household on Amazon Bedrock—open-weight fashions from Google DeepMind protecting dense and mixture-of-experts architectures, with built-in reasoning, native operate calling, and multimodal enter throughout textual content and picture. We walked via the bedrock-mantle endpoint and the safety structure that runs beneath it (Mannequin Deployment Account isolation with zero operator entry), and confirmed how one can get began utilizing the OpenAI Python SDK towards bedrock-mantle. We lined service tiers (Customary, Precedence, and Flex), the shared-throughput mannequin for on-demand inference, and how one can deal with 503 responses and ramp visitors in manufacturing.
To get began:
Open the Amazon Bedrock console and check out Gemma 4 within the Chat/Textual content playground.
Run the bedrock-mantle Python pattern on this publish towards your personal information.
Consider Gemma 4 31B, 26B-A4B, and E2B in your workloads to decide on the variant that matches your value and latency profile.
For manufacturing deployment, evaluation the Scaling and throughput greatest practices and contemplate the Precedence tier for latency-sensitive visitors.
For extra data, consult with the next sources:
Concerning the authors
Aris Tsakpinis
Aris is a Senior Specialist Options Architect for Generative AI specializing in open-weight fashions on Amazon Bedrock and the broader generative AI open-source ecosystem. Alongside his skilled function, he’s pursuing a PhD in Machine Studying Engineering on the College of Regensburg, the place his analysis focuses on utilized pure language processing in scientific domains.
Alex Thewsey
Alex is a Generative AI Specialist Options Architect at AWS, primarily based in Singapore. Alex helps clients throughout Southeast Asia to design and implement options with ML and Generative AI. He additionally enjoys karting, working with open supply initiatives, and making an attempt to maintain up with new ML analysis.
Saurabh Trikande
Saurabh is a Senior Product Supervisor for Amazon Bedrock and Amazon SageMaker Inference. He’s captivated with working with clients and companions, motivated by the aim of democratizing AI. He focuses on core challenges associated to deploying advanced AI purposes, inference with multi-tenant fashions, value optimizations, and making the deployment of generative AI fashions extra accessible. In his spare time, Saurabh enjoys mountaineering, studying about revolutionary applied sciences, following TechCrunch, and spending time along with his household.
Pradyun Ramadorai
Pradyun is a Principal Engineer at Amazon Bedrock. He focuses on core challenges associated to Generative AI purposes, scalable LLM inference and optimizations.
Zohreh Norouzi
Zohreh is a Safety Options Architect at Amazon Net Companies. She helps clients make good safety selections and speed up their journey to the AWS Cloud. She has been actively concerned in generative AI safety initiatives throughout APJ, utilizing her experience to assist clients construct safe generative AI options at scale.
