AI brokers have modified rather a lot within the final two years. The primary might solely reply one query at a time. Then got here multi-turn chat, the place the mannequin might hold some context throughout a session. In the present day, we’ve got long-horizon brokers. Methods that plan many steps, cut up work between sub-agents, hold context throughout an extended process, and run instruments in a protected sandbox.
The NVIDIA AI-Q Blueprint is an open supply reference for this sort of agent. It’s constructed on LangChain Deep Brokers and the NVIDIA NeMo Agent Toolkit. You need to use it for fast cited solutions, or for longer analysis reviews with sources.
This submit reveals you find out how to deploy AI-Q 2.0 on Oracle Cloud Infrastructure (OCI) utilizing Terraform to create the OCI sources and Helm to put in the workloads on OKE. By the tip, you should have a working AI-Q endpoint in your individual OCI tenancy, and one command to take all of it down if you find yourself carried out.
Who that is for: Builders and platform engineers snug with Kubernetes, Terraform, and the shell, and who wish to run AI-Q on OCI moderately than on a laptop computer.
What you’ll study: How AI-Q’s multi-agent structure maps to OCI companies, plus the precise instructions to provision, deploy, and open the blueprint from begin to end.
Extra background on the multi-agent structure (similar to intent router, shallow analysis agent, deep agent, planning sub-agent, researcher sub-agent), is on the AI-Q product web page and the NeMo Agent Toolkit docs.
Stipulations
Be sure you have:
OCI tenancy entry with a compartment you’ll be able to deploy into, and sufficient service limits for:
OKE: One enhanced cluster and one node pool
Block Quantity: At the least 10 GB (dynamically provisioned by the OKE CSI driver for the in-cluster PostgreSQL)
Load Balancer: One versatile
Vault: One vault plus secrets and techniques
API keys:
NGC API key from construct.nvidia.com, format nvapi-… used each because the NVIDIA inference key and to authenticate to the NGC container registry (nvcr.io).
Tavily API key from tavily.com, format tvly-…
Native instruments: terraform 1.5 or later, kubectl 1.28 or later, helm 3.x or later, the oci CLI arrange along with your API signing key
Some fundamental data of Kubernetes, Helm charts, Terraform, and the shell. LangChain or NeMo Agent Toolkit expertise is sweet to have, however not required.
Structure overview
AI-Q makes use of a multi-agent design. An intent router reads every consumer question and sends it to the proper workflow.
The blueprint is constructed to be extensible. Each layer (fashions, instruments, RAG backends, sub-agents, evaluators) might be swapped by way of YAML config or by way of the NeMo Agent Toolkit plugin system. We’ll use that extensibility in Elements 2 and three of this sequence.
OCI deployment structure
The deployment makes use of Terraform for the OCI sources and Helm for the Kubernetes workloads. This provides a clear cut up between infrastructure and utility, and one terraform destroy is sufficient to take away every part later.
The Helm chart installs three workloads on OKE:
Backend (aiq-backend): A FastAPI-based agent server that runs the AI-Q workflow.
Frontend (aiq-frontend): A subsequent.js internet UI uncovered over NodePort 30080.
PostgreSQL (aiq-postgres): An in-cluster database for the job retailer, checkpoints, and summaries.
Deployment steps
cd ai-q/oke-samples/aiq-2.0
Whole time: round 20 to 25 minutes. The complete reference is in aiq-2.0/README.md.
Step 1. Configure Terraform variables
Copy the instance file and edit it along with your tenancy particulars:
cp terraform.tfvars.instance terraform.tfvars
At minimal, set these variables in terraform.tfvars:
tenancy_ocid, compartment_id, area (for instance us-chicago-1)
user_ocid, fingerprint, private_key_path (similar values as your ~/.oci/config)
db_admin_password, used to bootstrap the in-cluster PostgreSQL, saved in OCI Vault.
nvidia_api_key, your NVIDIA NGC key from construct.nvidia.com. Used for inference and to drag container photographs from nvcr.io.
tavily_api_key, your Tavily key from tavily.com, for internet search.
Step 2. Create the infrastructure
Initialize the suppliers, test the plan, and apply:
terraform plan
terraform apply
This takes about 10 to fifteen minutes. Terraform creates the VCN, OKE cluster, Load Balancer, and the Vault with the NGC and Tavily API keys encrypted at relaxation.
Examine: terraform output ought to present values for oke_cluster_id and lb_public_ip. If both is empty, run terraform apply once more – the apply is protected to repeat.
Seize the 2 values you’ll want within the subsequent step:
export LB_PUBLIC_IP=”$(terraform output -raw lb_public_ip)”
Step 3. Set up AI-Q from the NGC Helm chart
The chart and container photographs are printed on NGC, so there’s nothing to construct domestically. We level kubectl on the new OKE cluster, create the secrets and techniques the chart consumes, then helm pull and helm set up.
3a. Configure kubectl for the OKE cluster
oci ce cluster create-kubeconfig
–cluster-id “$OKE_CLUSTER_ID”
–file ~/.kube/config
–region us-ashburn-1
–token-version 2.0.0
–kube-endpoint PUBLIC_ENDPOINT
# sanity test. nodes ought to be prepared
kubectl get nodes
3b. Export the API keys
Reuse the identical NGC and Tavily keys you set in terraform.tfvars. The NGC key does double responsibility. It’s each the inference key and the nvcr.io pull credential.
export TAVILY_API_KEY=”tvly-…” # from tavily.com
export DB_USER_PASSWORD=””
3c. Create the namespace and secrets and techniques
# Software credentials (NVIDIA + Tavily inference, Postgres consumer)
kubectl create secret generic aiq-credentials -n ns-aiq
–from-literal=NVIDIA_API_KEY=”$NGC_API_KEY”
–from-literal=TAVILY_API_KEY=”$TAVILY_API_KEY”
–from-literal=DB_USER_NAME=”aiq”
–from-literal=DB_USER_PASSWORD=”$DB_USER_PASSWORD”
# Picture-pull secret for nvcr.io (NGC container registry)
kubectl create secret docker-registry ngc-secret -n ns-aiq
–docker-server=nvcr.io
–docker-username=’$oauthtoken’
–docker-password=”$NGC_API_KEY”
3d. Pull and set up the chart from NGC
helm pull https://helm.ngc.nvidia.com/nvidia/blueprint/charts/aiq2-web-2.0.0.tgz
–username=’$oauthtoken’
–password=”$NGC_API_KEY”
helm improve –install aiq aiq2-web-2.0.0.tgz
-n ns-aiq
–wait –timeout 10m
-f values-oci-ngc.yaml
The OCI overlay (values-oci-ngc.yaml) is deliberately tiny — it solely pins the frontend service to NodePort 30080 (the port the OCI Load Balancer health-checks) and names the ngc-secret image-pull secret. Picture repositories, the Postgres init SQL, and the dynamically provisioned 10 Gi Block Quantity PVC all come from the chart’s personal defaults.
Examine: kubectl get pods -n ns-aiq ought to present aiq-backend, aiq-frontend, and aiq-postgres pods in Operating state after 3 to five minutes.
Step 4. Open AI-Q
The LB IP is already in your shell from Step 2:
In the event you opened a brand new shell since then, re-export it from Terraform:
export LB_PUBLIC_IP=”$(terraform output -raw lb_public_ip)”
echo “http://$LB_PUBLIC_IP”
Open http:// in your browser. You must see the AI-Q frontend.
Attempt a easy query first, for instance, “What’s the NeMo Agent Toolkit?”, to verify the routing works. Then strive a deeper one, for instance, “Evaluate the highest three open-source deep-research brokers by benchmark rating and value”, to see the deep agent in motion.
Troubleshooting
terraform apply fails on OKE creation with a quota error. Examine the service limits in your compartment for “Cluster rely” and “Node rely”, and ask for extra quota if wanted.
Pods caught in ImagePullBackOff. Examine that the image-pull secret was created (kubectl get secret -n ns-aiq) and that your NGC_API_KEY was right if you ran the kubectl create secret docker-registry ngc-secret command in Step 3c. To rotate, delete the key and re-create it, then kubectl rollout restart deployment -n ns-aiq aiq-backend aiq-frontend.
postgres pod stays in Pending for greater than 2 minutes. The Block Quantity PVC didn’t get dynamically provisioned. Run kubectl describe pvc -n ns-aiq. Typical causes are the OKE CSI driver not working, the default StorageClass lacking, or inadequate Block Quantity quota. Examine the storage class with kubectl get sc and your compartment’s Block Quantity service restrict.
Load Balancer IP comes again as null. OCI can take a minute or two after Terraform to complete the LB. Run terraform refresh after which terraform output lb_public_ip once more.
Frontend masses however queries return 500. Take a look at kubectl logs -n ns-aiq deploy/aiq-backend. The most typical trigger is a mistaken or lacking NVIDIA_API_KEY or TAVILY_API_KEY within the aiq-credentials secret you created in Step 3c.
Be taught extra
You now have a working AI-Q 2.0 deployment on OCI, and one command (terraform destroy) to take away it cleanly if you find yourself carried out. Just a few issues to bear in mind as you go additional:
Price: The OKE node pool and the Load Balancer hold costing you whereas they run. Destroy the stack between experiments, or scale the node pool all the way down to zero.
Secrets and techniques: Terraform shops the NGC and Tavily keys in OCI Vault at provision time (for audit and catastrophe restoration), however the working pods learn them from the aiq-credentials Kubernetes secret you created in Step 3c. To rotate, delete and re-create that secret with the brand new values, then kubectl rollout restart deployment -n ns-aiq aiq-backend. Modifying terraform.tfvars alone received’t attain the pods.
Extensibility: The whole lot you simply deployed is pushed by YAML and by the NeMo Agent Toolkit plugin system. Swapping an LLM, including a sub-agent, or plugging a brand new RAG backend is a configuration change, not a rewrite.
Clone the AI-Q in OCI repo and share on the NVIDIA Developer Discussion board the answer you constructed and what downside you solved.
