You’ll be able to spin up a personal, OpenAI-compatible LLM endpoint on Hugging Face infrastructure with a single command — no servers to provision, no Kubernetes, pay-per-second. As soon as it is up, you possibly can question it out of your laptop computer, a pocket book, or wherever else.
It is the quickest technique to rise up a mannequin for checks, evals, or batch technology. (For those who’re after a managed, production-ready service as an alternative, that is what Inference Endpoints are for — extra on when to select which on the finish.)
Here is the entire thing finish to finish.
Conditions
A cost methodology or a constructive pay as you go credit score stability (Jobs is billed per‑minute by {hardware} utilization).
huggingface_hub >= 1.20.0: pip set up -U “huggingface_hub>=1.20.0”.
Logged in regionally: hf auth login.
Launch the server
hf jobs run is docker run for HF infrastructure. We use the official vllm/vllm-openai picture, ask for a GPU with –flavor, and expose vLLM’s port with –expose:
hf jobs run –flavor a10g-large –expose 8000 —timeout 2h
vllm/vllm-openai:newest
vllm serve Qwen/Qwen3-4B –host 0.0.0.0 –port 8000
–expose 8000 routes the container’s port by way of HF’s public jobs proxy (see the Serve Fashions information for the total reference). The command prints the URL your server is reachable at:
✓ Job began
id: 6a381ca1953ed90bfb947332
url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332
Trace: Uncovered ports are reachable at (requires an HF token with learn entry to the job):
https://6a381ca1953ed90bfb947332–8000.hf.jobs
6a381ca1953ed90bfb947332 is your job ID. Maintain observe of it, we’ll want it. We’ll use as a placeholder for it in the remainder of the put up.
Give it a few minutes to obtain weights and boot. When the logs present Software startup full, you are dwell.
Question it from wherever
vLLM speaks the OpenAI API, and each request simply wants your HF token as a bearer token. The quickest technique to hit it’s curl:
curl https://–8000.hf.jobs/v1/chat/completions
-H “Authorization: Bearer $(hf auth token)“
-H “Content material-Sort: utility/json”
-d ‘{
“mannequin”: “Qwen/Qwen3-4B”,
“messages”: [{“role”: “user”, “content”: “Hello!”}],
“chat_template_kwargs”: {“enable_thinking”: false}
}’
which returns the same old OpenAI-style JSON, with decisions[0].message.content material holding “Hey! How can I help you at present? 😊”.
Or, from Python, level the OpenAI consumer on the uncovered URL and cross the token because the API key:
from huggingface_hub import get_token
from openai import OpenAI
consumer = OpenAI(
base_url=“https://–8000.hf.jobs/v1”,
api_key=get_token(),
)
resp = consumer.chat.completions.create(
mannequin=“Qwen/Qwen3-4B”,
messages=[{“role”: “user”, “content”: “Hello!”}],
extra_body={“chat_template_kwargs”: {“enable_thinking”: False}},
)
print(resp.decisions[0].message.content material)
Hey! How can I help you at present? 😊
Fast well being test earlier than you begin: curl https://–8000.hf.jobs/v1/fashions -H “Authorization: Bearer $(hf auth token)” ought to listing the mannequin.
🔐 The endpoint is gated, not public. Each request should carry an HF token with learn entry to the job’s namespace. A plain browser go to will likely be rejected. In impact, the roles proxy is your API gate: entry is scoped to you (and your org). That is tremendous for personal use, however deal with the URL accordingly: do not share it anticipating it to be open, and do not paste your token into untrusted locations. For those who want finer-grained or public entry, put a correct gateway in entrance as an alternative. Or see HF Jobs or Inference Endpoints? under.
Clear up
Jobs are billed per second, so cease the server while you’re performed:
hf jobs cancel
The –timeout you set is a security web (it will auto-stop), however cancelling explicitly is cheaper. An a10g-large runs at $1.50/hour — test hf jobs {hardware} for the total value listing and choose the smallest taste that matches your mannequin.
Going additional: greater fashions
The identical command scales to a lot bigger fashions — choose a beefier –flavor and inform vLLM to shard the mannequin throughout the GPUs with –tensor-parallel-size. For instance, the 122B Qwen3.5 mixture-of-experts mannequin on 2× H200:
hf jobs run –flavor h200x2 –expose 8000 —timeout 2h
vllm/vllm-openai:newest
vllm serve Qwen/Qwen3.5-122B-A10B
–host 0.0.0.0 –port 8000 –tensor-parallel-size 2
–max-model-len 32768 –max-num-seqs 256
–tensor-parallel-size ought to match the variety of GPUs within the taste (h200x2 → 2, h200x8 → 8). Run hf jobs {hardware} to see what’s accessible and provides greater fashions an extended –timeout, since they take longer to obtain and cargo. For giant fashions, H200 flavors are often the most effective worth.
The –max-model-len 32768 –max-num-seqs 256 flags are particular to this mannequin: Qwen3.5-122B is a hybrid Mamba/consideration structure with a 256K-token default context, which does not go away sufficient reminiscence for vLLM’s default batch settings. Capping the context size and concurrent-sequence rely retains it throughout the GPUs’ reminiscence. If a mannequin fails to start out with an out-of-memory or cache-block error, dialing these two down is the very first thing to strive. Every little thing else (the uncovered URL, the OpenAI consumer, the token auth) stays precisely the identical.
Going additional: Chat with it in a UI
Want a chat window over curl? Just a few strains of Gradio level on the identical endpoint. Add –reasoning-parser deepseek_r1 to the vllm serve command so Qwen3’s pondering comes again as a separate area (not needed, however useful), then run this code regionally (you may simply want the job ID):
import gradio as gr
from gradio import ChatMessage
from huggingface_hub import get_token
from openai import OpenAI
consumer = OpenAI(base_url=“https://–8000.hf.jobs/v1”, api_key=get_token())
def chat(message, historical past):
messages = [{“role”: m[“role”], “content material”: m[“content”]} for m in historical past if not m.get(“metadata”)]
messages.append({“position”: “consumer”, “content material”: message})
stream = consumer.chat.completions.create(mannequin=“Qwen/Qwen3-4B”, messages=messages, stream=True)
pondering, reply = “”, “”
for chunk in stream:
delta = chunk.decisions[0].delta
pondering += delta.model_extra.get(“reasoning”, “”)
reply += delta.content material or “”
out = []
if pondering.strip():
standing = “performed” if reply.strip() else “pending”
out.append(ChatMessage(position=“assistant”, content material=pondering, metadata={“title”: “💭 Considering”, “standing”: standing}))
if reply.strip():
out.append(ChatMessage(position=“assistant”, content material=reply))
yield out
gr.ChatInterface(chat).launch()
Run it, open http://127.0.0.1:7860, and chat — reasoning streams into the collapsible panel, the reply under.
Going additional: SSH into the working server
Have to debug a startup failure, watch GPU reminiscence, or tail logs interactively? You’ll be able to open a shell straight into the working job. Launch it with –ssh and ensure your public key’s registered at huggingface.co/settings/keys:
hf jobs run –flavor a10g-large –expose 8000 —timeout 2h –ssh
vllm/vllm-openai:newest
vllm serve Qwen/Qwen3-4B –host 0.0.0.0 –port 8000
then join with the job ID:
hf jobs ssh
You are now contained in the container, the place you possibly can run nvidia-smi, examine the method, or poke on the mannequin immediately — which makes debugging and monitoring a lot simpler than studying logs from the skin. SSH help requires huggingface_hub >= 1.20.0.
Going additional: Use it as a coding-agent backend with Pi
The identical endpoint can again a terminal coding agent. Pi is a provider-agnostic agent harness. Level it on the job and also you get a Learn/Write/Edit/Bash agent working by yourself self-hosted mannequin.
One factor to arrange first: brokers drive the mannequin by way of instrument calls, and vLLM solely accepts these if the server is launched with instrument calling enabled. So relaunch with –enable-auto-tool-choice and a –tool-call-parser matching the mannequin household (hermes for Qwen3). Brokers additionally profit from a stronger mannequin, so this can be a good place to herald the larger one:
hf jobs run –flavor h200x2 –expose 8000 —timeout 2h
vllm/vllm-openai:newest
vllm serve Qwen/Qwen3.5-122B-A10B
–host 0.0.0.0 –port 8000 –tensor-parallel-size 2
–max-model-len 32768 –max-num-seqs 256
–reasoning-parser deepseek_r1
–enable-auto-tool-choice –tool-call-parser hermes
Then add the job as a customized supplier in ~/.pi/agent/fashions.json:
{
“suppliers”: {
“hf-jobs”: {
“baseUrl”: “https://–8000.hf.jobs/v1”,
“api”: “openai-completions”,
“apiKey”: “!hf auth token”,
“fashions”: [
{ “id”: “Qwen/Qwen3.5-122B-A10B” }
]
}
}
}
Then launch the agent towards it:
pi
The mannequin you spun up a few instructions in the past, now driving an interactive coding agent in your terminal.
HF Jobs or Inference Endpoints?
HF Jobs is not the one technique to serve a mannequin on Hugging Face. Inference Endpoints are our managed product for a similar job, and which one suits will depend on what you are after.
Attain for HF Jobs while you need most flexibility and management: it is simply docker run on HF infrastructure, so that you choose the picture, the precise vllm serve flags, and the {hardware}, and also you pay per second for so long as the job runs. That makes it an ideal match for experiments, one-off evals, batch technology, or kicking the tires on a mannequin earlier than committing to something.
Attain for Inference Endpoints while you need one thing extra production-ready. They add the operational niceties a long-lived service wants: finer-grained entry management (an endpoint may be public, protected, or non-public), and scale-to-zero, so you are not billed during times of inactivity. For those who’re standing up a sturdy endpoint relatively than working a job, that is the instrument to seize.
Additional studying
This put up sticks to vLLM, however the identical expose-a-port sample works with any OpenAI-compatible server. To serve GGUFs with llama.cpp or run SGLang as an alternative, see the Serve Fashions on Jobs information, which walks by way of these backends.
