Kimi K3, and what we will nonetheless be taught from the pelican benchmark
sixteenth July 2026
Chinese language AI lab Moonshot AI introduced Kimi K3 this morning, describing it as their “most succesful mannequin so far, with 2.8 trillion parameters”. It’s presently obtainable through their web site and API, however an open weight launch is promised “by July 27, 2026”.
Moonshot are calling this the primary “open 3T-class mannequin” (I suppose they’re rounding 2.8 trillion as much as 3 trillion), taking the crown from DeepSeek’s 1.6T v4 Professional. Their self-reported benchmarks have K3 principally beating Claude Opus 4.8 max and GPT-5.5 excessive, whereas dropping out to Claude Fable 5 and GPT-5.6 Sol.
Just a few highlights from the Synthetic Evaluation report on the mannequin:
“On our personal long-horizon information work analysis, Kimi K3 reaches an total Elo of 1547, +732 factors from Kimi K2.6 and behind solely Claude Fable 5.”
“Value per job ($0.94) is much like GPT-5.6 Sol ($1.04), ~1/2 the value of Opus 4.8 ($1.80) and better than open weights friends”
“Kimi K3’s token utilization on the Synthetic Evaluation Intelligence Index decreased considerably, utilizing 21% fewer output tokens than K2.6.”
The mannequin can also be now the main mannequin on Area.ai’s Frontend Code enviornment, surpassing even Claude Fable 5.
The brand new mannequin is notable for the pricing: $3/million enter tokens and $15/million output tokens, placing it on the identical degree as Anthropic’s Claude Sonnet sequence and making it the most costly mannequin launched by a Chinese language AI lab so far. This can be a important improve on their earlier fashions akin to Kimi K2.6 at $0.95/$4. 2.8 trillion parameters can also be greater than twice the scale of that 1T mannequin.
However how does it pelican?
I used OpenRouter (to keep away from signing up for a Moonshot API key) with the llm-openrouter plugin to generate an SVG of a pelican driving a bicycle:
llm -m openrouter/moonshotai/kimi-k3 ‘Generate an SVG of a pelican driving a bicycle’
Right here’s the transcript. It appears to be like like this:
That pelican took 95 enter tokens and 16,658 output tokens (13,241 had been reasoning tokens), for a complete value of 25 cents!
Since K3 accepts picture enter I ran it towards that rendered SVG above (with my alt textual content immediate) and obtained again (for 0.6 cents):
Cartoon illustration of a white pelican sporting a crimson scarf, driving a crimson bicycle alongside a grey highway with white dashed traces; the pelican has a big orange beak and webbed orange ft pedaling, with white movement traces behind it; the background reveals a light-weight blue sky with white clouds, a yellow solar, two small black birds in flight, and inexperienced grass with tiny white flowers within the foreground
What can we be taught from the pelican?
My Generate an SVG of a pelican driving a bicycle take a look at is 21 months previous now. It was by no means a very nice benchmark. It began out as a joke on how absurdly troublesome it’s to match these fashions, however then for the primary 12 months it turned out to have a stunning correlation to how good the fashions truly had been.
That connection has been principally severed now. The GPT-5.6 and Claude Fable 5 pelicans are outclassed by GLM-5.2, and far as I like GLM I don’t suppose that’s a Fable-class mannequin.
(I’m nonetheless not satisfied that labs are coaching for the benchmark—in the event that they had been, I’d anticipate significantly better outcomes. There’s an opportunity that Gemini has optimized for any mixture of an animal on a automobile although!)
The most important limitation of the pelican is that it doesn’t contact in any respect on the factor that issues most for at this time’s mannequin: agentic device calling and the flexibility to function instruments reliably as conversations develop in size.
So don’t go utilizing pelicans to match fashions!
All of that mentioned, I nonetheless get an honest quantity of worth out of operating the benchmark myself.
Firstly, it’s a forcing perform for truly attempting the mannequin. If I present you a pelican, which means I’ve managed to run a immediate by it. If the mannequin has an official API I’ll use that, if it’s open weight (and sufficiently small to suit a 128GB M5 MacBook Professional) I’ll strive operating it by myself machine, often through llama.cpp or LM Studio or Ollama. I’ll continuously use OpenRouter since that often supplies a proxy to an official API with out me needing a brand new API key.
Most of my pelicans are generated utilizing my LLM CLI device, which helps encourage me to make sure the newest fashions are supported by that (through one among its plugins).
Extra importantly although, even the act of a single immediate to “Generate an SVG of a pelican driving a bicycle” can reveal fascinating mannequin traits.
Contemplate the outcome for Kimi K3 at this time. Operating these easy prompts helped emphasize a number of factors in regards to the mannequin.
It solely has one reasoning effort proper now, “max”—and it reveals. The mannequin consumed 13,241 reasoning tokens to output 3,417 tokens of response. That is costly—the pelican value 25 cents!
How does the immediate “Generate an SVG of a pelican driving a bicycle” add as much as 95 enter tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hello” to Kimi K3 counted 86 tokens, suggesting there could also be an 85 token hidden system immediate. It refused to leak it although.
Imaginative and prescient works nicely: the alt textual content it generated is superb.
K3 presently solely has one considering effort degree, however I’ve been deriving fairly a little bit of worth not too long ago from operating the identical pelican immediate by completely different effort ranges to get a fast thought for what influence these have. Right here’s my matrix for the GPT-5.6 mannequin household, for instance.
Actually although the principle issues I acquire from the pelican take a look at are:
It’s a “hiya world” train for prompting a mannequin
A tough value and reasoning estimate for a easy job
Affirmation that the mannequin can output legitimate SVG and has a fundamental thought of geometry and spatial consciousness. This can be a a lot larger deal for the smaller fashions that run on my laptop computer.
It’s nonetheless fascinating to match pelicans between releases in the identical mannequin household. K3’s pelican is a notable enchancment from Kimi 2.5.
It’s one thing I can share that demonstrates I’ve tried it. Plus a remark with a pelican in it’s type of a practice on Hacker Information at this level, any time I’m late I get feedback asking the place it’s!
