by yourself {hardware} is that it’s free. You got the machine, the electrical energy is a rounding error, and each token after that’s gravy. A current In direction of Knowledge Science piece, How A lot Does It Truly Value to Run a Native LLM?, put a quantity on that gravy by sampling the facility draw of an RTX 3090 and pricing it towards an actual electrical energy tariff. Its discovering caught with me: the costliest mannequin to run wasn’t the largest one. Value per token tracked throughput, not parameter depend.
I don’t have a 3090. I run every thing on an M3 Extremely Mac Studio with 96 GB of unified reminiscence—no discrete GPU, no VRAM, only one pool of reminiscence the CPU and GPU share. That’s a special sufficient machine that I needed to know whether or not the identical shock holds, and by how a lot. So I measured it. 5 fashions, sustained era, actual wall-socket power, priced at my precise utility price. The brief model: the shock doesn’t simply maintain on Apple Silicon, it will get greater. My 120-billion-parameter mannequin is about 5 occasions cheaper per token than a mannequin 1 / 4 its measurement.
Right here’s how I received the numbers, and why they land the place they do.
How the measurement works
The instrument is a small software I constructed known as TokenWatt. It’s an OpenAI-compatible proxy constructed only for Apple Silicon: you level it at no matter native inference server you already run, and it forwards each request byte-for-byte whereas bracketing it with an power measurement. It reads the whole-SoC energy rails by means of Apple’s IOReport interface—the identical counters Exercise Monitor’s power tab is constructed on—which suggests no sudo, no exterior instrumentation, simply the numbers the chip stories about itself. It subtracts a rolling idle baseline so that you’re measuring the marginal value of the request, not the machine sitting there respiration, and it costs the end result at your electrical energy price. Mine is a flat $0.31/kWh.
The honesty downside with any measurement like that is that the on-die counters can drift, and there’s no manner for a reader to know whether or not to belief them. So the one factor I insisted on—and the explanation the software exists in any respect—is calibration towards floor fact. I ran each measurement on this article with the Mac plugged right into a Shelly Plug US Gen4 that meters precise power on the wall, and TokenWatt’s built-in calibration match the SoC counters to it—no hand-tuning, and the identical process anybody with a metering plug can rerun. Each quantity under carries an actual error band from that match; they land between ±2.6% and ±4.5%. If you see “$0.109 per million tokens,” that’s a wall-verified determine with a acknowledged uncertainty, not a hopeful studying off an inner register.
For the headline numbers I ran every mannequin in a sustained era loop—the machine doing nothing however decoding tokens as quick as it will probably—at three durations (120, 360, and 720 seconds), three passes every, with a 15-minute idle baseline between runs. That’s the cleanest, most reproducible sign you will get: a saturated GPU, no gaps, no chilly begins. It’s a greatest case, and I’ll come again to what occurs in messy actual site visitors later.
The 5 fashions
All the pieces right here matches in 96 GB, which seems to matter (extra on that close to the tip). The fashions span from a 4-billion-parameter dense mannequin as much as a 120-billion-parameter mixture-of-experts, throughout totally different quantizations. “Footprint” is what the weights occupy in unified reminiscence, and “tok/s” is sustained decode throughput. Value is per million output (generated) tokens, at $0.31/kWh.
I sorted that desk by value on objective, as a result of sorting it by measurement would scramble it. The most affordable mannequin is the smallest, positive. However the costliest mannequin, by an element of 5 to 9, is the 27-billion-parameter dense one—smaller than three of the 4 fashions it’s dearer than. The 120B mannequin undercuts it 5 to at least one. Parameter depend merely doesn’t predict what a mannequin prices to run right here: the costliest one is a mid-size dense mannequin, and each mannequin bigger than it’s cheaper.
Why the large mannequin wins
The mechanism is identical one the RTX-3090 piece recognized, and it’s price stating plainly: the electrical energy value of a token is watts divided by throughput. Two issues set it—how a lot energy the machine attracts whereas producing, and what number of tokens it produces per second—and the 27B dense mannequin loses on each.
Have a look at its two numbers. It attracts 138 watts, essentially the most of any mannequin right here, and it generates solely 21.5 tokens per second, the fewest. The reason being the factor that governs single-stream decode on this {hardware}: to provide one token, the machine has to learn each energetic weight out of reminiscence, and era velocity is about by what number of bytes that’s divided by the chip’s reminiscence bandwidth. A dense 8-bit mannequin prompts all 27 billion parameters for each token—it streams its whole ~28 GB of weights per token. That’s essentially the most information motion of something within the desk, which is why it’s each the slowest and essentially the most power-hungry: excessive bytes-per-token, low velocity, the worst of each worlds.
The mixture-of-experts fashions break that hyperlink between measurement and bytes-per-token. gpt-oss-120b has 120 billion parameters sitting in reminiscence, however for any given token a router prompts solely about 5 billion of them—so it reads a small fraction of its 59 GB per token, regardless that the entire thing is resident. Much less information moved per token means extra velocity (74 tok/s) and fewer energy (94 W). Quantization compounds it: the MoEs right here run at 4 to six bits towards the dense mannequin’s 8, shrinking the bytes-per-token additional. The result’s a 120B mannequin that’s genuinely cheaper and quicker to run than a 27B one, on the identical machine, measured on the identical wall socket—as a result of what you pay for per token is bytes streamed, and parameter depend solely units the bytes if the mannequin prompts all of them.
The 4B mannequin wins on pure value as a result of it’s tiny and blisteringly quick (134 tok/s), but it surely’s a special high quality tier. The true lesson is in the course of the desk: amongst fashions you’d truly attain for on a tough job, the well-quantized MoEs beat the dense mid-size mannequin outright. For those who have been selecting by parameter depend—”27B is smaller, it have to be cheaper to run than a 120B”—you’d decide the one costliest possibility on the board.
Does it maintain when the site visitors is actual?
A sustained loop is a benchmark, not a workload. It retains the GPU pinned at 100% with no idle gaps, no chilly begins, no ready on a person. Actual serving is lumpier, and lumpier is dearer per token. So the trustworthy query is whether or not the ordering survives contact with precise use.
I’ve a second information supply for that: TokenWatt has been logging my actual inference site visitors for a month—6,300-odd requests throughout these identical fashions, no matter I truly threw at them. These numbers are the software’s estimated (not wall-calibrated) tier, so I belief their ordering greater than their third decimal, however the ordering is the entire level:
The hole is, if something, wider within the wild than on the bench: in day-to-day use the dense 27B value me roughly ten occasions as a lot per token because the MoEs. Two issues transfer these numbers relative to the managed desk—they’re priced per whole token (immediate plus output, not simply era), and actual site visitors runs the machine properly under saturation, which raises the per-token value by roughly two to a few occasions versus the best-case loop. However the form is an identical. The dense mid-size mannequin is the costly one regardless of how I slice it.
The Apple Silicon asterisks
Just a few issues are particular to this machine, they usually matter if you wish to reproduce this or purpose about your individual.
Holding a giant mannequin resident prices energy even between tokens. On a unified-memory machine the entire mannequin lives in the identical RAM the system makes use of, and holding tens of gigabytes of weights resident carries a standing overhead—reminiscence bandwidth, regulators, followers—separate from the arithmetic of era. I measured this overhead climb with mannequin measurement after which flatten out at roughly 21 watts as soon as a mannequin is sufficiently big; the calibration itself additionally shifts barely for the most important, memory-bound fashions. It’s a second-order impact towards the throughput story, but it surely’s actual—another reason to maintain solely the fashions you’re actively utilizing resident.
96 GB is a ceiling, and it’s why the desk stops the place it does. My largest mannequin, gpt-oss-120b at MXFP4, occupies about 59 GB resident. Add the OS and a working KV cache and I’m close to the sensible restrict of a 96 GB machine—which is precisely why there’s no 200B mannequin within the desk. An MoE has to carry all its specialists in reminiscence regardless that it solely makes use of a couple of per token, so the footprint is the total mannequin measurement, not the active-parameter measurement. A 192 GB or 256 GB Mac Studio might run considerably bigger fashions, and the identical watts-over-throughput logic predicts the place they’d land. The outcomes additionally reproduce downward: on a 16, 32, or 64 GB Mac, run the smaller rows right here and you must see the identical ordering, as a result of the mechanism doesn’t depend upon my specific machine.
That is marginal electrical energy solely. It isn’t the price of the Mac, which is by far the bigger quantity and relies upon solely on how onerous you employ it. Amortizing the machine this ran on—a Mac Studio that Apple presently sells for $5,299 (M3 Extremely, 96 GB, 1 TB)—over a couple of thousand requests dwarfs a tenth of a cent of electrical energy. The measurement right here is the power flooring beneath that equation—helpful for evaluating fashions towards one another, and for figuring out the true variable value of a token, however not a complete value of possession.
And a phrase on the cloud, since everybody asks. For scale, a budget hosted fashions folks truly attain for aren’t free both: as of mid-2026 GPT-5.4-nano lists at $1.25 per million output tokens, Gemini 2.5 Flash at $2.50, and Claude Haiku 4.5 at $5. My electrical energy for 1,000,000 output tokens ranges from $0.06 on the 4B to $0.55 on the dense 27B. That reads like a rout for native, however the comparability is rigged in native’s favor: my determine is marginal electrical energy solely, on a machine I already paid a number of thousand {dollars} for, operating smaller fashions than these hosted endpoints. The honest takeaway isn’t “native is ten occasions cheaper.” It’s that when the machine is purchased, the marginal value of a token actually is low—and which mannequin you decide swings it by an order of magnitude. The cloud bundles the {hardware} you’d in any other case amortize; native unbundles it and palms you the electrical energy invoice, which is the one quantity I can measure precisely.
The takeaway
Choose the smallest, quickest mannequin that clears your high quality bar—that’s the identical recommendation the RTX-3090 piece landed on, and Apple Silicon doesn’t change it. What Apple Silicon provides is a sharper corollary: as a result of unified reminiscence lets a giant MoE match and since an MoE streams solely a fraction of its weights per token, a well-quantized 120B mannequin might be cheaper and quicker to run than a dense 27B one. Don’t decide by parameter depend. Choose by throughput, and let the measurement inform you which is which.
One caveat the associated fee numbers can’t present: worth solely issues amongst fashions that may truly do the job. In my very own agentic coding work—constructing a Lisp interpreter with actual tool-calling loops—the Qwen3.6 fashions held instruction-following and gear calls collectively extra reliably than a equally sized Gemma-4-31b, which is why Gemma by no means earned a slot in my rotation, or this desk. Clear the standard bar first; then let value and throughput break the tie. Which native fashions truly maintain up underneath an actual tool-calling workload is the place I’m headed subsequent.
And measure your individual, as a result of your price and your fashions aren’t mine. TokenWatt is open supply, it sits in entrance of no matter native server you already run, and it takes two strains to level at it:
uv software set up tokenwatt # or: uvx tokenwatt · pip set up tokenwatt
tokenwatt serve –upstream http://127.0.0.1:8080 –rate 0.31
It logs a per-request, per-model ledger just like the one above, and if in case you have a metering good plug it would calibrate itself towards the wall so your numbers carry an actual error bar as an alternative of a shrug. The repo is at github.com/mmmugh/tokenwatt. I’d genuinely prefer to see these numbers reproduced on different machines and different charges—the entire level of measuring is that you simply don’t must take my phrase for it.
All figures measured on a Mac Studio (Apple M3 Extremely, 96 GB, macOS 26), wall-calibrated towards a Shelly Plug US Gen4, priced at $0.31/kWh. Sustained-generation figures are plug-calibrated with ±2.6–4.5% bands; real-traffic figures are the uncalibrated estimated tier and are directional. Prices are marginal electrical energy solely and exclude {hardware}.
