The sample. That is Google’s Open Data Format skeleton — a Markdown file with a YAML frontmatter block — repurposed for agent hand-off. The repo’s frontmatter carries one further load-bearing subject the overall OKF spec doesn’t outline: token_pointer, an absolute path to the pre-computed .npy array in shared reminiscence. Human-readable physique, machine-readable pointer.
The mechanism. Three Qwen2.5-Coder fashions of various sizes (7B / 3B / 1.5B) can not share a KV cache — they’ve totally different architectures. However they can share pre-computed token IDs, as a result of the entire Qwen2.5-Coder household ships one equivalent BPE vocabulary. This repo tokenizes as soon as, fingers off the integer array by way of /dev/shm/qwen_tokens/, and lets each downstream agent skip its personal tokenizer fully on the enter facet.
The numbers. Median of seven trials per immediate, 3 blocks, grasping decoding, 64 new tokens: on the 3B mannequin, imply baseline TTFT drops from 69.3 ms to 49.9 ms — a 28.0% discount. On the 1.5B mannequin, from 49.6 ms to 30.9 ms — a 37.8% discount. Each fashions cross the coherence heuristic on each pattern. Full pipeline wall clock is 41.3 s finish to finish (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).
The guardrail. Feeding a downstream mannequin an integer array that meant a totally different subword below its personal vocabulary doesn’t crash something. It generates a fluent, coherent-looking, fully flawed report. So earlier than any agent trusts one other agent’s integers, this pipeline runs a full ~151,936-entry get_vocab() dict equality test — not a vocab_size comparability, the true factor.
What this does NOT declare. Quick-block regime (few-hundred-token blocks). No customized CUDA — that is orchestration on high of transformers‘ present mannequin.generate(input_ids=…) API. Tokenizer equivalence is verified for the precise three checkpoints this repo pins, not a family-wide standing assure.
TL;DR up entrance, so you’ll be able to go away with the purpose: when you have ever wired three or extra LLM-based brokers from the identical mannequin household right into a pipeline that followers out over one shared doc, your CPU is operating the very same Byte-Pair Encoding merges over the very same characters two or 3 times in a row, as a result of every agent’s tokenizer is a stateless new child that has no thought the earlier agent already produced the identical integer array. This submit is a few small pipeline of three Qwen2.5-Coder fashions (7B, 3B, 1.5B) the place the upstream agent tokenizes as soon as, drops a NumPy array of int64 token IDs into /dev/shm/qwen_tokens/, and each downstream agent calls mannequin.generate(input_ids=…) immediately on that array. It additionally — and that is the place the truly fascinating engineering lives — refuses to let anybody else within the pipeline belief that array till it has confirmed, byte for byte, that each mannequin within the chain agrees on what these integers imply. That is orchestration, not a CUDA kernel. However when you have ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the form of the issue this piece of infrastructure is designed to stop.
Github repo: https://github.com/AnubhabBanerjee/inter-llm-tokf
1. A confession: your second agent is doing all of your first agent’s homework, twice
Let me dramatise the second this entire repo is about.
Think about you’ve got three LLM brokers chained collectively. Agent 1 is a giant mannequin, it reads a design doc. Agent 2 is a mid-sized mannequin, it evaluates a part of it. Agent 3 is a small mannequin, it writes the ultimate report. All three of them come from the identical mannequin household — identical tokenizer, identical vocabulary, identical every thing above the hidden layers — simply at three totally different sizes. Since you aren’t made from H100s, and operating a 7B mannequin 3 times when a 1.5B mannequin will do for the final step can be, frankly, impolite to your GPU.
Now watch what occurs on a naive setup:
You: “Agent 1, please learn this design doc and cross the related sections to Agent 2.”
Agent 1 (7B): “On it. Loading tokenizer. Operating BPE over the entire doc. Sections break up. Handing off the fascinating sections to Agent 2 as strings. ✅”
You: “Nice. Agent 2?”
Agent 2 (3B): “Hey, I’m a stupendous, stateless new child. Loading my very own tokenizer. Operating BPE over the identical characters Agent 1 already ran BPE over three seconds in the past. Writing an analysis.”
You: “Wait, you’ve got the very same tokenizer as Agent 1.”
Agent 2 (3B): “I do?”
You: “Sure. You might be actually in the identical mannequin household. Identical vocabulary, identical subword IDs, identical every thing.”
Agent 2 (3B): “That’s good. Anyway, I’ve re-tokenized the enter from scratch and I’m able to generate. Please stand by. 🫡”
You: “…and Agent 3?”
Agent 3 (1.5B): “Loading tokenizer. Operating BPE over Agent 2’s output—”
You: “You understand what, overlook I requested.”
That’s the joke, and it’s the soiled secret of each multi-agent LLM pipeline that followers out over one shared piece of textual content utilizing fashions from the identical household. The tokenizer will not be the bottleneck — a quick Rust-backed BPE tokenizer will not be gradual, and I can’t deceive you and faux it’s. However the tokenizer is redundant work, and what number of occasions you do redundant work will not be a operate of how briskly the redundant work is. It’s a operate of what number of downstream customers you fanned out to.
The purpose of this piece of infrastructure, and the entire cause it took greater than a fifteen-line patch, is that the second you resolve to skip the tokenizer on the downstream facet, you’ve got inherited a correctness downside that the tokenizer was beforehand doing for you. The remainder of this submit is what that appears like whenever you draw it out truthfully, and the one runtime test that’s doing all of the load-bearing work.
2. Why three sizes in any respect? (a one-minute crash course on which layer is definitely shared)
Skip this in the event you already know. For everybody else, right here is the quick model.
The three fashions on this pipeline are Qwen/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen2.5-Coder-3B-Instruct, and Qwen/Qwen2.5-Coder-1.5B-Instruct. Identical structure household, identical tokenizer, three totally different sizes. The rationale they’re three totally different sizes and never one large one is intentionally telecom-flavored, as a result of that’s the world I truly got here from: the concrete instance this repo is constructed in opposition to is a design doc proposing {that a} chain of LLM brokers assist a cell core community’s operations staff cause a few new control-plane characteristic — particularly, bolting MCP (Mannequin Context Protocol) and A2A (Agent-to-Agent protocol) fashion orchestration onto the present 5G Service-Based mostly Interface. The plan requires a big “Architect” agent that constructions the doc, a mid-sized “Protocol Engineer” that evaluates the fascinating sections, and a small “Edge Analyst” that produces deployment-ready latency steering — sufficiently small to run at a far-edge web site subsequent to a UPF.
Three sizes, three roles, one pipeline.
Now, one structural reality drives your complete design: you can not share a KV cache throughout these three fashions. Totally different sizes imply totally different hidden_size values — 3584 for the 7B, 2048 for the 3B, 1536 for the 1.5B. The form of a KV cache is derived immediately from that quantity, so there isn’t a reinterpreting one mannequin’s cache as one other’s. That door is closed, completely, by the mathematics.
What’s not closed is the tokenizer. Qwen2.5-Coder ships one BPE vocabulary throughout its complete dimension vary — the entire household is documented to agree on the identical integer-to-subword mapping. So whilst you can’t share activations between differently-sized fashions, you completely can share token IDs, supplied — and this “supplied” is doing plenty of work, extra on that in a minute — each mannequin within the chain actually does use that very same vocabulary.
When you’ve got learn sufficient distributed-systems papers to be harmful, this form is acquainted. Two community capabilities on the identical message bus don’t get to imagine they agree on message semantics simply because they’re each plugged into the identical bus. Two fashions in the identical household don’t get to imagine they agree on hidden states simply because they agree on vocabulary. Totally different layer, identical self-discipline: discover the precise layer of the stack the place interoperability is definitely assured, and refuse to imagine it holds one layer greater simply because the layers are adjoining.
The tokenizer is that layer. The whole lot above it’s a form mismatch. The whole lot at or beneath it, if we’re fortunate and if we test, is a free integer array.
3. OKF: the “simply hand off the integers” sample
Right here is the pitch in 5 bullets:
Agent 1 hundreds solely the 7B mannequin’s tokenizer — by no means its weights. It splits the doc, tags every part, and tokenizes every part.
It saves every part’s token IDs as a NumPy int64 array into /dev/shm/qwen_tokens/. That may be a RAM-backed tmpfs mount, not disk, so studying it again is a memcpy, by no means a search.
It additionally writes one Markdown file per part into okf_workspace/. The Markdown physique is the part’s human-readable textual content. The YAML frontmatter carries the metadata — block_id, tags, token_pointer, token_count, tokenizer_model_id, and so forth.
Agent 2 (the 3B mannequin) reads the frontmatter, follows token_pointer into shared reminiscence, hundreds the .npy, and calls mannequin.generate(input_ids=…) immediately on the loaded tensor. No tokenizer name on the enter facet.
Agent 2 tokenizes its personal output (that textual content has, by definition, by no means been tokenized earlier than — nothing to reuse), saves that array to shm, writes one other OKF file, and Agent 3 (1.5B) does the identical trick once more.
A fast introduction on the “OKF” (for individuals who don’t know but)
OKF stands for Open Data Format, and earlier than you learn the frontmatter block beneath, one factor is value being trustworthy about.
The Open Data Format is a printed spec — Google Cloud shipped v0.1 in June 2026 and v0.2 is now the present model (see GoogleCloudPlatform/knowledge-catalog on GitHub). Its pitch is deliberately minimal: a bundle is a listing of UTF-8 Markdown information, every file is one idea, and every file carries a YAML frontmatter block plus a Markdown physique. The one frontmatter subject the spec requires is kind — a brief human-readable string like BigQuery Desk, Playbook, or Attested Computation. The whole lot else is non-obligatory metadata. It’s a format, not a platform: no schema registry, no SDK, no central authority. In case you can cat a file, you’ll be able to learn OKF.
This repo’s okf/ reuses that actual skeleton — one Markdown file per unit of labor, YAML frontmatter plus a human-readable physique — however interprets it for a job the overall spec was not written for: an agent-to-agent hand-off of pre-tokenized integer arrays. So this repo’s required frontmatter fields aren’t Google’s kind; they’re block_id, source_agent, stage, title, tags, token_pointer, token_count, tokenizer_model_id, and created_at (see utils/okf_parser.py‘s REQUIRED_FRONTMATTER_KEYS). The load-bearing one is token_pointer — an absolute path into /dev/shm/qwen_tokens/ — which has no equal within the common OKF spec as a result of Google’s OKF was designed for sturdy information sharing, not for a shared-memory hand-off between short-lived agent processes on the identical GPU host. Put plainly: this repo’s information are not legitimate Google-OKF bundles as-is (they lack kind, they add token_pointer); the repo is conforming in spirit — identical Markdown+YAML aesthetic, identical “standardise the interoperability floor, not the content material mannequin” intuition — with one domain-specific required subject bolted on. This submit retains the repo’s terminology as a result of that’s what the supply code and the generated information truly use.
With that out of the way in which, right here is the schema within the wild — the precise frontmatter block from okf_workspace/block_004_routing_and_signaling_integration_points.md, unedited:
—
block_id: block_004_routing_and_signaling_integration_points
source_agent: agent_1_architect
stage: 1
title: Routing and Signaling Integration Factors
tags:
– routing
– signaling
– safety
– deployment
token_pointer: /dev/shm/qwen_tokens/block_004_routing_and_signaling_integration_points.npy
token_count: 3437
tokenizer_model_id: Qwen/Qwen2.5-Coder-7B-Instruct
created_at: ‘2026-08-04T12:41:39.249368+00:00’
—
The load-bearing subject is token_pointer. The whole lot else — source_agent, stage, tags, token_count, tokenizer_model_id, created_at — exists to help routing and provenance selections round that one array. Agent 2 filters the workspace by tag (routing or signaling, each set off it). Agent 3 filters by supply agent (agent_2_protocol_eval, so it by no means by accident picks up its personal output on a re-run). The tokenizer_model_id subject is there so a future audit can cross-check per-file which tokenizer truly produced the bytes at that path, as a substitute of trusting one pipeline-start assertion for all eternity.
Yet one more architectural element value calling out: every agent is a separate OS course of. src/run_pipeline.py launches them through subprocess.run, one by one. That’s deliberate, not lazy: a CUDA context solely releases its VRAM again to the driving force when the method holding it exits. So operating three multi-GB fashions sequentially inside one course of would leak every prior mannequin’s VRAM into the following agent’s reminiscence funds until each caller remembered to manually del mannequin; torch.cuda.empty_cache() — and even that’s not at all times ample to completely reclaim CUDA context overhead. Subprocess isolation makes VRAM launch unconditional and computerized. On a single-GPU field, that is what lets the 7B, then the 3B, then the 1.5B every get the entire card to themselves in flip, with out ever needing all three resident in reminiscence concurrently.
4. The precise save/load code, all six significant strains of it
Now the code that does the precise hand-off. From utils/token_manager.py, verbatim:
def save_token_array(token_ids: torch.Tensor, block_name: str) -> Path:
…
token_ids_as_numpy_int64 = token_ids.detach().cpu().numpy().astype(TOKEN_ARRAY_DTYPE)
destination_path = QWEN_TOKENS_SHM_DIR / f”{block_name}.npy”
np.save(destination_path, token_ids_as_numpy_int64, allow_pickle=False)
return destination_path
That’s the write half. Three strains that truly transfer information. QWEN_TOKENS_SHM_DIR is /dev/shm/qwen_tokens, a RAM-backed tmpfs mount. TOKEN_ARRAY_DTYPE is np.int64, matching torch’s default torch.lengthy, particularly so the load facet by no means wants a casting step. And allow_pickle=False is there as a result of a .npy file with allow_pickle=True will fortunately deserialise and execute pickled Python objects from disk — pointless assault floor for an array that’s, by definition, pure numeric information.
Right here is the learn half:
def load_token_array(pointer_path: Path) -> torch.Tensor:
…
token_ids_as_numpy_int64 = np.load(pointer_path, allow_pickle=False)
if token_ids_as_numpy_int64.dtype != TOKEN_ARRAY_DTYPE:
increase TypeError(…)
return torch.from_numpy(token_ids_as_numpy_int64)
Additionally three significant strains. np.load reads again the precise .npy header (which embeds dtype, form, and byte-order, all specific), the defensive dtype test refuses to silently .astype() if some future code path ever writes one thing aside from int64 into this namespace, and torch.from_numpy(…) shares reminiscence with the NumPy array — zero-copy, since token IDs from this level ahead are by no means mutated in place by any agent.
That’s the complete on-wire format. A NumPy .npy file, int64, on a RAM-backed mount. In case you have been anticipating one thing unique, sorry to disappoint you.
The final piece of the puzzle is what a downstream agent truly does with the loaded tensor. From utils/model_loader.py, the 2 entry factors that Agent 2 and Agent 3 can name — the naive baseline, and the optimized path. Have a look at them facet by facet, as a result of the entire optimization is one operate name’s value of distinction:
def generate_from_text(mannequin, tokenizer, prompt_text, max_new_tokens):
…
wall_clock_start = time.perf_counter()
encoded_prompt = tokenizer(prompt_text, return_tensors=”pt”)
input_ids = encoded_prompt[“input_ids”].to(mannequin.gadget)
attention_mask = encoded_prompt[“attention_mask”].to(mannequin.gadget)
return _generate_and_measure_ttft(
mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
)
Baseline. Clock begins earlier than tokenizer(…) runs, so the tokenizer-encode price this pipeline exists to skip is totally included within the reported TTFT. That isn’t unintentional — it’s intentionally trustworthy. If the baseline began its clock after tokenization, the comparability would understate the true financial savings and faux the tokenizer was free. It’s not free. It’s quick, however it isn’t free.
Now the optimized facet:
def generate_from_token_ids(mannequin, tokenizer, token_ids, max_new_tokens):
…
wall_clock_start = time.perf_counter()
input_ids = token_ids.unsqueeze(0).to(mannequin.gadget)
attention_mask = torch.ones_like(input_ids)
return _generate_and_measure_ttft(
mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
)
The clock additionally begins right here, with no tokenizer name previous it — the entire level of the comparability. token_ids was already produced by an upstream agent’s tokenizer, already saved into shm, already loaded off shm. All this operate does earlier than beginning the mannequin is unsqueeze a batch dimension and duplicate the array to the GPU. The tokenizer argument continues to be handed in, however solely as a result of _generate_and_measure_ttft wants it to produce pad_token_id and to decode the output tokens again to textual content — the enter facet genuinely by no means hits the tokenizer.
The one-line distinction between these two capabilities — one line, tokenizer(prompt_text, …) — is your complete financial savings. It sounds virtually too small to put in writing an article about. Preserve studying, as a result of the failure mode on the opposite facet of “virtually too small” will not be small in any respect.
5. The half the place I ended trusting the seller docs
Right here is the sentence from my very own challenge notes that made me nervous sufficient to put in writing code as a substitute of simply transport the pipeline: “Qwen2.5-Coder is documented to share one tokenizer throughout the entire household.” Documented. By whom? Checked how not too long ago? What occurs to a few brokers’ value of generated textual content if that seems to be true for six of the seven sizes and subtly not true for the one I picked?
A tokenizer mismatch right here doesn’t crash something. That’s the scary half. mannequin.generate(input_ids=[1234, 5678, …]) doesn’t know or care whether or not 1234 meant the identical subword to whoever produced it because it means to the mannequin about to embed it. It would fortunately run a ahead cross on integers that decode to finish nonsense below its personal vocabulary, and it’ll fortunately generate a fluent-looking continuation of that nonsense. You get a confidently flawed report, not an error. Your tokenizer: not the bottleneck. Your assumptions about your tokenizer: fully the bottleneck.
So earlier than any agent is allowed to belief a token array it didn’t produce itself, this runs — from utils/env_checks.py:
def verify_tokenizer_equivalence(
model_ids: tuple[str, …] = PIPELINE_MODEL_IDS,
) -> None:
…
loaded_tokenizers = {
model_id: AutoTokenizer.from_pretrained(model_id) for model_id in model_ids
}
reference_model_id = model_ids[0]
reference_tokenizer = loaded_tokenizers[reference_model_id]
reference_vocab_size = reference_tokenizer.vocab_size
reference_vocab = reference_tokenizer.get_vocab()
for candidate_model_id in model_ids[1:]:
candidate_tokenizer = loaded_tokenizers[candidate_model_id]
if candidate_tokenizer.vocab_size != reference_vocab_size:
increase RuntimeError(
f”Tokenizer vocab_size mismatch: {reference_model_id} has ”
f”vocab_size={reference_vocab_size}, however {candidate_model_id} ”
f”has vocab_size={candidate_tokenizer.vocab_size}. Token IDs ”
“produced by one aren’t protected to feed into the opposite’s ”
“embedding layer.”
)
if candidate_tokenizer.get_vocab() != reference_vocab:
increase RuntimeError(
f”Tokenizer vocabulary mismatch between {reference_model_id} ”
f”and {candidate_model_id}: not less than one token string maps ”
“to a unique integer id between the 2. Direct token ”
“injection throughout these fashions would silently corrupt ”
“downstream generations.”
)
if candidate_tokenizer.special_tokens_map != reference_tokenizer.special_tokens_map:
increase RuntimeError(
f”Particular-tokens map mismatch between {reference_model_id} ”
f”({reference_tokenizer.special_tokens_map}) and ”
f”{candidate_model_id} ({candidate_tokenizer.special_tokens_map}).”
)
Three checks, intentionally layered.
The primary test is vocab_size. It exists purely so a mismatch right here produces a brief, immediately-readable error naming the 2 integers that disagree, as a substitute of forcing whoever is debugging this to diff two ~151,936-entry dicts by hand to seek out that the sizes alone differ.
The second test — the load-bearing one — is full dictionary equality on get_vocab(). Not a vocab_size comparability. A full dict != dict over your complete ~151,936-entry mapping of each subword string to each integer id. Two tokenizers can have equivalent sizes and nonetheless disagree about what integer 42 means. That is the test that will catch a “shuffled id task for even a single subword” mismatch, which is precisely the sort of failure that produces fluent nonsense downstream as a substitute of a loud error.
The third test is special_tokens_map. A mannequin’s chat template and stopping habits depend upon these actual strings/ids matching too — an accurate major vocabulary with a divergent EOS id, for instance, would make a downstream agent’s generate() name fail to cease on the boundary Agent 1 supposed.
I wished the precise assure, not a budget proxy for it. Ran it in opposition to the true triplet earlier than writing one other line of pipeline code, and it held: Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-1.5B-Instruct all agree, byte for byte. Good. However “it held, this time, for this triplet” is a really totally different sentence from “it’s documented to carry,” and solely a kind of two sentences belongs in a pipeline you’re going to run unattended.
6. The receipts
Identical 3 sections of the design doc (those Agent 1’s key phrase scan tagged routing or signaling — block_002 at 1948 tokens, block_003 at 2292 tokens, block_004 at 3437 tokens). Identical grasping decoding. Max 64 new tokens for the timed comparability. One throwaway warm-up name absorbed earlier than any timed measurement so cuBLAS’s first-call kernel choice doesn’t contaminate the numbers. Median of seven repeated trials per block, to easy out millisecond-scale scheduling and GPU-clock jitter.
Straight from scripts/benchmark.py‘s output:
=== Benchmarking Qwen/Qwen2.5-Coder-3B-Instruct ===
Metric 1 (TTFT discount): mean_baseline=69.3 ms, mean_injection=49.9 ms, discount=28.0% — PASS
Metric 2 (semantic constancy): PASS
=== Benchmarking Qwen/Qwen2.5-Coder-1.5B-Instruct ===
Metric 1 (TTFT discount): mean_baseline=49.6 ms, mean_injection=30.9 ms, discount=37.8% — PASS
Metric 2 (semantic constancy): PASS
[benchmark] ALL ACCEPTANCE METRICS PASSED
In desk kind:
The fascinating bit will not be that each fashions obtained sooner — in fact they did, they stopped doing redundant work. The fascinating bit is why the 1.5B mannequin’s share discount is noticeably larger than the 3B mannequin’s, although absolutely the variety of milliseconds saved is roughly comparable. The reason is within the repo’s personal README, and it’s value quoting as a result of it’s the sort of factor that journeys folks up in the event that they solely learn the desk:
The tokenizer’s CPU price is similar string, tokenized as soon as, no matter which mannequin reads the outcome — however GPU forward-pass latency scales with mannequin dimension. For the smaller 1.5B mannequin, that GPU-side ground is decrease, so the (roughly mounted) tokenizer price it avoids is a bigger fraction of its whole time-to-first-token.
That can be, by the way, why blindly rising the enter doc additional doesn’t push the discount towards 100%. Previous a sure enter size, GPU compute time itself begins rising too, and the proportion plateaus slightly than climbing indefinitely. The financial savings scale with how a lot textual content you’d in any other case redundantly re-tokenize, occasions what number of downstream brokers share that very same enter, divided by how large every downstream mannequin’s personal ahead cross is. On a brief single-hop demo, you get double-digit %. On a big supply doc fanned out to many downstream brokers of the identical household, you pay the BPE price as soon as as a substitute of N occasions, which is precisely the regime the plan was constructed for.
The semantic-fidelity facet of the receipts is a heuristic, on objective. Two ratios: printable-character ratio ≥ 0.98, and unique-word ratio ≥ 0.25 throughout the pattern’s tokens. Low-cost sufficient to run on each technology, calibrated to catch the particular “rubbish output” failure mode a tokenizer mismatch or byte-order bug produces — degenerate repetition of 1 token, or a wall of non-printable control-character noise — not a common high quality judgment. Each pattern from each mannequin handed. Learn extra particulars concerning the outcomes right here.
7. Wrap: the truly fascinating half was the guardrail
The fascinating a part of this challenge was by no means “skip the tokenizer, it’s gradual.” Tokenizers, particularly the quick Rust-backed variety, aren’t the bottleneck anybody thinks they’re — the numbers above show that themselves. Saving 20 ms of TTFT is good. It’s not the purpose.
The fascinating half was constructing the one piece of infrastructure that makes skipping the tokenizer protected: a runtime test that refuses to let one agent belief one other agent’s integers till it has truly confirmed they communicate the identical language, byte for byte, vocabulary entry for vocabulary entry. That test is what turns “20 ms sooner” from a footgun right into a dependable engineering transfer. With out it, you’ve got a pipeline that’s quick when it really works and confidently flawed when it doesn’t, and no clear approach to inform which one you’re at the moment residing in.
Each multi-agent pipeline that passes state between fashions is making an assumption like this someplace, normally silently. Generally it’s about tokenizer vocabularies. Generally it’s about hidden-state dimensions. Generally it’s concerning the which means of a selected chat-template string. Generally it’s about which facet of an RPC boundary the retries reside on. Mine simply occurs to be about BPE integer-to-subword mappings, as a result of that’s what this repo’s optimization technique leans on. Yours is some place else. Go discover it. It’s in all probability not documented both.
If you wish to reproduce the numbers, python scripts/benchmark.py on a CUDA GPU with sufficient VRAM for a bf16 3B checkpoint will do it. If you wish to reproduce the pipeline itself in opposition to your individual enter, drop your doc into information/raw_input.txt and python src/run_pipeline.py walks by way of the three phases, cleans up shm on the way in which out, and leaves three OKF information behind in okf_workspace/.
Small pipeline. Modest numbers. One load-bearing test. That’s the entire form of it.
Disclaimer: The illustrations on this article have been generated utilizing AI (Claude Opus 4.8). They’re illustrative, not photographic, and any labels seen inside the photographs are stylized slightly than authoritative — check with the article physique and the code itself for exact operate names, metric values, and structure particulars.
