\n
The sample.\u00a0That is Google\u2019s\u00a0Open Data Format\u00a0skeleton \u2014 a Markdown file with a YAML frontmatter block \u2014 repurposed for agent hand-off. The repo\u2019s frontmatter carries one further load-bearing subject the overall OKF spec doesn’t outline:\u00a0token_pointer, an absolute path to the pre-computed\u00a0.npy\u00a0array in shared reminiscence. Human-readable physique, machine-readable pointer.<\/p>\n
The mechanism.\u00a0Three Qwen2.5-Coder fashions of various sizes (7B \/ 3B \/ 1.5B) can not share a KV cache \u2014 they’ve totally different architectures. However they\u00a0can\u00a0share pre-computed token IDs, as a result of the entire Qwen2.5-Coder household ships one equivalent BPE vocabulary. This repo tokenizes as soon as, fingers off the integer array by way of\u00a0\/dev\/shm\/qwen_tokens\/, and lets each downstream agent skip its personal tokenizer fully on the enter facet.<\/p>\n
The numbers.\u00a0Median of seven trials per immediate, 3 blocks, grasping decoding, 64 new tokens: on the 3B mannequin, imply baseline TTFT drops from\u00a069.3 ms to 49.9 ms\u00a0\u2014 a\u00a028.0%\u00a0discount. On the 1.5B mannequin, from\u00a049.6 ms to 30.9 ms\u00a0\u2014 a\u00a037.8%\u00a0discount. Each fashions cross the coherence heuristic on each pattern. Full pipeline wall clock is\u00a041.3 s\u00a0finish to finish (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).<\/p>\n
The guardrail.\u00a0Feeding a downstream mannequin an integer array that meant a\u00a0totally different\u00a0subword below its personal vocabulary doesn’t crash something. It generates a fluent, coherent-looking, fully flawed report. So earlier than any agent trusts one other agent\u2019s integers, this pipeline runs a full ~151,936-entry\u00a0get_vocab()\u00a0dict equality test \u2014 not a\u00a0vocab_size\u00a0comparability, the true factor.<\/p>\n
What this does NOT declare.\u00a0Quick-block regime (few-hundred-token blocks). No customized CUDA \u2014 that is orchestration on high of\u00a0transformers\u2018 present\u00a0mannequin.generate(input_ids=…)\u00a0API. Tokenizer equivalence is verified for the precise three checkpoints this repo pins, not a family-wide standing assure.<\/p>\n
TL;DR up entrance, so you’ll be able to go away with the purpose:\u00a0when you have ever wired three or extra LLM-based brokers from the identical mannequin household right into a pipeline that followers out over one shared doc, your CPU is operating the very same Byte-Pair Encoding merges over the very same characters two or 3 times in a row, as a result of every agent\u2019s tokenizer is a stateless new child that has no thought the earlier agent already produced the identical integer array. This submit is a few small pipeline of three Qwen2.5-Coder fashions (7B, 3B, 1.5B) the place the upstream agent tokenizes as soon as, drops a NumPy array of\u00a0int64\u00a0token IDs into\u00a0\/dev\/shm\/qwen_tokens\/, and each downstream agent calls\u00a0mannequin.generate(input_ids=…)\u00a0immediately on that array. It additionally \u2014 and that is the place the\u00a0truly\u00a0fascinating engineering lives \u2014 refuses to let anybody else within the pipeline belief that array till it has confirmed, byte for byte, that each mannequin within the chain agrees on what these integers imply. That is orchestration, not a CUDA kernel. However when you have ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the form of the issue this piece of infrastructure is designed to stop.<\/p>\n Github repo: https:\/\/github.com\/AnubhabBanerjee\/inter-llm-tokf<\/p>\n<\/blockquote>\n<\/figure>\n Let me dramatise the second this entire repo is about.<\/p>\n Think about you’ve got three LLM brokers chained collectively. Agent 1 is a giant mannequin, it reads a design doc. Agent 2 is a mid-sized mannequin, it evaluates a part of it. Agent 3 is a small mannequin, it writes the ultimate report. All three of them come from the identical mannequin household \u2014 identical tokenizer, identical vocabulary, identical every thing above the hidden layers \u2014 simply at three totally different sizes. Since you aren’t made from H100s, and operating a 7B mannequin 3 times when a 1.5B mannequin will do for the final step can be, frankly, impolite to your GPU.<\/p>\n Now watch what occurs on a naive setup:<\/p>\n You:\u00a0\u201cAgent 1, please learn this design doc and cross the related sections to Agent 2.\u201d<\/p>\n Agent 1 (7B):\u00a0\u201cOn it. Loading tokenizer. Operating BPE over the entire doc. Sections break up. Handing off the fascinating sections to Agent 2 as strings. \u2705\u201d<\/p>\n You:\u00a0\u201cNice. Agent 2?\u201d<\/p>\n Agent 2 (3B):\u00a0\u201cHey, I’m a stupendous, stateless new child. Loading my very own tokenizer. Operating BPE over the identical characters Agent 1 already ran BPE over three seconds in the past. Writing an analysis.\u201d<\/p>\n You:\u00a0\u201cWait, you’ve got the very same tokenizer as Agent 1.\u201d<\/p>\n Agent 2 (3B):\u00a0\u201cI do?\u201d<\/p>\n You:\u00a0\u201cSure. You might be actually in the identical mannequin household. Identical vocabulary, identical subword IDs, identical every thing.\u201d<\/p>\n Agent 2 (3B):\u00a0\u201cThat\u2019s good. Anyway, I’ve re-tokenized the enter from scratch and I’m able to generate. Please stand by. \ud83e\udee1\u201d<\/p>\n You:\u00a0\u201c\u2026and Agent 3?\u201d<\/p>\n Agent 3 (1.5B):\u00a0\u201cLoading tokenizer. Operating BPE over Agent 2\u2019s output\u2014\u201d<\/p>\n You:\u00a0\u201cYou understand what, overlook I requested.\u201d<\/p>\n That’s the joke, and it’s the soiled secret of each multi-agent LLM pipeline that followers out over one shared piece of textual content utilizing fashions from the identical household. The tokenizer will not be the bottleneck \u2014 a quick Rust-backed BPE tokenizer will not be gradual, and I can’t deceive you and faux it’s. However the tokenizer is redundant work, and\u00a0what number of occasions you do redundant work\u00a0will not be a operate of how briskly the redundant work is. It’s a operate of what number of downstream customers you fanned out to.<\/p>\n The purpose of this piece of infrastructure, and the entire cause it took greater than a fifteen-line patch, is that the second you resolve to skip the tokenizer on the downstream facet, you’ve got inherited a\u00a0correctness\u00a0downside that the tokenizer was beforehand doing for you. The remainder of this submit is what that appears like whenever you draw it out truthfully, and the one runtime test that’s doing all of the load-bearing work.<\/p>\n Skip this in the event you already know. For everybody else, right here is the quick model.<\/p>\n The three fashions on this pipeline are\u00a0Qwen\/Qwen2.5-Coder-7B-Instruct,\u00a0Qwen\/Qwen2.5-Coder-3B-Instruct, and\u00a0Qwen\/Qwen2.5-Coder-1.5B-Instruct. Identical structure household, identical tokenizer, three totally different sizes. The rationale they’re three totally different sizes and never one large one is intentionally telecom-flavored, as a result of that’s the world I truly got here from: the concrete instance this repo is constructed in opposition to is a design doc proposing {that a} chain of LLM brokers assist a cell core community\u2019s operations staff cause a few new control-plane characteristic \u2014 particularly, bolting MCP (Mannequin Context Protocol) and A2A (Agent-to-Agent protocol) fashion orchestration onto the present 5G Service-Based mostly Interface. The plan requires a big \u201cArchitect\u201d agent that constructions the doc, a mid-sized \u201cProtocol Engineer\u201d that evaluates the fascinating sections, and a small \u201cEdge Analyst\u201d that produces deployment-ready latency steering \u2014 sufficiently small to run at a far-edge web site subsequent to a UPF.<\/p>\n Three sizes, three roles, one pipeline.<\/p>\n Now, one structural reality drives your complete design:\u00a0you can not share a KV cache throughout these three fashions.\u00a0Totally different sizes imply totally different\u00a0hidden_size\u00a0values \u2014 3584 for the 7B, 2048 for the 3B, 1536 for the 1.5B. The form of a KV cache is derived immediately from that quantity, so there isn’t a reinterpreting one mannequin\u2019s cache as one other\u2019s. That door is closed, completely, by the mathematics.<\/p>\n What’s\u00a0not\u00a0closed is the tokenizer. Qwen2.5-Coder ships one BPE vocabulary throughout its complete dimension vary \u2014 the entire household is documented to agree on the identical integer-to-subword mapping. So whilst you can\u2019t share\u00a0activations\u00a0between differently-sized fashions, you completely can share\u00a0token IDs, supplied \u2014 and this \u201csupplied\u201d is doing plenty of work, extra on that in a minute \u2014 each mannequin within the chain actually does use that very same vocabulary.<\/p>\n When you’ve got learn sufficient distributed-systems papers to be harmful, this form is acquainted. Two community capabilities on the identical message bus don\u2019t get to imagine they agree on message\u00a0semantics\u00a0simply because they\u2019re each plugged into the identical bus. Two fashions in the identical household don\u2019t get to imagine they agree on\u00a0hidden states\u00a0simply because they agree on vocabulary. Totally different layer, identical self-discipline: discover the precise layer of the stack the place interoperability is definitely assured, and refuse to imagine it holds one layer greater simply because the layers are adjoining.<\/p>\n The tokenizer is that layer. The whole lot above it’s a form mismatch. The whole lot at or beneath it, if we\u2019re fortunate and if we test, is a free integer array.<\/p>\n Right here is the pitch in 5 bullets:<\/p>\n Agent 1 hundreds solely the 7B mannequin\u2019s\u00a0tokenizer\u00a0\u2014 by no means its weights. It splits the doc, tags every part, and tokenizes every part.<\/p>\n It saves every part\u2019s token IDs as a NumPy\u00a0int64\u00a0array into\u00a0\/dev\/shm\/qwen_tokens\/. That may be a RAM-backed tmpfs mount, not disk, so studying it again is a memcpy, by no means a search.<\/p>\n It additionally writes one Markdown file per part into\u00a0okf_workspace\/. The Markdown physique is the part\u2019s human-readable textual content. The YAML frontmatter carries the metadata \u2014\u00a0block_id,\u00a0tags,\u00a0token_pointer,\u00a0token_count,\u00a0tokenizer_model_id, and so forth.<\/p>\n Agent 2 (the 3B mannequin) reads the frontmatter, follows\u00a0token_pointer\u00a0into shared reminiscence, hundreds the\u00a0.npy, and calls\u00a0mannequin.generate(input_ids=…)\u00a0immediately on the loaded tensor. No tokenizer name on the enter facet.<\/p>\n Agent 2 tokenizes\u00a0its personal output\u00a0(that textual content has, by definition, by no means been tokenized earlier than \u2014 nothing to reuse), saves that array to shm, writes one other OKF file, and Agent 3 (1.5B) does the identical trick once more.<\/p>\n OKF stands for\u00a0Open Data Format, and earlier than you learn the frontmatter block beneath, one factor is value being trustworthy about.<\/p>\n The\u00a0Open Data Format\u00a0is a printed spec \u2014 Google Cloud shipped v0.1 in June 2026 and v0.2 is now the present model (see\u00a0GoogleCloudPlatform\/knowledge-catalog\u00a0on GitHub). Its pitch is deliberately minimal: a\u00a0bundle\u00a0is a listing of UTF-8 Markdown information, every file is one\u00a0idea, and every file carries a YAML frontmatter block plus a Markdown physique. The one frontmatter subject the spec\u00a0requires\u00a0is\u00a0kind\u00a0\u2014 a brief human-readable string like\u00a0BigQuery Desk,\u00a0Playbook, or\u00a0Attested Computation. The whole lot else is non-obligatory metadata. It’s a\u00a0format, not a platform: no schema registry, no SDK, no central authority. In case you can\u00a0cat\u00a0a file, you’ll be able to learn OKF.<\/p>\n This repo\u2019s\u00a0okf\/\u00a0reuses that actual skeleton \u2014 one Markdown file per unit of labor, YAML frontmatter plus a human-readable physique \u2014 however interprets it for a job the overall spec was not written for: an agent-to-agent hand-off of pre-tokenized integer arrays. So this repo\u2019s required frontmatter fields aren’t Google\u2019s\u00a0kind; they’re\u00a0block_id,\u00a0source_agent,\u00a0stage,\u00a0title,\u00a0tags,\u00a0token_pointer,\u00a0token_count,\u00a0tokenizer_model_id, and\u00a0created_at\u00a0(see\u00a0utils\/okf_parser.py\u2018s\u00a0REQUIRED_FRONTMATTER_KEYS). The load-bearing one is\u00a0token_pointer\u00a0\u2014 an absolute path into\u00a0\/dev\/shm\/qwen_tokens\/\u00a0\u2014 which has no equal within the common OKF spec as a result of Google\u2019s OKF was designed for sturdy information sharing, not for a shared-memory hand-off between short-lived agent processes on the identical GPU host. Put plainly: this repo\u2019s information are\u00a0not\u00a0legitimate Google-OKF bundles as-is (they lack\u00a0kind, they add\u00a0token_pointer); the repo is conforming in\u00a0spirit\u00a0\u2014 identical Markdown+YAML aesthetic, identical \u201cstandardise the interoperability floor, not the content material mannequin\u201d intuition \u2014 with one domain-specific required subject bolted on. This submit retains the repo\u2019s terminology as a result of that’s what the supply code and the generated information truly use.<\/p>\n With that out of the way in which, right here is the schema within the wild \u2014 the precise frontmatter block from\u00a0okf_workspace\/block_004_routing_and_signaling_integration_points.md, unedited:<\/p>\n — The load-bearing subject is\u00a0token_pointer. The whole lot else \u2014\u00a0source_agent,\u00a0stage,\u00a0tags,\u00a0token_count,\u00a0tokenizer_model_id,\u00a0created_at\u00a0\u2014 exists to help routing and provenance selections\u00a0round\u00a0that one array. Agent 2 filters the workspace by tag (routing\u00a0or\u00a0signaling, each set off it). Agent 3 filters by supply agent (agent_2_protocol_eval, so it by no means by accident picks up its personal output on a re-run). The\u00a0tokenizer_model_id\u00a0subject is there so a future audit can cross-check per-file which tokenizer truly produced the bytes at that path, as a substitute of trusting one pipeline-start assertion for all eternity.<\/p>\n Yet one more architectural element value calling out: every agent is a\u00a0separate OS course of.\u00a0src\/run_pipeline.py\u00a0launches them through\u00a0subprocess.run, one by one. That’s deliberate, not lazy: a CUDA context solely releases its VRAM again to the driving force when the method holding it exits. So operating three multi-GB fashions sequentially inside\u00a0one\u00a0course of would leak every prior mannequin\u2019s VRAM into the following agent\u2019s reminiscence funds until each caller remembered to manually\u00a0del mannequin; torch.cuda.empty_cache()\u00a0\u2014 and even that’s not at all times ample to completely reclaim CUDA context overhead. Subprocess isolation makes VRAM launch unconditional and computerized. On a single-GPU field, that is what lets the 7B, then the 3B, then the 1.5B every get the entire card to themselves in flip, with out ever needing all three resident in reminiscence concurrently.<\/p>\n Now the code that does the precise hand-off. From\u00a0utils\/token_manager.py, verbatim:<\/p>\n def save_token_array(token_ids: torch.Tensor, block_name: str) -> Path: That’s the write half. Three strains that truly transfer information.\u00a0QWEN_TOKENS_SHM_DIR\u00a0is\u00a0\/dev\/shm\/qwen_tokens, a RAM-backed tmpfs mount.\u00a0TOKEN_ARRAY_DTYPE\u00a0is\u00a0np.int64, matching torch\u2019s default\u00a0torch.lengthy, particularly so the load facet by no means wants a casting step. And\u00a0allow_pickle=False\u00a0is there as a result of a\u00a0.npy\u00a0file with\u00a0allow_pickle=True\u00a0will fortunately deserialise and\u00a0execute\u00a0pickled Python objects from disk \u2014 pointless assault floor for an array that’s, by definition, pure numeric information.<\/p>\n Right here is the learn half:<\/p>\n def load_token_array(pointer_path: Path) -> torch.Tensor: Additionally three significant strains.\u00a0np.load\u00a0reads again the precise\u00a0.npy\u00a0header (which embeds dtype, form, and byte-order, all specific), the defensive dtype test refuses to silently\u00a0.astype()\u00a0if some future code path ever writes one thing aside from\u00a0int64\u00a0into this namespace, and\u00a0torch.from_numpy(…)\u00a0shares reminiscence with the NumPy array \u2014 zero-copy, since token IDs from this level ahead are by no means mutated in place by any agent.<\/p>\n That’s the\u00a0complete\u00a0on-wire format. A NumPy\u00a0.npy\u00a0file,\u00a0int64, on a RAM-backed mount. In case you have been anticipating one thing unique, sorry to disappoint you.<\/p>\n The final piece of the puzzle is what a downstream agent truly does with the loaded tensor. From\u00a0utils\/model_loader.py, the 2 entry factors that Agent 2 and Agent 3 can name \u2014 the naive baseline, and the optimized path. Have a look at them facet by facet, as a result of the entire optimization is one operate name\u2019s value of distinction:<\/p>\n def generate_from_text(mannequin, tokenizer, prompt_text, max_new_tokens): encoded_prompt = tokenizer(prompt_text, return_tensors=”pt”)<\/p>\n input_ids = encoded_prompt[“input_ids”].to(mannequin.gadget) return _generate_and_measure_ttft( Baseline. Clock begins\u00a0earlier than\u00a0tokenizer(…)\u00a0runs, so the tokenizer-encode price this pipeline exists to skip is totally included within the reported TTFT. That isn’t unintentional \u2014 it’s intentionally trustworthy. If the baseline began its clock\u00a0after\u00a0tokenization, the comparability would understate the true financial savings and faux the tokenizer was free. It’s not free. It’s quick, however it isn’t free.<\/p>\n Now the optimized facet:<\/p>\n def generate_from_token_ids(mannequin, tokenizer, token_ids, max_new_tokens): input_ids = token_ids.unsqueeze(0).to(mannequin.gadget) return _generate_and_measure_ttft( The clock additionally begins right here, with\u00a0no\u00a0tokenizer name previous it \u2014 the entire level of the comparability.\u00a0token_ids\u00a0was already produced by an upstream agent\u2019s tokenizer, already saved into shm, already loaded off shm. All this operate does earlier than beginning the mannequin is unsqueeze a batch dimension and duplicate the array to the GPU. The\u00a0tokenizer\u00a0argument continues to be handed in, however solely as a result of\u00a0_generate_and_measure_ttft\u00a0wants it to produce\u00a0pad_token_id\u00a0and to decode the\u00a0output\u00a0tokens again to textual content \u2014 the enter facet genuinely by no means hits the tokenizer.<\/p>\n The one-line distinction between these two capabilities \u2014 one line,\u00a0tokenizer(prompt_text, …)\u00a0\u2014 is your complete financial savings. It sounds virtually too small to put in writing an article about. Preserve studying, as a result of the failure mode on the opposite facet of \u201cvirtually too small\u201d will not be small in any respect.<\/p>\n Right here is the sentence from my very own challenge notes that made me nervous sufficient to put in writing code as a substitute of simply transport the pipeline:\u00a0\u201cQwen2.5-Coder is documented to share one tokenizer throughout the entire household.\u201d\u00a0Documented. By whom? Checked how not too long ago? What occurs to a few brokers\u2019 value of generated textual content if that seems to be true for six of the seven sizes and subtly not true for the one I picked?<\/p>\n A tokenizer mismatch right here doesn\u2019t crash something. That\u2019s the scary half.\u00a0mannequin.generate(input_ids=[1234, 5678, …])\u00a0doesn\u2019t know or care whether or not\u00a01234\u00a0meant the identical subword to whoever produced it because it means to the mannequin about to embed it. It would fortunately run a ahead cross on integers that decode to finish nonsense below its personal vocabulary, and it’ll fortunately generate a fluent-looking continuation of that nonsense. You get a confidently flawed report, not an error. Your tokenizer: not the bottleneck. Your assumptions about your tokenizer: fully the bottleneck.<\/p>\n So earlier than any agent is allowed to belief a token array it didn\u2019t produce itself, this runs \u2014 from\u00a0utils\/env_checks.py:<\/p>\n def verify_tokenizer_equivalence( reference_model_id = model_ids[0] reference_vocab = reference_tokenizer.get_vocab()<\/p>\n for candidate_model_id in model_ids[1:]: if candidate_tokenizer.vocab_size != reference_vocab_size: if candidate_tokenizer.get_vocab() != reference_vocab: if candidate_tokenizer.special_tokens_map != reference_tokenizer.special_tokens_map: Three checks, intentionally layered.<\/p>\n The primary test is\u00a0vocab_size. It exists purely so a mismatch right here produces a brief, immediately-readable error naming the 2 integers that disagree, as a substitute of forcing whoever is debugging this to diff two ~151,936-entry dicts by hand to seek out that the sizes alone differ.<\/p>\n The second test \u2014 the load-bearing one \u2014 is full dictionary equality on\u00a0get_vocab(). Not a\u00a0vocab_size\u00a0comparability. A full\u00a0dict != dict\u00a0over your complete ~151,936-entry mapping of each subword string to each integer id. Two tokenizers can have equivalent sizes and nonetheless disagree about what integer\u00a042\u00a0means. That is the test that will catch a \u201cshuffled id task for even a single subword\u201d mismatch, which is precisely the sort of failure that produces fluent nonsense downstream as a substitute of a loud error.<\/p>\n The third test is\u00a0special_tokens_map. A mannequin\u2019s chat template and stopping habits depend upon these actual strings\/ids matching too \u2014 an accurate major vocabulary with a divergent EOS id, for instance, would make a downstream agent\u2019s\u00a0generate()\u00a0name fail to cease on the boundary Agent 1 supposed.<\/p>\n I wished the precise assure, not a budget proxy for it. Ran it in opposition to the true triplet earlier than writing one other line of pipeline code, and it held:\u00a0Qwen2.5-Coder-7B-Instruct,\u00a0Qwen2.5-Coder-3B-Instruct, and\u00a0Qwen2.5-Coder-1.5B-Instruct\u00a0all agree, byte for byte. Good. However \u201cit held, this time, for this triplet\u201d is a really totally different sentence from \u201cit\u2019s documented to carry,\u201d and solely a kind of two sentences belongs in a pipeline you\u2019re going to run unattended.<\/p>\n Identical 3 sections of the design doc (those Agent 1\u2019s key phrase scan tagged\u00a0routing\u00a0or\u00a0signaling\u00a0\u2014 block_002 at 1948 tokens, block_003 at 2292 tokens, block_004 at 3437 tokens). Identical grasping decoding. Max 64 new tokens for the timed comparability. One throwaway warm-up name absorbed earlier than any timed measurement so cuBLAS\u2019s first-call kernel choice doesn\u2019t contaminate the numbers. Median of seven repeated trials per block, to easy out millisecond-scale scheduling and GPU-clock jitter.<\/p>\n Straight from\u00a0scripts\/benchmark.py\u2018s output:<\/p>\n === Benchmarking Qwen\/Qwen2.5-Coder-3B-Instruct === === Benchmarking Qwen\/Qwen2.5-Coder-1.5B-Instruct === [benchmark] ALL ACCEPTANCE METRICS PASSED<\/p>\n In desk kind:<\/p>\n The fascinating bit will not be that each fashions obtained sooner \u2014 in fact they did, they stopped doing redundant work. The fascinating bit is\u00a0why\u00a0the 1.5B mannequin\u2019s share discount is noticeably larger than the 3B mannequin\u2019s, although absolutely the variety of milliseconds saved is roughly comparable. The reason is within the repo\u2019s personal README, and it’s value quoting as a result of it\u2019s the sort of factor that journeys folks up in the event that they solely learn the desk:<\/p>\n The tokenizer\u2019s CPU price is similar string, tokenized as soon as, no matter which mannequin reads the outcome \u2014 however GPU forward-pass latency scales with mannequin dimension. For the smaller 1.5B mannequin, that GPU-side ground is decrease, so the (roughly mounted) tokenizer price it avoids is a\u00a0bigger\u00a0fraction of its whole time-to-first-token.<\/p>\n<\/blockquote>\n That can be, by the way, why blindly rising the enter doc additional doesn’t push the discount towards 100%. Previous a sure enter size, GPU compute time itself begins rising too, and the proportion plateaus slightly than climbing indefinitely. The financial savings scale with\u00a0how a lot textual content you’d in any other case redundantly re-tokenize, occasions\u00a0what number of downstream brokers share that very same enter, divided by\u00a0how large every downstream mannequin\u2019s personal ahead cross is. On a brief single-hop demo, you get double-digit %. On a big supply doc fanned out to many downstream brokers of the identical household, you pay the BPE price as soon as as a substitute of N occasions, which is precisely the regime the plan was constructed for.<\/p>\n The semantic-fidelity facet of the receipts is a heuristic, on objective. Two ratios: printable-character ratio \u2265 0.98, and unique-word ratio \u2265 0.25 throughout the pattern\u2019s tokens. Low-cost sufficient to run on each technology, calibrated to catch the particular \u201crubbish output\u201d failure mode a tokenizer mismatch or byte-order bug produces \u2014 degenerate repetition of 1 token, or a wall of non-printable control-character noise \u2014 not a common high quality judgment. Each pattern from each mannequin handed. Learn extra particulars concerning the outcomes right here.<\/p>\n The fascinating a part of this challenge was by no means \u201cskip the tokenizer, it\u2019s gradual.\u201d Tokenizers, particularly the quick Rust-backed variety, aren’t the bottleneck anybody thinks they’re \u2014 the numbers above show that themselves. Saving 20 ms of TTFT is good. It’s not the purpose.<\/p>\n The fascinating half was constructing the one piece of infrastructure that makes skipping the tokenizer\u00a0protected: a runtime test that refuses to let one agent belief one other agent\u2019s integers till it has truly confirmed they communicate the identical language, byte for byte, vocabulary entry for vocabulary entry. That test is what turns \u201c20 ms sooner\u201d from a footgun right into a dependable engineering transfer. With out it, you’ve got a pipeline that’s quick when it really works and confidently flawed when it doesn\u2019t, and no clear approach to inform which one you\u2019re at the moment residing in.<\/p>\n Each multi-agent pipeline that passes state between fashions is making an assumption like this someplace, normally silently. Generally it\u2019s about tokenizer vocabularies. Generally it\u2019s about hidden-state dimensions. Generally it\u2019s concerning the which means of a selected chat-template string. Generally it\u2019s about which facet of an RPC boundary the retries reside on. Mine simply occurs to be about BPE integer-to-subword mappings, as a result of that’s what this repo\u2019s optimization technique leans on. Yours is some place else. Go discover it. It\u2019s in all probability not documented both.<\/p>\n If you wish to reproduce the numbers,\u00a0python scripts\/benchmark.py\u00a0on a CUDA GPU with sufficient VRAM for a bf16 3B checkpoint will do it. If you wish to reproduce the pipeline itself in opposition to your individual enter, drop your doc into\u00a0information\/raw_input.txt\u00a0and\u00a0python src\/run_pipeline.py\u00a0walks by way of the three phases, cleans up shm on the way in which out, and leaves three OKF information behind in\u00a0okf_workspace\/.<\/p>\n Small pipeline. Modest numbers. One load-bearing test. That\u2019s the entire form of it.<\/p>\n Disclaimer: The illustrations on this article have been generated utilizing AI (Claude Opus 4.8). They’re illustrative, not photographic, and any labels seen inside the photographs are stylized slightly than authoritative \u2014 check with the article physique and the code itself for exact operate names, metric values, and structure particulars.<\/p>\n<\/div>\n\n
1. A confession: your second agent is doing all of your first agent\u2019s homework, twice<\/h2>\n
2. Why three sizes in any respect? (a one-minute crash course on which layer is definitely shared)<\/h2>\n
![\"A](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/08\/hidden_size_vs_vocab-1024x683.png\")
3. OKF: the \u201csimply hand off the integers\u201d sample<\/h2>\n
A fast introduction on the \u201cOKF\u201d (for individuals who don\u2019t know but)<\/h3>\n
\nblock_id: block_004_routing_and_signaling_integration_points
\nsource_agent: agent_1_architect
\nstage: 1
\ntitle: Routing and Signaling Integration Factors
\ntags:
\n– routing
\n– signaling
\n– safety
\n– deployment
\ntoken_pointer: \/dev\/shm\/qwen_tokens\/block_004_routing_and_signaling_integration_points.npy
\ntoken_count: 3437
\ntokenizer_model_id: Qwen\/Qwen2.5-Coder-7B-Instruct
\ncreated_at: ‘2026-08-04T12:41:39.249368+00:00’
\n—<\/p>\n![\"Left-to-right](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/08\/okf_architecture-1024x683.png\")
4. The precise save\/load code, all six significant strains of it<\/h2>\n
\n …
\n token_ids_as_numpy_int64 = token_ids.detach().cpu().numpy().astype(TOKEN_ARRAY_DTYPE)
\n destination_path = QWEN_TOKENS_SHM_DIR \/ f”{block_name}.npy”
\n np.save(destination_path, token_ids_as_numpy_int64, allow_pickle=False)
\n return destination_path<\/p>\n
\n …
\n token_ids_as_numpy_int64 = np.load(pointer_path, allow_pickle=False)
\n if token_ids_as_numpy_int64.dtype != TOKEN_ARRAY_DTYPE:
\n increase TypeError(…)
\n return torch.from_numpy(token_ids_as_numpy_int64)<\/p>\n
\n …
\n wall_clock_start = time.perf_counter()<\/p>\n
\n attention_mask = encoded_prompt[“attention_mask”].to(mannequin.gadget)<\/p>\n
\n mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
\n )<\/p>\n
\n …
\n wall_clock_start = time.perf_counter()<\/p>\n
\n attention_mask = torch.ones_like(input_ids)<\/p>\n
\n mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
\n )<\/p>\n5. The half the place I ended trusting the seller docs<\/h2>\n
\n model_ids: tuple[str, …] = PIPELINE_MODEL_IDS,
\n) -> None:
\n …
\n loaded_tokenizers = {
\n model_id: AutoTokenizer.from_pretrained(model_id) for model_id in model_ids
\n }<\/p>\n
\n reference_tokenizer = loaded_tokenizers[reference_model_id]
\n reference_vocab_size = reference_tokenizer.vocab_size<\/p>\n
\n candidate_tokenizer = loaded_tokenizers[candidate_model_id]<\/p>\n
\n increase RuntimeError(
\n f”Tokenizer vocab_size mismatch: {reference_model_id} has ”
\n f”vocab_size={reference_vocab_size}, however {candidate_model_id} ”
\n f”has vocab_size={candidate_tokenizer.vocab_size}. Token IDs ”
\n “produced by one aren’t protected to feed into the opposite’s ”
\n “embedding layer.”
\n )<\/p>\n
\n increase RuntimeError(
\n f”Tokenizer vocabulary mismatch between {reference_model_id} ”
\n f”and {candidate_model_id}: not less than one token string maps ”
\n “to a unique integer id between the 2. Direct token ”
\n “injection throughout these fashions would silently corrupt ”
\n “downstream generations.”
\n )<\/p>\n
\n increase RuntimeError(
\n f”Particular-tokens map mismatch between {reference_model_id} ”
\n f”({reference_tokenizer.special_tokens_map}) and ”
\n f”{candidate_model_id} ({candidate_tokenizer.special_tokens_map}).”
\n )<\/p>\n6. The receipts<\/h2>\n
\n Metric 1 (TTFT discount): mean_baseline=69.3 ms, mean_injection=49.9 ms, discount=28.0% — PASS
\n Metric 2 (semantic constancy): PASS<\/p>\n
\n Metric 1 (TTFT discount): mean_baseline=49.6 ms, mean_injection=30.9 ms, discount=37.8% — PASS
\n Metric 2 (semantic constancy): PASS<\/p>\n![\"A](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/08\/benchmark_bar-1024x683.png\")
\n
7. Wrap: the truly fascinating half was the guardrail<\/h2>\n