TL;DR
We introduce NeoMME, a household of 260M and 800M multilingual multimodal encoders. In contrast to many generative visible language fashions, NeoMME doesn’t use a separate pretrained imaginative and prescient tower or a causal language mannequin. A single bidirectional Transformer processes each textual content tokens and uncooked picture patches, and we practice the complete mannequin from scratch with a masked discrete-diffusion goal.
We fine-tuned NeoMME for visible doc retrieval utilizing ColPali’s page-image strategy. NeoMME-Retriever returns dense and late-interaction embeddings in a single ahead move. Each mannequin sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and mannequin dimension. At a matched 2048×2048 picture enter dimension on an NVIDIA L40S GPU, the 260M mannequin encodes about 51 pages per second, or about twice ColModernVBERT’s throughput. Hierarchical token pooling and uneven quantization scale back late-interaction index storage from roughly 1.5 MB to six kB per web page (255× smaller) whereas retaining greater than 95% of baseline nDCG@10.
NeoMME is out there in Hugging Face Transformers. We launch all mannequin checkpoints beneath the Apache 2.0 license.
Why one other multimodal encoder?
Many current visible doc retrievers are tailored from pretrained generative visible language fashions. A individually pretrained imaginative and prescient encoder produces visible options, which a projector maps into the language mannequin’s enter area. A causal decoder then processes the mixed picture and textual content representations. Retrieval, classification, and token labeling don’t generate textual content autoregressively, so they don’t require a causal decoder or the parameter and compute overhead of this structure.
ModernBERT introduced environment friendly structure and coaching enhancements to bidirectional encoders. For visible doc retrieval, ModernVBERT utilized a bidirectional ModernBERT-style textual content encoder whereas retaining a separate pretrained SigLIP2 imaginative and prescient tower. We needed to push this even additional by designing and coaching a multimodal encoder with out having to hold over the parameter and compute overhead of a VLM.
NeoMME (pronounced “nee-oh-me”, IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal basis encoder that generates vector representations for enter textual content and/or photographs utilizing a single Transformer encoder. It’s not based mostly on an present pretrained imaginative and prescient tower, textual content encoder, or textual content decoder.
Pictures and textual content use the identical computational path, so NeoMME can extra simply assist pretraining, fine-tuning, parallelization, and serving throughout each modalities.
NeoMME encoder spine
One Transformer for photographs and textual content
NeoMME is available in two sizes, 260M and 800M. Each variants share the identical structure:
Native multimodal inputs: textual content inputs use factorized token embeddings, whereas photographs are divided right into a grid of non-overlapping 32×32 patches and projected with a small MLP. Each enter the identical Transformer encoder.
Dynamic picture decision: photographs maintain their facet ratio and dimension. This enables the mannequin to make use of extra tokens on a high-resolution, information-dense doc web page than on a smaller picture with much less content material.
Lengthy bidirectional context: each fashions have a context size of 16,384 tokens (sufficient for as much as two normal 3840×2160 4K UHD photographs). Most layers use symmetric sliding-window consideration, whereas each sixth layer and the ultimate layer use international consideration.
A contemporary encoder stack: NeoMME makes use of current encoder enhancements equivalent to grouped-query consideration, query-key normalization, gated consideration, 2D rotary place embeddings, and squared-ReLU MLPs, amongst others.
Multilingual textual content: we educated a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual textual content, code, arithmetic, and machine-produced picture transcripts.
Studying from photographs via masked textual content
We pretrain NeoMME from scratch as a discrete masked-diffusion textual content denoiser. For every text-only instance, we pattern a corruption charge uniformly between 0 and 1. Every eligible textual content token is then independently masked at that charge.
Multimodal examples use corruption charges between 0.3 and 1. The picture patches stay seen whereas NeoMME reconstructs masked textual content. With gentle masking, the mannequin can typically recuperate a lacking phrase from the encircling textual content alone. For instance, “cat” is a believable completion of “The [MASK] sat on the mat,” even with out a picture. However excessive masking forces the mannequin to study image-grounded descriptions with little to no sign from the non-masked enter textual content tokens.
Pretraining mixes multilingual textual content, code, arithmetic, pure photographs, and doc photographs. Every mannequin processes about 524 billion packed enter tokens, together with 290 billion tokens from text-only examples. This textual content price range is comparatively small in contrast with ModernBERT’s 2 trillion coaching token price range. Therefore, we selected the NorMuon optimizer to enhance knowledge effectivity throughout coaching.
NeoMME-Retriever
To get a significant downstream analysis of the spine, we fine-tune NeoMME for visible doc retrieval utilizing the page-image methodology launched by ColPali. Whereas conventional text-based retrieval consists of retrieving textual content chunks, NeoMME-Retriever ranks doc web page screenshots and bypasses all of the preprocessing OCR steps essential to extract textual content from PDFs. Treating the pages as photographs preserves format, charts, tables, font sort and dimension, and different visible clues that can not be captured even by an ideal OCR mannequin.
A dual-head design for dense and late-interaction retrieval
NeoMME-Retriever reuses the NeoMME spine however provides two collectively educated heads on high of it for retrieval:
The dense head averages the spine’s hidden state vectors right into a normalized vector (imply pooling). Dense embeddings are most typical at the moment: they’re compact and work naturally with approximate nearest-neighbor (ANN) strategies for quick retrieval.
The late-interaction head tasks every textual content token or picture patch from the spine’s output hidden states to a 128-dimensional normalized vector. In comparison with dense embeddings, the finer granularity preserves native matches between particular person question tokens and picture areas.
Omar Khattab, who launched late-interaction in ColBERT, explains why the time period is extra exact than “multi-vector.” It describes the granularity and learnability of the scoring operate, not merely the variety of saved vectors.
To study extra about late-interaction, we advocate studying this crash course by Amélie Chatelain.
One NeoMME-Retriever ahead move returns each representations, which supplies you flexibility irrespective of your use case and infrastructure. We advocate utilizing late-interaction embeddings on the whole since they’re extra highly effective and can be utilized simply with open-source libraries like NextPlaid. Nevertheless, when you’ve got a really massive corpora, you’ll be able to run a single ahead move with NeoMME-Retriever to get the dense embedding, retrieve a small variety of paperwork via an ANN index, after which use late-interaction to rerank the retrieved candidates.
Aggressive retrieval at compact mannequin sizes
We report nDCG@10 on ViDoRe v3. NeoMME-Retriever-260M reaches 0.523, the best rating amongst evaluated fashions strictly beneath 800M parameters. It’s inside 0.002 nDCG@10 of ColQwen2.5 whereas utilizing about 14× fewer parameters. NeoMME-Retriever-800M reaches 0.556, inside 0.009 nDCG@10 of the equally sized Vultron Retriever Flash (0.8B). Each NeoMME-Retriever fashions lie on the model-size Pareto frontier.
ViDoRe v1 and v2 use nDCG@5. On each benchmarks, NeoMME-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. NeoMME-Retriever-800M outperforms ColPali v1.3 whereas utilizing 3.6 occasions fewer parameters.
† Scores from MTEB. ‡ Outcomes from our personal evaluations.
Making high-resolution retrieval sensible for late-interaction
Late-interaction storage scales linearly with the variety of vectors within the output embedding. Increased-resolution photographs include extra patches, so that they produce bigger embeddings. For instance, a 2048×2048 sq. web page produces embeddings containing 4,200 vectors with NeoMME-Retriever, or about 2.1 MB in float32. Throughout the ViDoRe v3 benchmark, the measured common is about 1.5 MB per doc.
To scale back the storage footprint of the late-interaction index, we mix two complementary compression strategies:
Hierarchical token pooling clusters related doc vectors in a given multi-vector embedding and replaces every cluster with its imply, therefore decreasing the variety of vectors saved for every web page.
Uneven quantization quantizes doc embeddings to int8 or binary. As a result of question embeddings will not be saved and solely generated on-the-fly, they are often stored at a better precision.
We examined this setup on ViDoRe v3. With a pooling issue 10 and int8 queries and paperwork, storage decreased from about 1.5 MB to 39 kB per web page, a 39× discount, whereas maintaining greater than 99% of the baseline nDCG@10. A extra aggressive configuration makes use of pooling issue 8, int8 queries, and binary paperwork. That model makes use of 6 kB per web page (255× smaller) and retains greater than 95% of the unique retrieval high quality.
Customers can decide a compression setting from that frontier based mostly on storage price range and required retrieval high quality.
Quick inference for cheaper multimodal corpus indexing
Earlier than you’ll be able to search a corpus, a retriever mannequin should flip your paperwork into embeddings, which can be saved in a vector retailer like Qdrant, Weaviate, or Milvus. Sooner encoding makes constructing and including new paperwork to the index quicker, thus decreasing the GPU uptime and compute value required.
So we measured picture encoding speeds for NeoMME-Retriever in opposition to different multimodal doc retrievers. We used preprocessed picture tensors and calibrated the batch dimension individually for every mannequin and picture dimension. At a matched 2048×2048 enter dimension on one NVIDIA L40S, NeoMME-Retriever-260M encodes about 51 pages per second, practically twice ColModernVBERT’s 26 pages per second. Each 260M and 800M NeoMME-Retriever fashions are additionally quicker than the opposite fashions we in contrast on smaller enter photographs.
Attempt NeoMME-Retriever your self!
NeoMME-Retriever (260M and 800M) returns dense and multi-vector embeddings collectively. The instance beneath scores two textual content queries in opposition to two document-page photographs with MeanMaxSim late interplay and dense cosine similarity.
Click on to see the entire 🤗 transformers instance snippet
pip set up -U speed up “transformers @ git+https://github.com/huggingface/transformers.git@important” “sentence-transformers>=6.0.0”
from typing import Any, Literal
import requests
import torch
from PIL import Picture
from sentence_transformers.util import cos_sim, mean_maxsim
from transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor
def encode(
messages: record[list[dict[str, Any]]],
activity: Literal[“query”, “document”],
) -> BatchFeature:
return processor.apply_chat_template(
messages,
activity=activity,
tokenize=True,
return_dict=True,
return_tensors=“pt”,
processor_kwargs={“padding”: “longest”},
)
model_name = “Hcompany/NeoMME-260M-Retriever”
processor = NeoMMEProcessor.from_pretrained(model_name)
mannequin = NeoMMEForRetrieval.from_pretrained(model_name, device_map=“auto”)
image_urls = [
“https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true”,
“https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true”,
]
paperwork = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]
queries = [
“Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?”,
“Which hour of the day had the highest overall electricity generation in 2019?”,
]
document_messages = [
[{“role”: “user”, “content”: [{“type”: “image”, “image”: document}]}] for doc in paperwork
]
query_messages = [[{“role”: “user”, “content”: query}] for question in queries]
inputs_documents = encode(document_messages, “doc”).to(mannequin.gadget)
inputs_text = encode(query_messages, “question”).to(mannequin.gadget)
with torch.inference_mode():
document_outputs = mannequin(**inputs_documents)
query_outputs = mannequin(**inputs_text)
late_scores = mean_maxsim(
query_outputs.embeddings,
document_outputs.embeddings,
a_mask=inputs_text[“attention_mask”],
b_mask=inputs_documents[“attention_mask”],
)
dense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)
print(late_scores, dense_scores)
Fantastic-tuning with Sentence Transformers
We offer separate dense and late-interaction checkpoints for fine-tuning with Sentence Transformers v6. Following the identical sample as textual content encoders equivalent to ModernBERT, Sentence Transformers masses the spine via NeoMMEModel fairly than the dual-head NeoMMEForRetrieval class. Sentence Transformers at present helps one retrieval head per mannequin, so every checkpoint allows you to fine-tune the dense or late-interaction head independently. To coach each heads collectively, use NeoMMEForRetrieval with a customized Coach.
From retrieval to visible RAG
Visible doc retrieval can be utilized as the primary stage of a visible retrieval-augmented era (RAG) system. In contrast to textual content RAG, which retrieves extracted textual content chunks, visible RAG retrieves the unique web page photographs and sends them to a visible language mannequin. The mannequin can then use tables, plots, diagrams, and web page format that textual content extraction might flatten or omit. Right here is how visible RAG works:
Indexing: Convert every PDF web page to a picture, generate an embedding with a retrieval mannequin, and retailer the embeddings in a vector retailer.
Retrieval: Generate an embedding for the consumer’s question with the identical mannequin and retrieve the top-k most related pages.
Era: Append the photographs after the question within the chat message (e.g., {question}{img_1}{img_2}…{img_k}) and ship it to a VLM to generate the reply.
You may take a look at visible RAG straight with NeoMME-Retriever in our HF Area: 🤗 tonywu71/neomme-retriever-demo.
Conclusion
NeoMME replaces separate pretrained picture and textual content encoders with one long-context bidirectional Transformer. We practice it from scratch to course of each multilingual textual content tokens and uncooked 32×32 picture patches.
NeoMME-Retriever is a fine-tuned model of NeoMME for visible doc retrieval. One ahead move produces each dense and late-interaction representations. The 260M mannequin outperforms all evaluated fashions strictly beneath 800M parameters and, at a matched 2048×2048 enter dimension, encodes pages at about 2× ColModernVBERT’s throughput. To scale back the massive storage footprint of late-interaction embeddings for high-resolution paperwork, we experimented with hierarchical token pooling and uneven quantization and managed to scale back the late-interaction embeddings from roughly 1.5 MB to six kB per web page, a 255× compression, whereas retaining greater than 95% of the baseline nDCG@10.
We launch all NeoMME mannequin checkpoints and a day-zero Hugging Face Transformers implementation to permit practitioners to construct environment friendly multimodal and multilingual illustration fashions on high of our work.
Acknowledgements
NeoMME started as a facet quest between two good associates. We labored with restricted time and compute, and we determined to share the outcomes so the neighborhood can construct on them. We thank H Firm for supporting the work and offering the compute used to coach NeoMME.
Quotation
@misc{lac2026neommesingletowermultimodalnativemultilingual,
title={NeoMME: A Single-Tower Multimodal-Native Multilingual Basis Encoder for Environment friendly Fantastic-Tuning and Inference},
writer={Aurélien Lac and Tony Wu},
yr={2026},
eprint={2609.01657},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2609.01657},
}
