\n
\n
![\"arXiv\"\/](\"https:\/\/img.shields.io\/badge\/arXiv-2609.01657-b31b1b.svg?style=for-the-badge\")
<\/p>\n\n ![\"NeoMME](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8\/assets\/neomme\/neomme_logo.webp?raw=true\"){kind=logo}
\n<\/p>\n
\n
\t We introduce NeoMME, a household of 260M and 800M multilingual multimodal encoders. In contrast to many generative visible language fashions, NeoMME doesn’t use a separate pretrained imaginative and prescient tower or a causal language mannequin. A single bidirectional Transformer processes each textual content tokens and uncooked picture patches, and we practice the complete mannequin from scratch with a masked discrete-diffusion goal.<\/p>\n We fine-tuned NeoMME for visible doc retrieval utilizing ColPali’s page-image strategy. NeoMME-Retriever returns dense and late-interaction embeddings in a single ahead move. Each mannequin sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and mannequin dimension. At a matched 2048\u00d72048 picture enter dimension on an NVIDIA L40S GPU, the 260M mannequin encodes about 51 pages per second, or about twice ColModernVBERT’s throughput. Hierarchical token pooling and uneven quantization scale back late-interaction index storage from roughly 1.5 MB to six kB per web page (255\u00d7 smaller) whereas retaining greater than 95% of baseline nDCG@10.<\/p>\n NeoMME is out there in Hugging Face Transformers. We launch all mannequin checkpoints beneath the Apache 2.0 license.<\/p>\n \t Many current visible doc retrievers are tailored from pretrained generative visible language fashions. A individually pretrained imaginative and prescient encoder produces visible options, which a projector maps into the language mannequin’s enter area. A causal decoder then processes the mixed picture and textual content representations. Retrieval, classification, and token labeling don’t generate textual content autoregressively, so they don’t require a causal decoder or the parameter and compute overhead of this structure.<\/p>\n ModernBERT introduced environment friendly structure and coaching enhancements to bidirectional encoders. For visible doc retrieval, ModernVBERT utilized a bidirectional ModernBERT-style textual content encoder whereas retaining a separate pretrained SigLIP2 imaginative and prescient tower. We needed to push this even additional by designing and coaching a multimodal encoder with out having to hold over the parameter and compute overhead of a VLM.<\/p>\n NeoMME (pronounced “nee-oh-me”, IPA \/\u02c8ni\u02d0.o\u028a.mi\/) is a multilingual, multimodal basis encoder that generates vector representations for enter textual content and\/or photographs utilizing a single Transformer encoder. It’s not based mostly on an present pretrained imaginative and prescient tower, textual content encoder, or textual content decoder.<\/p>\n Pictures and textual content use the identical computational path, so NeoMME can extra simply assist pretraining, fine-tuning, parallelization, and serving throughout each modalities.<\/p>\n \t \t NeoMME is available in two sizes, 260M and 800M. Each variants share the identical structure:<\/p>\n Native multimodal inputs: textual content inputs use factorized token embeddings, whereas photographs are divided right into a grid of non-overlapping 32\u00d732 patches and projected with a small MLP. Each enter the identical Transformer encoder. \t We pretrain NeoMME from scratch as a discrete masked-diffusion textual content denoiser. For every text-only instance, we pattern a corruption charge uniformly between 0 and 1. Every eligible textual content token is then independently masked at that charge.<\/p>\n Multimodal examples use corruption charges between 0.3 and 1. The picture patches stay seen whereas NeoMME reconstructs masked textual content. With gentle masking, the mannequin can typically recuperate a lacking phrase from the encircling textual content alone. For instance, “cat” is a believable completion of “The [MASK] sat on the mat,” even with out a picture. However excessive masking forces the mannequin to study image-grounded descriptions with little to no sign from the non-masked enter textual content tokens.<\/p>\n Pretraining mixes multilingual textual content, code, arithmetic, pure photographs, and doc photographs. Every mannequin processes about 524 billion packed enter tokens, together with 290 billion tokens from text-only examples. This textual content price range is comparatively small in contrast with ModernBERT’s 2 trillion coaching token price range. Therefore, we selected the NorMuon optimizer to enhance knowledge effectivity throughout coaching.<\/p>\n \t To get a significant downstream analysis of the spine, we fine-tune NeoMME for visible doc retrieval utilizing the page-image methodology launched by ColPali. Whereas conventional text-based retrieval consists of retrieving textual content chunks, NeoMME-Retriever ranks doc web page screenshots and bypasses all of the preprocessing OCR steps essential to extract textual content from PDFs. Treating the pages as photographs preserves format, charts, tables, font sort and dimension, and different visible clues that can not be captured even by an ideal OCR mannequin.<\/p>\n \t NeoMME-Retriever reuses the NeoMME spine however provides two collectively educated heads on high of it for retrieval:<\/p>\n The dense head averages the spine’s hidden state vectors right into a normalized vector (imply pooling). Dense embeddings are most typical at the moment: they’re compact and work naturally with approximate nearest-neighbor (ANN) strategies for quick retrieval. Omar Khattab, who launched late-interaction in ColBERT, explains why the time period is extra exact than “multi-vector.” It describes the granularity and learnability of the scoring operate, not merely the variety of saved vectors.<\/p>\n To study extra about late-interaction, we advocate studying this crash course by Am\u00e9lie Chatelain.<\/p>\n<\/blockquote>\n One NeoMME-Retriever ahead move returns each representations, which supplies you flexibility irrespective of your use case and infrastructure. We advocate utilizing late-interaction embeddings on the whole since they’re extra highly effective and can be utilized simply with open-source libraries like NextPlaid. Nevertheless, when you’ve got a really massive corpora, you’ll be able to run a single ahead move with NeoMME-Retriever to get the dense embedding, retrieve a small variety of paperwork via an ANN index, after which use late-interaction to rerank the retrieved candidates.<\/p>\n \t We report nDCG@10 on ViDoRe v3. NeoMME-Retriever-260M reaches 0.523, the best rating amongst evaluated fashions strictly beneath 800M parameters. It’s inside 0.002 nDCG@10 of ColQwen2.5 whereas utilizing about 14\u00d7 fewer parameters. NeoMME-Retriever-800M reaches 0.556, inside 0.009 nDCG@10 of the equally sized Vultron Retriever Flash (0.8B). Each NeoMME-Retriever fashions lie on the model-size Pareto frontier.<\/p>\n ViDoRe v1 and v2 use nDCG@5. On each benchmarks, NeoMME-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. NeoMME-Retriever-800M outperforms ColPali v1.3 whereas utilizing 3.6 occasions fewer parameters.<\/p>\n \u2020 Scores from MTEB. \u2021 Outcomes from our personal evaluations.<\/p>\n \t Late-interaction storage scales linearly with the variety of vectors within the output embedding. Increased-resolution photographs include extra patches, so that they produce bigger embeddings. For instance, a 2048\u00d72048 sq. web page produces embeddings containing 4,200 vectors with NeoMME-Retriever, or about 2.1 MB in float32. Throughout the ViDoRe v3 benchmark, the measured common is about 1.5 MB per doc.<\/p>\n To scale back the storage footprint of the late-interaction index, we mix two complementary compression strategies:<\/p>\n Hierarchical token pooling clusters related doc vectors in a given multi-vector embedding and replaces every cluster with its imply, therefore decreasing the variety of vectors saved for every web page. We examined this setup on ViDoRe v3. With a pooling issue 10 and int8 queries and paperwork, storage decreased from about 1.5 MB to 39 kB per web page, a 39\u00d7 discount, whereas maintaining greater than 99% of the baseline nDCG@10. A extra aggressive configuration makes use of pooling issue 8, int8 queries, and binary paperwork. That model makes use of 6 kB per web page (255\u00d7 smaller) and retains greater than 95% of the unique retrieval high quality.<\/p>\n Customers can decide a compression setting from that frontier based mostly on storage price range and required retrieval high quality.<\/p>\n \t Earlier than you’ll be able to search a corpus, a retriever mannequin should flip your paperwork into embeddings, which can be saved in a vector retailer like Qdrant, Weaviate, or Milvus. Sooner encoding makes constructing and including new paperwork to the index quicker, thus decreasing the GPU uptime and compute value required.<\/p>\n So we measured picture encoding speeds for NeoMME-Retriever in opposition to different multimodal doc retrievers. We used preprocessed picture tensors and calibrated the batch dimension individually for every mannequin and picture dimension. At a matched 2048\u00d72048 enter dimension on one NVIDIA L40S, NeoMME-Retriever-260M encodes about 51 pages per second, practically twice ColModernVBERT’s 26 pages per second. Each 260M and 800M NeoMME-Retriever fashions are additionally quicker than the opposite fashions we in contrast on smaller enter photographs.<\/p>\n \t NeoMME-Retriever (260M and 800M) returns dense and multi-vector embeddings collectively. The instance beneath scores two textual content queries in opposition to two document-page photographs with MeanMaxSim late interplay and dense cosine similarity.<\/p>\n Click on to see the entire \ud83e\udd17 transformers instance snippet<\/p>\n pip set up -U speed up “transformers @ git+https:\/\/github.com\/huggingface\/transformers.git@important”<\/span> “sentence-transformers>=6.0.0”<\/span><\/p>\n from<\/span> typing import<\/span> Any<\/span>, Literal<\/span><\/p>\n import<\/span> requests from<\/span> transformers import<\/span> BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor<\/p>\n def<\/span> encode<\/span>( model_name = “Hcompany\/NeoMME-260M-Retriever”<\/span> image_urls = [ queries = [ document_messages = [ inputs_documents = encode(document_messages, “doc”<\/span>).to(mannequin.gadget) with<\/span> torch.inference_mode(): late_scores = mean_maxsim( print<\/span>(late_scores, dense_scores)<\/p>\n \t We offer separate dense and late-interaction checkpoints for fine-tuning with Sentence Transformers v6. Following the identical sample as textual content encoders equivalent to ModernBERT, Sentence Transformers masses the spine via NeoMMEModel fairly than the dual-head NeoMMEForRetrieval class. Sentence Transformers at present helps one retrieval head per mannequin, so every checkpoint allows you to fine-tune the dense or late-interaction head independently. To coach each heads collectively, use NeoMMEForRetrieval with a customized Coach.<\/p>\n \t Visible doc retrieval can be utilized as the primary stage of a visible retrieval-augmented era (RAG) system. In contrast to textual content RAG, which retrieves extracted textual content chunks, visible RAG retrieves the unique web page photographs and sends them to a visible language mannequin. The mannequin can then use tables, plots, diagrams, and web page format that textual content extraction might flatten or omit. Right here is how visible RAG works:<\/p>\n Indexing: Convert every PDF web page to a picture, generate an embedding with a retrieval mannequin, and retailer the embeddings in a vector retailer. You may take a look at visible RAG straight with NeoMME-Retriever in our HF Area: \ud83e\udd17 tonywu71\/neomme-retriever-demo.<\/p>\n \t NeoMME replaces separate pretrained picture and textual content encoders with one long-context bidirectional Transformer. We practice it from scratch to course of each multilingual textual content tokens and uncooked 32\u00d732 picture patches.<\/p>\n NeoMME-Retriever is a fine-tuned model of NeoMME for visible doc retrieval. One ahead move produces each dense and late-interaction representations. The 260M mannequin outperforms all evaluated fashions strictly beneath 800M parameters and, at a matched 2048\u00d72048 enter dimension, encodes pages at about 2\u00d7 ColModernVBERT’s throughput. To scale back the massive storage footprint of late-interaction embeddings for high-resolution paperwork, we experimented with hierarchical token pooling and uneven quantization and managed to scale back the late-interaction embeddings from roughly 1.5 MB to six kB per web page, a 255\u00d7 compression, whereas retaining greater than 95% of the baseline nDCG@10.<\/p>\n We launch all NeoMME mannequin checkpoints and a day-zero Hugging Face Transformers implementation to permit practitioners to construct environment friendly multimodal and multilingual illustration fashions on high of our work.<\/p>\n
\n\t\tTL;DR
\n\t<\/span>
\n<\/h2>\n\n
\n\t\tWhy one other multimodal encoder?
\n\t<\/span>
\n<\/h2>\n![\"Comparison](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa\/assets\/neomme\/architecture.webp?raw=true\")
\n
\n\t\tNeoMME encoder spine
\n\t<\/span>
\n<\/h2>\n\n
\n\t\tOne Transformer for photographs and textual content
\n\t<\/span>
\n<\/h3>\n
\nDynamic picture decision: photographs maintain their facet ratio and dimension. This enables the mannequin to make use of extra tokens on a high-resolution, information-dense doc web page than on a smaller picture with much less content material.
\nLengthy bidirectional context: each fashions have a context size of 16,384 tokens (sufficient for as much as two normal 3840\u00d72160 4K UHD photographs). Most layers use symmetric sliding-window consideration, whereas each sixth layer and the ultimate layer use international consideration.
\nA contemporary encoder stack: NeoMME makes use of current encoder enhancements equivalent to grouped-query consideration, query-key normalization, gated consideration, 2D rotary place embeddings, and squared-ReLU MLPs, amongst others.
\nMultilingual textual content: we educated a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual textual content, code, arithmetic, and machine-produced picture transcripts.<\/p>\n![\"Alternating](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa\/assets\/neomme\/encoder-stack.webp?raw=true\")
\n
\n\t\tStudying from photographs via masked textual content
\n\t<\/span>
\n<\/h3>\n![\"Effect](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa\/assets\/neomme\/masked-diffusion.webp?raw=true\")
\n
\n\t\tNeoMME-Retriever
\n\t<\/span>
\n<\/h2>\n\n
\n\t\tA dual-head design for dense and late-interaction retrieval
\n\t<\/span>
\n<\/h3>\n
\nThe late-interaction head tasks every textual content token or picture patch from the spine’s output hidden states to a 128-dimensional normalized vector. In comparison with dense embeddings, the finer granularity preserves native matches between particular person question tokens and picture areas.<\/p>\n![\"Late-interaction](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa\/assets\/neomme\/retrieval-heads.webp?raw=true\")
\n
\n
\n\t\tAggressive retrieval at compact mannequin sizes
\n\t<\/span>
\n<\/h3>\n![\"ViDoRe](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/c5e737e7e474363822b6f2d95290c0120e0fa993\/assets\/neomme\/vidore-v3-model-size.webp?raw=true\")
\n
\n\t\tMaking high-resolution retrieval sensible for late-interaction
\n\t<\/span>
\n<\/h3>\n
\nUneven quantization quantizes doc embeddings to int8 or binary. As a result of question embeddings will not be saved and solely generated on-the-fly, they are often stored at a better precision.<\/p>\n![\"Quality](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa\/assets\/neomme\/compression-frontier-260m.webp?raw=true\")
\n
\n\t\tQuick inference for cheaper multimodal corpus indexing
\n\t<\/span>
\n<\/h4>\n![\"Document-encoding](\"https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/c5e737e7e474363822b6f2d95290c0120e0fa993\/assets\/neomme\/indexing-throughput-l40s.webp?raw=true\")
\n
\n\t\tAttempt NeoMME-Retriever your self!
\n\t<\/span>
\n<\/h3>\n
\nimport<\/span> torch
\nfrom<\/span> PIL import<\/span> Picture
\nfrom<\/span> sentence_transformers.util import<\/span> cos_sim, mean_maxsim<\/p>\n
\n messages: record<\/span>[list<\/span>[dict<\/span>[str<\/span>, Any<\/span>]]],<\/span>
\n activity: Literal<\/span>[“query”<\/span>, “document”<\/span>],<\/span>
\n) -> BatchFeature:
\n return<\/span> processor.apply_chat_template(
\n messages,
\n activity=activity,
\n tokenize=True<\/span>,
\n return_dict=True<\/span>,
\n return_tensors=“pt”<\/span>,
\n processor_kwargs={“padding”<\/span>: “longest”<\/span>},
\n )<\/p>\n
\nprocessor = NeoMMEProcessor.from_pretrained(model_name)
\nmannequin = NeoMMEForRetrieval.from_pretrained(model_name, device_map=“auto”<\/span>)<\/p>\n
\n “https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8\/examples\/data\/shift_kazakhstan.jpg?raw=true”<\/span>,
\n “https:\/\/github.com\/tonywu71\/colpali-cookbooks\/blob\/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8\/examples\/data\/energy_electricity_generation.jpg?raw=true”<\/span>,
\n]
\npaperwork = [Image.open<\/span>(requests.get(url, stream=True<\/span>).raw) for<\/span> url in<\/span> image_urls]<\/p>\n
\n “Quelle partie de la production p\u00e9troli\u00e8re du Kazakhstan provient de champs en mer ?”<\/span>,
\n “Which hour of the day had the highest overall electricity generation in 2019?”<\/span>,
\n]<\/p>\n
\n [{“role”<\/span>: “user”<\/span>, “content”<\/span>: [{“type”<\/span>: “image”<\/span>, “image”<\/span>: document}]}] for<\/span> doc in<\/span> paperwork
\n]
\nquery_messages = [[{“role”<\/span>: “user”<\/span>, “content”<\/span>: query}] for<\/span> question in<\/span> queries]<\/p>\n
\ninputs_text = encode(query_messages, “question”<\/span>).to(mannequin.gadget)<\/p>\n
\n document_outputs = mannequin(**inputs_documents)
\n query_outputs = mannequin(**inputs_text)<\/p>\n
\n query_outputs.embeddings,
\n document_outputs.embeddings,
\n a_mask=inputs_text[“attention_mask”<\/span>],
\n b_mask=inputs_documents[“attention_mask”<\/span>],
\n)
\ndense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)<\/p>\n\n
\n\t\tFantastic-tuning with Sentence Transformers
\n\t<\/span>
\n<\/h3>\n\n
\n\t\tFrom retrieval to visible RAG
\n\t<\/span>
\n<\/h3>\n
\nRetrieval: Generate an embedding for the consumer’s question with the identical mannequin and retrieve the top-k most related pages.
\nEra: Append the photographs after the question within the chat message (e.g., {question}{img_1}{img_2}…{img_k}) and ship it to a VLM to generate the reply.<\/p>\n\n
\n\t\tConclusion
\n\t<\/span>
\n<\/h2>\n\n