This submit was co-authored with Max Silfverberg (Information Scientist, AI Options Lead), Antti Hallavo (Lead AI Software program Engineer), and Pontus Huotari (Lead Information Scientist). We work at Alma Media, a Finnish digital providers, marketplaces and media firm. Considered one of our focus areas is creating AI/ML options for actual property itemizing providers, the place understanding picture content material performs an essential position.
providers deal with tons of of 1000’s of listings a 12 months. Most of these include dozens of photographs with no details about what they present. In the meantime, search, suggestions, and a spread of inside use circumstances all profit from understanding whether or not a photograph represents a kitchen, ground plan, or backyard.
Our answer is to routinely tag photographs with room-type and content material lessons. Our room varieties embody LIVING ROOM, KITCHEN, and BEDROOM. We additionally tag schematic content material like ground plans and web site plans. Moreover, we acknowledge realtor advertising supplies, aerial photographs, and backyard photographs. Altogether, there are 23 lessons. As Determine 1 exhibits, this can be a traditional multi-label classification activity; the identical area can embody a number of room varieties without delay.
On the face of it, this sounds easy, however we have to make some tough choices. How do you have to deal with a front room picture that exhibits a bed room by a doorway? What if the picture solely exhibits 10% front room and the remaining 90% is eating space? The solutions rely upon the applying.
If we have to discover all photographs exhibiting kitchens, we additionally need to determine front room photographs that present a kitchen within the background. Nonetheless, if the person particularly asks for kitchen photographs, we solely need to present those the place the kitchen is in focus. To assist resolve what to return, classification confidence is essential. However relying on the way you implement your classifier, you would possibly not have entry to that info.
Picture classifiers may be inbuilt many methods. The trendy default method is to run photos by a third-party API which internally makes use of a vision-language mannequin (VLM) to investigate photos and generate tags in line with a immediate.
One other possibility is to coach picture classifiers on high of open-source ViT basis fashions like Google SigLIP and Meta DINO, both freezing the inspiration mannequin or fine-tuning it. Every of those designs comes with its personal benefits and trade-offs.
There already exists lots of work evaluating the approaches based mostly on numerical efficiency [1]. This weblog submit goes additional; we ask the generally ignored query: How do you have to construct picture classifiers in a enterprise context?
Three questions earlier than you practice something
We constructed our proprietary classifiers by fine-tuning google/siglip-base-patch16-224. The query is: why do this? Examine Determine 2 for the TL;DR. Learn on for the total story.
Query 1: Immediate an API or practice your personal mannequin?
The selection to categorise by prompting by an exterior API or construct your personal classifier closely relies on your use case. First, you have to think about whether or not your classification activity may even be prompted. It’s simple to immediate automobile and kitchen equipment classifiers however how about click-through charge (CTR) for YouTube video thumbnails? Right here we want a trainable classifier as a result of we actually don’t know what influences the click on determination. Conversely, if you want to extract structured JSON recordsdata from photographs representing constructing schematics, a easy classifier simply received’t lower it.
Our actual property use case sits within the center. Many lessons like KITCHEN and BATHROOM are simply promptable whereas others, like HALLWAY, LOFT and ALCOVE are fuzzier and tougher to verbalize.
When you resolve to coach your personal classifiers as we did, you in fact want coaching knowledge, in all probability not less than just a few thousand examples per class. When launching a brand new product, that’s one thing you may not have. When you not less than have entry to plain photographs with out annotations,you possibly can launch with aprompted VLM as your first classifier. Its predictions steadily accumulate into an annotated dataset, which you’ll later use to coach a customized classifier. This will likely require a cleanup go, because the dataset inherits the VLM’s errors.
Value is one other main query. With a quantity within the tens of millions, the totally different classification approaches end in dramatically divergent value profiles. Utilizing Google’s Agent Platform and the gemini-3.5-flash mannequin, the July 2026 worth is roughly $1.50 per 1,000 photos (at 1K decision), so classifying one million photographs prices roughly $1,500.
Utilizing our personal classifier on a devoted AWS EC2 g4dn.xlarge occasion with a T4 GPU, we will classify not less than 400 photos per second. At a July 2026 on-demand hourly charge of $0.53, classifying one million inputs comes out to $0.37 or roughly 1/4000th of the value for the API answer (inference compute solely).
Nonetheless, if you classify just a few hundred photographs a day, from the price perspective it actually doesn’t matter the way you do it. Prices change into a problem solely at scale.
Along with labels, classification confidence is usually helpful. As talked about above, if we provide kitchen photographs to the person, we should always in all probability go together with assured matches. It’s, nonetheless, tough to derive dependable confidence estimates from a VLM; verbalized confidence estimates are identified to be poorly calibrated [2] and token log-likelihoods from an API often don’t symbolize the class-probabilities you’re really considering.
When you use an API, you would possibly due to this fact must depend on granular tags like PROBABLE/POSSIBLE/UNLIKELY [3], and there’s no assure that these might be dependable both. When you as an alternative practice your personal classifier, you get usable per-class scores which may be calibrated when wanted. Desk 1 summarizes how the 2 approaches examine.
Query 2: Which basis mannequin to use?
When you resolve to coach your personal classifier, the one affordable alternative for many is to start out with a pretrained open-source imaginative and prescient mannequin, sometimes a imaginative and prescient transformer. For enterprise use, first test that the mannequin’s license permits business use.
Past that, your enterprise objective ought to drive the selection, as a result of totally different pretraining methods produce totally different representations:
SigLIP [4] (Google) is educated on captioned photos, so it attends to caption-worthy issues: canine, vehicles, individuals. Its representations are extremely object-oriented; background and digital camera angle obtain far much less emphasis.
DINO [5, 6] (Meta) is self-supervised with patch-level goals: each area of the picture contributes to the loss, not simply the caption-worthy objects. That makes it a robust candidate when background or format issues [7]. We put this to the take a look at under.
RADIO / AM-RADIO [8] (NVIDIA) agglomerate representations from a number of ViT basis fashions by distillation.
I-JEPA[9] Meta) is self-supervised like DINO however based mostly on masked prediction.
Query 3: To fine-tune or to not fine-tune?
The only strategy to begin is coaching a linear classifier on high of frozen ViT representations. There are two main benefits: it’s conceptually easy and lightning quick. You’ll be able to practice on a laptop computer in a matter of minutes utilizing a 100k-instance coaching set. Sometimes, this results in very affordable efficiency.
When you resolve to fine-tune, one of the best observe is to make use of low-rank adapters (LoRA), which freeze the precise ViT spine and inject just a few skinny trainable parameter layers into the mannequin [10]. After coaching, these may be merged with the unique mannequin to keep away from prices at inference time. LoRA retains coaching tractable even on a modest GPU setup, whereas delivering almost the identical efficiency acquire as full fine-tuning.
Since a shallow linear classifier normally performs effectively, fine-tuning may end up in modest beneficial properties by way of uncooked F1 rating. Nonetheless, under-labeling generally is a actual downside whenever you freeze your basis mannequin as we see under.
Easy has a price ticket
The key downside with the frozen mannequin is low classification confidence. At a normal 0.5 working threshold, a whopping 35% of photographs obtain no labels from the mannequin. Tuning down the edge helps, nevertheless it comes at the price of decrease precision. Per-class thresholds would possibly assist, however downstream purposes want scores that imply the identical factor throughout all 23 lessons, and class-specific thresholds would drift with each retraining.
In observe, we settled on a compromise of 0.2, which offers affordable protection and precision. Determine 3 illustrates what this appears like for a single picture: at t = 0.5 nothing clears the bar, whereas at t = 0.2 the 2 right labels come by.
A secondary downside is poor classification on just a few frequent lessons like GARDEN and HALLWAY. Edge circumstances additionally trigger issues: when a eating set is seen in a front room picture, we want to label it each LIVING ROOM and DINING AREA. Nonetheless, when the eating set is seen solely by a doorway, we don’t need the DINING AREA label.
These issues may be addressed by LoRA fine-tuning.
Placing it to the take a look at
We determined to coach our personal classifier and in contrast the 2 customized approaches outlined above: a frozen basis mannequin mixed with a shallow linear classifier, and fine-tuning with LoRA. In each circumstances, we added 23 unbiased classification heads on high of the inspiration mannequin, one per class.
The enter is a picture vector generated by SigLIP. We moreover experiment with DINOv2 as a frozen baseline to see how caption-training compares to self-supervised coaching. LoRA fine-tuning is finished solely on SigLIP. We used the unique SigLIP mannequin moderately than SigLIP 2 in these experiments; since we examine a frozen setup in opposition to fine-tuning on the identical spine, the conclusions don’t hinge on the mannequin technology.
For analysis, we use micro averaged F1 rating. This emphasizes efficiency on widespread lessons like KITCHEN and LIVING ROOM, that are most central for our use circumstances.
Moreover, we consider protection on the take a look at set: how most of the photographs get not less than one label? Whereas there’s a pure residual of inputs that don’t fall into any of the 23 lessons, we need to discover all of the photographs that may be labeled.
Coaching
We practice our classifiers on our personal proprietary set of 40k manually annotated photographs, the place every enter will get 1-3 class labels. Our validation knowledge has 1.9k examples; we break up this into 100 growth and 1.8k take a look at examples. Coaching, growth and take a look at photographs come from distinct listings, so photographs of the identical property by no means seem in multiple break up.
For each our frozen baselines, we educated 23 separate sklearn LogisticRegression fashions.
We educated LoRA utilizing the PEFT library. Following widespread observe [10], we wrapped the SigLIP ViT self-attention question and worth layers in LoRA adapters, leaving the MLP layers untouched, and used BCE loss on high of 23 unbiased logistic classification heads. This meant coaching solely about 0.6% of the mannequin’s parameters, roughly a 99% discount in comparison with full fine-tuning. Additionally it is why the entire sweep suits on a single T4.
We did a random 40-trial hyperparameter sweep [11] over the configurations in Desk 2, fixing all different hyperparameters to plain values.
For quick and numerically safer coaching, we used combined precision with fp16 autocast and loss scaling [12]. We educated for 20 epochs and picked the mannequin that delivers one of the best F1 rating on the event set.
All coaching is finished on an AWS EC2 g4dn.xlarge occasion with a single NVIDIA T4 having 16 GB VRAM.
Analysis
By way of plain micro averaged F1, variations are modest. At 82.6% F1, the fine-tuned mannequin beats each frozen SigLIP’s 78.4% F1 and frozen DINOv2’s 78.3% F1, however the distinction is simply round 4 factors. The frozen SigLIP and DINOv2 classifiers ship basically an identical efficiency. Frozen fashions are reported at their finest dev-set thresholds (0.2 for SigLIP, 0.35 for DINOv2); the fine-tuned mannequin at its default threshold of 0.5, which marginally understates its finest achievable F1 (83.1%). Desk 3 exhibits the total outcomes.
The rise in F1 rating is principally as a consequence of recall, which improves by roughly 7 factors from 72.8% (SigLIP) and 72.4% (DINOv2) to 79.3%. At 85.1%, the frozen fashions’ precision is already very excessive, and it solely improves by about 1 level.
As Determine 4 exhibits, these outcomes will not be an artifact of the working threshold; the fine-tuned classifier outperforms the frozen SigLIP classifier at each working threshold, exhibiting that fine-tuning doesn’t merely push confidence up however genuinely improves classification efficiency. With solely 100 growth examples, we deal with the chosen thresholds and stopping epoch as coarse decisions moderately than extremely tuned optima.
The modest beneficial properties in micro averaged F1 conceal substantial enhancements for particular person lessons, particularly for GARDEN (help in take a look at set: 237) with a formidable 26-point rise in comparison with frozen SigLIP, and DINING AREA (help in take a look at set: 150) with a good 15-point enchancment.
The GARDEN class is a very fascinating instance, as a result of it’s sometimes all background, one thing that SigLIP doesn’t do effectively off the shelf, as mentioned above. In such circumstances, fine-tuning can ship dramatic enhancements.
Nonetheless, after we have a look at efficiency for the GARDEN class utilizing the frozen DINOv2 mannequin, a unique sample emerges: frozen DINOv2 F1 rating is 58%, a 15-point enchancment over the frozen SigLIP mannequin. Simply by selecting a extra appropriate basis mannequin, we’ve got gained greater than half of the efficiency hole in comparison with a fine-tuned SigLIP mannequin. DINING AREA additionally exhibits an enchancment of 5 factors F1 rating.
On the similar time, DINOv2 underperforms in comparison with SigLIP on many lessons the place semantic understanding of the picture appears extra essential: it by no means predicts MARKETING (help in take a look at set: 24), and KITCHEN (help in take a look at set: 272) slips 7 factors. Curiously, efficiency additionally degrades on HALLWAY (help in take a look at set: 86), a basically architectural class the place we might have anticipated DINOv2 to excel.
The one class the place fine-tuning degrades efficiency is KITCHEN: an F1 drop of 5 factors. We think about this minor, however that is naturally case dependent.
The true promoting level for LoRA fine-tuning is that it largely solves under-labeling. The frozen SigLIP classifier leaves 9.4% of photographs with out labels and DINOv2 does even worse at 10.8%. SigLIP’s charge is 2.4x the pure charge (3.9%) of photographs that genuinely fall into none of our 23 lessons. LoRA finally ends up at 3.4%, barely under the pure charge, that means it as an alternative very sometimes over-labels.
Determine 5 exhibits that the under-labeling charge of the fine-tuned classifier stays low for affordable working thresholds. We will commerce a little bit of recall for even larger precision. In distinction, the under-labeling charge of the frozen classifier shoots towards the sky if one tries to sharpen precision by elevating the working threshold. As a classifier, it’s due to this fact far much less versatile than the fine-tuned one.
So, when ought to you fine-tune?
There are some things value contemplating. If under-labeling is an issue for you, then fine-tuning may be value it. The issue basically disappeared in our case.
For particular person lessons, we did see massive beneficial properties, particularly in recall. GARDEN and DINING AREA at the moment are acknowledged way more usually. Nonetheless, simply selecting an applicable basis mannequin (DINOv2 moderately than SigLIP) recovered greater than half of the GARDEN hole with out fine-tuning. Nonetheless, we didn’t observe degradation of precision, so beneficial properties are real albeit modest by way of uncooked F1.
With a coaching set of 40k examples, the price for a full hyperparameter sweep turned out to be round $30, which is negligible. Nonetheless, if under-labeling is just not a problem, you would possibly choose to make use of a frozen spine, particularly when periodic retraining is required. Coaching 23 classification heads on a CPU takes minutes and wishes no GPU; a full LoRA sweep takes days on a devoted GPU occasion, and that value repeats each time you retrain.
The bottom-effort possibility can be VLM-based classification, however at excessive volumes that turns into a big recurring value. The distinction between $1,500 and $0.37 for one million inputs provides up shortly. Nonetheless, keep in mind that coaching your personal classifier requires annotated knowledge. We use 40k manually annotated photographs. That isn’t free both.
For us, the funding has already paid off. The fine-tuned classifier now runs in manufacturing, and the labeling high quality is nice sufficient that Alma has constructed new performance on high of it. As a result of the labels are produced routinely, they’re accessible at scale for downstream purposes to construct on.
Conclusion
Whichever method you select, picture labeling pays off throughout the true property itemizing service: search outcomes, suggestions, and a spread of inside use circumstances all enhance. When you’re uncertain whether or not it’s value it, begin small and immediate a VLM to categorise a subset of your knowledge. From there, a light-weight classification head on high of an present embedding mannequin will lower your prices, and if you happen to want extra accuracy, fine-tuning your personal mannequin is the pure closing step.
References
[1] N. Kisel, I. Volkov, Okay. Janouskova and J. Matas, Multimodal massive language fashions as picture classifiers (2026), arXiv:2603.06578
[2] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He and B. Hooi, Can LLMs categorical their uncertainty? An empirical analysis of confidence elicitation in LLMs (2024), Worldwide Convention on Studying Representations (ICLR)
[3] S. Lin, J. Hilton and O. Evans, Educating fashions to precise their uncertainty in phrases (2022), Transactions on Machine Studying Analysis
[4] X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, Sigmoid loss for language picture pre-training (2023), IEEE/CVF Worldwide Convention on Laptop Imaginative and prescient (ICCV)
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski and A. Joulin, Rising properties in self-supervised imaginative and prescient transformers (2021), IEEE/CVF Worldwide Convention on Laptop Imaginative and prescient (ICCV)
[6] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al., DINOv2: Studying strong visible options with out supervision (2024), Transactions on Machine Studying Analysis
[7] M. El Banani, A. Raj, Okay.-Okay. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Solar, L. Guibas, J. Johnson and V. Jampani, Probing the 3D consciousness of visible basis fashions (2024), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[8] M. Ranzinger, G. Heinrich, J. Kautz and P. Molchanov, AM-RADIO: Agglomerative imaginative and prescient basis mannequin cut back all domains into one (2024), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[9] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun and N. Ballas, Self-supervised studying from photos with a joint-embedding predictive structure (2023), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, LoRA: Low-rank adaptation of enormous language fashions (2022), Worldwide Convention on Studying Representations (ICLR)
[11] J. Bergstra and Y. Bengio, Random seek for hyper-parameter optimization (2012), Journal of Machine Studying Analysis, 13
[12] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, Combined precision coaching (2018), Worldwide Convention on Studying Representations (ICLR)
