, vector search has change into a important piece of AI infrastructure, powering use instances from RAG and semantic search to agentic reminiscence and context layers. With the rise of agentic techniques, firms are attempting to offer as a lot context to the brokers as potential, which requires vector db indexes to develop from an preliminary million or dozens of tens of millions scale to the tons of of tens of millions and even billions. At this scale, storing indexes and related information in RAM will price hundreds of {dollars} monthly, and HNSW can change into a scalability bottleneck.
On this article I wish to dive into the main points of what really makes semantic search quick and environment friendly: approximate nearest neighbor (ANN) algorithms, what completely different choices exist, and their trade-offs.
Deep dive into vector DB
Vector databases include three major elements:
embeddings – the numeric illustration of the corpus
search algorithm and index construction – the algorithm defines the search high quality and pace
storage – how the info is saved (in reminiscence, on disk, payload along with embeddings, and so forth.). Whether or not in RAM or on disk, it determines the prices and latency at scale
Embeddings are already properly outlined and mentioned in lots of articles, and this one will deal with search algorithms, and particularly ANN ones. There are typically two approaches for the search execution:
Precise search – which demonstrates one of the best retrieval metrics though it doesn’t scale properly when it comes to latency
Approximate nearest neighbor (ANN) – which trades off the retrieval high quality for the latency and scalability.
The precise search is a straightforward strategy which loops via the entire entries within the index and calculates the space between your search question and current information. There aren’t any losses associated to any approximation or generalization with the trade-off of the latency and scalability. It’s an amazing strategy for both very small indexes or experimentation, however typically not very appropriate for the manufacturing scale.
Fascinating truth: many trendy vector databases will let you bypass index constructing for small collections, falling again to kNN search as a result of the overhead of constructing an index shouldn’t be value it for just a few thousand vectors.
The second choice is approximate nearest neighbor algorithms, which is a gaggle of algorithms with the primary goal of enhancing scalability by avoiding visiting all of the entries within the index. The thought right here is to offer some shortcuts to hurry up the search and ingestion. The implementations differ, though many trendy ANN algorithms (for instance, HNSW and DiskANN) depend on a graph construction to offer low question latency.
Approximate nearest neighbor algorithms
It’s necessary to debate that though ANN algorithms are all following an analogous idea to realize the aim of offering a brief path to the top end result, there are completely different implementations with completely different algorithms having distinctive units of trade-offs, which makes it essential to pick the one that matches your precise use case.
On this article I wish to deal with two completely different teams of the ANN algorithms:
RAM-based – such algorithms are optimized for storing all or an enormous chunk of information in reminiscence, which offers extraordinarily low latency with the prices as a trade-off. The usual instance right here is HNSW (Hierarchical Navigable Small World)
On-disk – these algorithms are minimizing RAM utilization and relying closely on disk to load the required information. An instance right here is DiskANN or SPANN
It’s required to say that it’s potential to retailer underlying information buildings of each of those algorithm teams both on disk or in reminiscence (at the very least partially), however they’re optimized for the precise storage sort and due to this fact will present one of the best outcomes using what they had been designed for.
In-memory ANN
HNSW (Hierarchical Navigable Small World)
The most well-liked ANN algorithm utilized by virtually any trendy vector database. The thought is to make the most of a layered graph-based information construction to attach vectors with close to neighbors, which offers extraordinarily quick retrieval when storing the entire index in reminiscence. It’s an amazing match for small-medium use instances and can present one of the best retrieval pace.
Nonetheless, as soon as the index is sufficiently big that it now not suits in RAM or storing it in RAM turns into very costly, the choice is to both transfer information to disk, which can trigger a drastic efficiency hit, or extremely quantize it, which might trigger a big drop in retrieval high quality. The primary purpose for such a efficiency hit is that the HNSW construction shouldn’t be optimized for the clustered disk entry, and consequently, the search would produce a variety of non-sequential I/O operations. Contemplating a number of hops per search and comparatively excessive learn site visitors, the disk I/O will change into a bottleneck, which might considerably improve latency, from milliseconds to tons of of milliseconds or worse underneath heavy I/O strain.
The in-memory algorithms are extremely optimized to retailer all vectors and connections in RAM, which makes them extraordinarily quick, however with a trade-off of being reminiscence hungry. Furthermore, with on-disk choices such algorithms depend on random disk entry, which might change into a possible bottleneck for large-scale indexes (100 million+).
Instance vector databases: Qdrant, Milvus, pgvector, OpenSearch, Weaviate, Redis
On-Disk ANN
This group of algorithms is designed particularly to interrupt the RAM consumption limitation of the in-memory ANN algorithms and scale back the storage prices whereas offering acceptable latency. It’s an amazing selection if search latency shouldn’t be important and the index dimension is predicted to be giant. We will think about two major algorithms on this group:
SPANN
It’s a disk-based ANN algorithm that follows the inverted-index (IVF) methodology: vectors are grouped into clusters, every represented by a centroid. It was particularly designed to deal with extraordinarily giant billion-vector+ indexes that received’t slot in RAM or might be too costly to be saved in reminiscence. The thought is to arrange factors in clusters, which is a pure property of the embedding area, choose a centroid illustration of the cluster, and put it to use for the routing layer. Centroids and mainly the entire routing layer might be saved in RAM whereas the vectors represented by the centroids are saved on disk. The necessary element is that vectors represented by the identical centroid are saved on disk sequentially and due to this fact might be loaded from it quick and effectively. Throughout search, centroids are used to search out the closest teams of vectors, after which the vectors related to these centroids are loaded from disk to carry out a full scan.
Notice: In comparison with HNSW with the on-disk storage choice, SPANN ensures that as a substitute of random disk entry, the vectors represented by the one centroid are grouped on disk and due to this fact loaded as blocks, which dramatically reduces the variety of required disk I/O operations whereas offering acceptable latency.
Instance vector databases: Turbopuffer (constructed on SPFresh, a SPANN successor), Chroma DB (cloud)
DiskANN
As an alternative of a centroid-based strategy, DiskANN maintains a single-layer graph referred to as Vamana. The primary thought behind it’s to reduce the variety of hops required to search out the highest okay factors and due to this fact the variety of random disk entry operations. It’s achieved by retaining some longer-range connections as a substitute of solely the closest ones, so fewer hops are wanted to achieve the goal. The unique vectors are saved on disk whereas the extremely quantized model of the vectors is saved in RAM, which additionally contributes to decreasing the required variety of disk entry operations. In comparison with SPANN, DiskANN’s Vamana graph is constructed over each level, so the graph itself scales with the dataset, which is an actual reminiscence consideration at billion scale, the place SPANN solely wants its centroids resident. The info on disk shouldn’t be clustered, and the disk I/O is minimized by the routing layer doing a minimal variety of hops to get to related vectors, resulting in extremely environment friendly search in apply. There are a variety of inner particulars on how precisely it’s applied, and I extremely suggest exploring the origin paper, which is linked within the references for this text.
Instance vector databases: Milvus, PostgreSQL (by way of pg_diskann)
Economics
Notice: the costs beneath are approximate and present as of writing. Cloud pricing shifts over time and varies by supplier, area, and dedication, so deal with these figures as illustrative of the RAM-vs-disk ratio moderately than precise quotes
With RAM costing about 5$ per GB via cloud suppliers, EBS is about 50 occasions cheaper, round 0.08-0.10$ per GB, and native NVMe SSD round 0.20-0.25$ per GB. Subsequently, for the 100,000,000 index in 1024 dimensions with float32 precision, it is going to be 1024 * 4 bytes = 4 KB per vector, and with production-grade replication of three it can require 12 KB of storage per vector.
Subsequently, for 100,000,000 vectors, the overall required quantity of storage is 1.2 TB. In fact, there’s a quantization choice, which can scale back this quantity, and the most well-liked and least invasive scalar quantization would require 25% of the storage, which is 300 GB.
Subsequently, the approximate month-to-month storage related prices:
Non-quantized in RAM ~6000 USD
Scalar-quantized in RAM ~ 1500 USD
Distant Disk ~120 USD
Native Disk ~ 300 USD
And since it is a linear relationship, the hole solely widens because the index grows towards the dimensions agentic techniques are pushing towards:
Consequently, though scalar quantization reduces the invoice considerably, it’s nonetheless a excessive price in comparison with the on-disk choice.
The trade-off
As with the whole lot in engineering, the associated fee discount offered by on-disk ANN algorithms shouldn’t be free. Whereas the routing layer does guarantee environment friendly information retrieval and narrows down the exploration to the smaller subset, the info nonetheless must be loaded from the disk, which is considerably slower than loading it from RAM. It’s value mentioning that for lots of use instances it is probably not a deal breaker. Contemplating instances reminiscent of RAG, the place outcomes from the vector db are then handed to the reranker and LLM, the 100ms delay on the retrieval shouldn’t be going to be the primary bottleneck, however for instances reminiscent of agentic reminiscence, context, and so forth., it really could also be most popular to have the ability to execute search as quick as potential, particularly if there are a number of calls throughout a single agent request processing.
It’s genuinely exhausting to offer a transparent quantity for on-disk latency, and that’s kind of the purpose, it extremely depends upon the precise setup. For instance, the SPANN paper studies reaching 90% recall in round 1ms at billion scale, however that’s a imply latency on a single machine with the index saved on native SSD. As soon as you progress to an actual deployment, the image adjustments. Turbopuffer’s benchmark on a 10M-vector index reveals round 14ms at p50 when the index is heat on quick storage, however near 874ms when it’s chilly and must be fetched from object storage, which is round a 60x distinction on the identical information simply from the cache state. Elements like {hardware} and whether or not the info is heat (already in cache) can every transfer the quantity by 10x or extra. Basic steerage is that disk-based ANN algorithms present slower latency than HNSW (in RAM) simply because RAM entry is far quicker.
Select properly
With each on-disk and in-memory algorithms, it’s necessary to make the proper selection about which one might be a greater match to your use case. Whereas HNSW offers a simple, well-rounded resolution for small and medium dimension indexes, it might be value exploring the on-disk choices as soon as your index grows greater and the related prices of storing vectors in RAM change into a burden. Furthermore, there are at all times edge instances like comparatively high-dimensional vectors for which RAM might be a bottleneck comparatively early or large low-dimensionality indexes which might make the most of RAM for for much longer. For engineers, it’s necessary to concentrate on such use instances and make a complete choice on the trade-offs acceptable for his or her use instances.
