Vector Databases in System Design Interviews: When to Use Them and What to Say
Vector databases have become a hot topic in system design interviews because they enable fast similarity search on high-dimensional data. In a sentence, a vector database is a storage engine that indexes embeddings – numeric representations of items – and returns the most similar vectors for a given query.
The Problem Vector Databases Solve
Traditional relational or key-value stores excel at exact matches. They struggle when the query is "find items like this" – for example, "show images that look like this photo" or "recommend articles similar to the one a user just read". The underlying issue is that similarity is a continuous metric (often cosine or Euclidean distance) rather than a discrete key.
Concrete System Example
Imagine an e-commerce site that wants to power a "visually similar products" feature. Users upload a photo of a shoe, and the system must return shoes with a similar style. The pipeline looks like this:
- Upload – user sends an image.
- Embedding Service – a neural model (e.g., CLIP) converts the image to a 512-dimensional vector.
- Vector Store – the vector is stored alongside the product ID.
- Query – another image is embedded and the store returns the top-k nearest vectors.
- Result Assembly – product details are fetched from the relational catalog and displayed.
A regular database would need to scan every row to compute distances, which is infeasible at scale. A vector database indexes the embeddings so that step 4 can be answered in milliseconds.
How a Vector Database Works – Step by Step
1. Ingestion
- The embedding service produces a dense vector
v(e.g.,[0.12, -0.03, …]). - The vector, together with a primary key (product ID), is written to the vector store.
2. Index Construction
Most production vector databases use an approximate nearest neighbor (ANN) index because exact search is O(N). A common scheme is IVF-PQ (Inverted File with Product Quantization):
| Phase | What Happens |
|---|---|
| Coarse quantization | Vectors are clustered (e.g., k-means with 10,000 centroids). Each vector is assigned to the nearest centroid, forming an inverted list. |
| Product quantization | Within each list, vectors are compressed into short codes (e.g., 8-byte) that approximate the original values. |
| Metadata store | The original primary key is kept alongside the compressed code for retrieval. |
3. Query Execution
- Embed the query – the same model produces a query vector
q. - Locate centroids – compute distances from
qto all centroids (cheap because centroids are few). Select the nearest few (e.g., 10). - Scan lists – retrieve the compressed codes from those lists and compute approximate distances to
q. - Rerank – optionally fetch the full vectors for the top-k candidates and compute exact distances.
- Return IDs – the IDs are sent back to the caller.
Because the search touches only a small fraction of the total data, latency stays in the low-single-digit milliseconds range even for billions of vectors.
Classic Interview Questions Where Vectors Appear
| Question | Where the Vector DB Appears |
|---|---|
| Semantic Image Search – "Design a service that finds visually similar images" | In the search component that receives an embedding and queries the vector store for nearest neighbors. |
| Personalized Recommendation – "Build a news feed that suggests articles like the one the user just read" | In the ranking layer that stores article embeddings and retrieves the nearest ones for a given article vector. |
| Code Search – "Design a system that returns code snippets similar to a natural-language query" | In the retrieval stage where the query is embedded and the vector DB returns the most similar code embeddings. |
In each case the vector database replaces a traditional index (e.g., inverted index) and provides a similarity-driven retrieval path.
Trade-offs and the Expected Alternative
Advantages
- Low latency at scale – ANN indexes let you search billions of vectors in <10 ms.
- Unified API – stores handle insert, delete, and update of vectors alongside metadata.
- Built-in metrics – cosine, Euclidean, inner product are supported out of the box.
Drawbacks
- Approximation error – you may miss the true nearest neighbor; recall depends on index parameters.
- Higher write cost – building or updating the ANN index is more expensive than a simple key-value write.
- Limited transactional guarantees – many vector stores favor eventual consistency for speed.
The Alternative Interviewer May Probe
Interviewers often ask how the design would look with a traditional approach. The usual alternative is an inverted index + feature hashing or a brute-force scan on a relational store. Expect follow-up questions like:
- "What if the data volume grows tenfold?" – discuss re-indexing cost.
- "Can we guarantee exact results?" – explain switching to exact K-NN (O(N)) and the latency impact.
- "How do we handle updates to embeddings?" – talk about background re-indexing or delete-and-re-insert strategies.
30-Second Pitch
"A vector database stores embedding vectors and lets you find the most similar items quickly using approximate nearest-neighbor indexes. It’s the go-to solution for semantic search, recommendation, and code-search features where you need "like this" results rather than exact key matches. The trade-off is that you accept a small approximation error in exchange for sub-10-ms latency at billions of items."
Common Mistakes Candidates Make
- Treating vectors like normal rows – claiming O(1) lookups without an ANN index shows a lack of understanding.
- Ignoring dimensionality – forgetting that higher dimensions increase index build time and memory usage.
- Over-promising accuracy – stating that the system always returns the true nearest neighbor without discussing recall.
- Skipping the embedding step – not describing how the vectors are generated and why the model choice matters.
- Neglecting write path – forgetting that updates require re-indexing, which can become a bottleneck.
Being clear about these points demonstrates practical knowledge rather than buzzword recitation.
How to Practice This
- Study a worked design – the System Design AI library includes a full walkthrough of a semantic image-search service that uses a vector database.
- Sketch the flow – on the canvas, draw the end-to-end pipeline: upload → embed → store → query → retrieve.
- Run a mock interview – use the practice brief and rubric to explain the design aloud, focusing on the trade-offs and alternatives.
By iterating through the brief and refining your explanation, you’ll be ready to discuss vector databases confidently in any interview.
Questions people ask
When is a vector database overkill?
If you only need exact key lookups or the data set is tiny (a few thousand items), a simple key-value or relational store is cheaper and easier to manage.
Can I combine a vector DB with a traditional relational DB?
Yes. Store embeddings in the vector DB for similarity search, and keep the authoritative record (price, description, etc.) in a relational database, joining after the nearest-neighbor step.
What similarity metrics are typically available?
Cosine similarity, Euclidean distance, and inner product. Cosine is common for text and image embeddings; choose the one that matches your model’s training.
How should I handle frequent updates to vectors?
Most vector stores treat an update as a delete followed by an insert. For high-frequency changes, batch updates and rebuild the ANN index during low-traffic periods.