System Design AI

Vector Databases in System Design Interviews: When to Use Them and What to Say

Vector databases have become a hot topic in system design interviews because they enable fast similarity search on high-dimensional data. In a sentence, a vector database is a storage engine that indexes embeddings – numeric representations of items – and returns the most similar vectors for a given query.

The Problem Vector Databases Solve

Traditional relational or key-value stores excel at exact matches. They struggle when the query is "find items like this" – for example, "show images that look like this photo" or "recommend articles similar to the one a user just read". The underlying issue is that similarity is a continuous metric (often cosine or Euclidean distance) rather than a discrete key.

Concrete System Example

Imagine an e-commerce site that wants to power a "visually similar products" feature. Users upload a photo of a shoe, and the system must return shoes with a similar style. The pipeline looks like this:

  1. Upload – user sends an image.
  2. Embedding Service – a neural model (e.g., CLIP) converts the image to a 512-dimensional vector.
  3. Vector Store – the vector is stored alongside the product ID.
  4. Query – another image is embedded and the store returns the top-k nearest vectors.
  5. Result Assembly – product details are fetched from the relational catalog and displayed.

A regular database would need to scan every row to compute distances, which is infeasible at scale. A vector database indexes the embeddings so that step 4 can be answered in milliseconds.

How a Vector Database Works – Step by Step

1. Ingestion

  • The embedding service produces a dense vector v (e.g., [0.12, -0.03, …]).
  • The vector, together with a primary key (product ID), is written to the vector store.

2. Index Construction

Most production vector databases use an approximate nearest neighbor (ANN) index because exact search is O(N). A common scheme is IVF-PQ (Inverted File with Product Quantization):

PhaseWhat Happens
Coarse quantizationVectors are clustered (e.g., k-means with 10,000 centroids). Each vector is assigned to the nearest centroid, forming an inverted list.
Product quantizationWithin each list, vectors are compressed into short codes (e.g., 8-byte) that approximate the original values.
Metadata storeThe original primary key is kept alongside the compressed code for retrieval.

3. Query Execution

  1. Embed the query – the same model produces a query vector q.
  2. Locate centroids – compute distances from q to all centroids (cheap because centroids are few). Select the nearest few (e.g., 10).
  3. Scan lists – retrieve the compressed codes from those lists and compute approximate distances to q.
  4. Rerank – optionally fetch the full vectors for the top-k candidates and compute exact distances.
  5. Return IDs – the IDs are sent back to the caller.

Because the search touches only a small fraction of the total data, latency stays in the low-single-digit milliseconds range even for billions of vectors.

Classic Interview Questions Where Vectors Appear

QuestionWhere the Vector DB Appears
Semantic Image Search – "Design a service that finds visually similar images"In the search component that receives an embedding and queries the vector store for nearest neighbors.
Personalized Recommendation – "Build a news feed that suggests articles like the one the user just read"In the ranking layer that stores article embeddings and retrieves the nearest ones for a given article vector.
Code Search – "Design a system that returns code snippets similar to a natural-language query"In the retrieval stage where the query is embedded and the vector DB returns the most similar code embeddings.

In each case the vector database replaces a traditional index (e.g., inverted index) and provides a similarity-driven retrieval path.

Trade-offs and the Expected Alternative

Advantages

  • Low latency at scale – ANN indexes let you search billions of vectors in <10 ms.
  • Unified API – stores handle insert, delete, and update of vectors alongside metadata.
  • Built-in metrics – cosine, Euclidean, inner product are supported out of the box.

Drawbacks

  • Approximation error – you may miss the true nearest neighbor; recall depends on index parameters.
  • Higher write cost – building or updating the ANN index is more expensive than a simple key-value write.
  • Limited transactional guarantees – many vector stores favor eventual consistency for speed.

The Alternative Interviewer May Probe

Interviewers often ask how the design would look with a traditional approach. The usual alternative is an inverted index + feature hashing or a brute-force scan on a relational store. Expect follow-up questions like:

  • "What if the data volume grows tenfold?" – discuss re-indexing cost.
  • "Can we guarantee exact results?" – explain switching to exact K-NN (O(N)) and the latency impact.
  • "How do we handle updates to embeddings?" – talk about background re-indexing or delete-and-re-insert strategies.

30-Second Pitch

"A vector database stores embedding vectors and lets you find the most similar items quickly using approximate nearest-neighbor indexes. It’s the go-to solution for semantic search, recommendation, and code-search features where you need "like this" results rather than exact key matches. The trade-off is that you accept a small approximation error in exchange for sub-10-ms latency at billions of items."

Common Mistakes Candidates Make

  1. Treating vectors like normal rows – claiming O(1) lookups without an ANN index shows a lack of understanding.
  2. Ignoring dimensionality – forgetting that higher dimensions increase index build time and memory usage.
  3. Over-promising accuracy – stating that the system always returns the true nearest neighbor without discussing recall.
  4. Skipping the embedding step – not describing how the vectors are generated and why the model choice matters.
  5. Neglecting write path – forgetting that updates require re-indexing, which can become a bottleneck.

Being clear about these points demonstrates practical knowledge rather than buzzword recitation.

How to Practice This

  1. Study a worked design – the System Design AI library includes a full walkthrough of a semantic image-search service that uses a vector database.
  2. Sketch the flow – on the canvas, draw the end-to-end pipeline: upload → embed → store → query → retrieve.
  3. Run a mock interview – use the practice brief and rubric to explain the design aloud, focusing on the trade-offs and alternatives.

By iterating through the brief and refining your explanation, you’ll be ready to discuss vector databases confidently in any interview.

Questions people ask

When is a vector database overkill?

If you only need exact key lookups or the data set is tiny (a few thousand items), a simple key-value or relational store is cheaper and easier to manage.

Can I combine a vector DB with a traditional relational DB?

Yes. Store embeddings in the vector DB for similarity search, and keep the authoritative record (price, description, etc.) in a relational database, joining after the nearest-neighbor step.

What similarity metrics are typically available?

Cosine similarity, Euclidean distance, and inner product. Cosine is common for text and image embeddings; choose the one that matches your model’s training.

How should I handle frequent updates to vectors?

Most vector stores treat an update as a delete followed by an insert. For high-frequency changes, batch updates and rebuild the ANN index during low-traffic periods.