Moss vs Milvus vs Pinecone vs Qdrant: Vector DB Benchmark

Which vector database is actually faster when used inside a real AI application?

That was the question behind this benchmark. In AI pipelines, the model is not always the only bottleneck. Query speed also depends on how fast embeddings are generated, searched, and retrieved from the vector database.

To test this, we benchmarked Moss, Milvus, Pinecone, and Qdrant under the same setup using a consistent dataset, embedding model, and query workflow. The goal was to measure real end-to-end latency instead of relying only on documentation or vendor claims.

What are Vector Databases?

A vector database is a database designed to store and search embeddings, which are numerical representations of data like text, images, audio, or documents.

Unlike traditional databases that search using exact words or filters, vector databases find results based on similarity. This makes them useful for AI applications where the system needs to understand the meaning of a query and return the most relevant results.

What is the Importance of Latency?

Latency is the time it takes for a system to process a query and return results. In AI applications, this matters because users expect quick responses, especially in chatbots, recommendation systems, semantic search, and real-time assistants.

For this benchmark, latency means the complete time taken from query input to final result. It includes embedding generation, vector search, and result retrieval. Lower latency means the AI application feels faster and more responsive to the user.

Benchmark Setup: How We Tested Each Vector Database

To keep the comparison fair, the same benchmark setup was used across Moss, Milvus, Pinecone, and Qdrant. Each system was tested with the same dataset size, embedding model, query type, number of runs, and latency metrics.

This setup measures the full query flow, including embedding generation, vector search, and result retrieval. The goal is to compare real response time under consistent conditions instead of looking only at database-level search speed.

How We Ran the Vector Database Benchmark?

Each system followed the same workflow to keep the comparison consistent and unbiased.

The process included:

• Loading the dataset of 100 sentences
• Converting the data into embeddings
• Inserting the embeddings into the vector database
• Creating an index, where applicable
• Executing search queries
• Measuring latency across multiple runs

This ensured that Moss, Milvus, Pinecone, and Qdrant were tested under the same conditions, making the results easier to compare.

Benchmarking Code Used for the Test

The complete benchmarking code is available on GitHub: Vector-DB-Benchmark

Or, if you want it in one line:

Benchmarking code: tejaswini-creator/Vector-DB-Benchmark

This looks clean and keeps the GitHub repo name as the anchor.

Vector Database Benchmark Results: Moss vs Milvus vs Pinecone vs Qdrant

The benchmark results show a clear latency difference across the four vector databases. Moss recorded the lowest latency in this setup, while Pinecone had the highest average and maximum latency.

In this test, Moss delivered the fastest median response time with a P50 latency of 4.24 ms and a P90 latency of 5.53 ms. Milvus and Qdrant showed moderate latency, while Pinecone had the highest latency in this API-based setup.

Benchmark Output Logs for Each Vector Database

Below are the output logs captured during the benchmark run for each vector database. These logs show the measured query latency for Moss, Milvus, Pinecone, and Qdrant under the same testing conditions.

Moss Output:

Milvus Output:

Pinecone Output:

Qdrant Output:

Latency Visualization: P50 vs P99

A graphical representation was created to compare P50 and P99 latency across systems.

Benchmark Architecture and Test Configuration

Systems

• Moss (managed semantic search)
• Milvus (managed / Zilliz Cloud)
• Pinecone (managed serverless)
• Qdrant (managed cloud)

Dataset

• https://huggingface.co/datasets/sh0416/ag_news
• 100 documents for the final API run
• 10 fixed queries (multi‑query benchmark)

Metrics

• P50, P90 (median and tail latency)
• Average, Min, Max

End-to-End Vector Search Pipeline Used in the Benchmark

All systems follow the same two‑phase flow:

1. Index Build (Ingestion)

• Load documents
• Generate embedding
• Insert vectors into the index
• Build index structures
• Load/ready for search

2. Query (Online Search)

• Embed query text
• Run vector search (top‑k)
• Return the nearest results

This architecture mimics a real product pipeline: documents are pre‑indexed, and users query continuously.

Vector Database Benchmark Architecture Diagram

Performance Analysis: Which Vector Database Was Faster?

1. Moss

Moss achieved the lowest latency among all systems.P50: 4.24 msP90: 5.53 msThis performance is likely due to:

• optimized managed retrieval pipeline
• fast in‑memory query execution on the service side
• minimal client‑side overhead after indexing

2. Milvus

Milvus showed moderate latency with some tail variation.P50: 209.60 msP90: 246.02 msThis suggests:

• stable median performance
• tail latency influenced by network or cloud‑side load
• Performance is sensitive to managed service conditions

3. Pinecone

Pinecone showed the highest median latency in this setup.P50: 306.08 msP90: 395.70 msThe higher latency is mainly due to:

• network communication
• managed API overheadHowever, it provides:
• elastic scalability
• strong managed reliability

4. Qdrant

Qdrant latency was higher than Moss but lower than Pinecone in median.P50: 233.51 msP90: 296.24 ms

This is primarily because:

• it was accessed through a cloud API
• batching and service limits affect tail latencyPerformance may improve with a higher‑tier cluster.

Qdrant vs Moss Accuracy Comparison

Overview of the Evaluation

This study evaluates the retrieval performance of two vector search systems, Qdrant and Moss, using a controlled benchmark. The objective is to measure how effectively each system retrieves semantically relevant documents under identical conditions.

The evaluation focuses on ranking quality and relevance using standard information retrieval metrics.

DatasetThe evaluation uses the Hugging Face AG News Dataset.

• Source: https://huggingface.co/datasets/sh0416/ag_news
• Type: News classification dataset
• Total classes: 4
  • 0 = World
  • 1 = Sports
  • 2 = Business
  • 3 = Sci/Tech

Data preparation

• Each document is constructed as:“title. description”
• The first 10,000 training samples are used for indexing.
• Each document retains its label as the ground truth for evaluation.

3. Query Construction and IngestionQueries are derived from the dataset itself but are intentionally paraphrased to avoid exact text matching and enforce semantic retrieval.

Query generation strategy

• “Headline removed. {short description}”
• “Summarize the situation: {short description}”
• “Identify the topic based on this: {short description}”
• “Which story matches this clue? {short description}”

Process:

1. Randomly sample 200 documents from the dataset.
2. Extract short descriptions.
3. Generate paraphrased queries programmatically.
4. Associate each query with its original label as ground truth.

These queries are then embedded and submitted to both systems for retrieval.

4. Evaluation Procedure
5. Load 10,000 labeled documents.
6. Build vector indexes in both Qdrant and Moss.
7. Generate 200 paraphrased queries.
8. Retrieve the top 5 results for each query from both systems.
9. Compare retrieved document labels with query labels.
10. Compute evaluation metrics and average across all queries.

Metrics Used:

Precision@5Definition: Measures the proportion of relevant documents in the top-5 results.Formula:Precision@5 = (Number of relevant documents in top-5) / 5

Accuracy@1Definition: Checks whether the top-ranked result is correct.Formula:Accuracy@1 = 1 if the top-1 label matches the query label, else 0

Recall@5Definition: Measures whether at least one relevant document appears in the top-5.Formula:Recall@5 = 1 if any correct label appears in the top-5, else 0

MRR@5 (Mean Reciprocal Rank)Definition: Measures how early the first correct result appears in the ranking.Formula:If the first correct result is at rank r, score = 1/rIf no correct result in the top 5, score = 0MRR@5 = average of scores across all queries

Qdrant vs Moss Accuracy Results

Qdrant vs Moss Retrieval Quality Comparison

Both systems demonstrate nearly identical performance across all evaluation metrics.

• Accuracy@1 is perfect for both systems, indicating that the top result is always correct.
• Recall@5 is also perfect, meaning at least one correct result consistently appears in the top-5.
• Precision@5 values are almost identical, showing similar quality of retrieved result sets.
• MRR@5 values indicate that correct results are ranked very early in both systems.

The minor differences observed are due to variations in embedding generation:

• Qdrant uses SentenceTransformer (all-MiniLM-L6-v2) locally.
• Moss uses its internal moss-minilm embedding model.

These small embedding differences slightly affect ranking order but not overall retrieval correctness.

What Do the Accuracy Results Mean?

The evaluation demonstrates that Qdrant and Moss achieve equivalent retrieval performance on this benchmark. Both systems effectively handle paraphrased, semantically challenging queries and consistently return correct results.

The differences between the systems are minimal and primarily related to ranking order rather than accuracy. Therefore, system selection should be based on factors such as latency, scalability, and infrastructure requirements rather than retrieval quality alone.

Which Vector Database Should You Choose?

The benchmark results show that there is no single best vector database for every use case. The right choice depends on whether you care more about latency, scalability, control, or open-source flexibility.

If your application needs the fastest response time, Moss is the stronger choice based on this benchmark. If you want a managed system that can scale with less infrastructure effort, Pinecone is a good fit. 

Milvus works well when you need more control over performance tuning, while Qdrant is useful when open-source flexibility and deployment choice matter more.

Limitations of the Benchmark

This benchmark was designed to compare latency under the same test conditions, but the results should be read with a few limitations in mind:

• All systems were accessed via API endpoints, so network latency is included
• Results reflect one region and one client location
• Dataset is real (sh0416/ag_news) but only 100 documents
• Query set was multi‑query (10 queries), not a single query

The queries used were:

QUERIES = [
    "oil prices surge as stock market faces pressure from slowing economy",
    "central bank signals interest rate pause amid inflation concerns",
    "tech company reports strong earnings with cloud revenue growth",
    "global markets react to geopolitical tensions in eastern europe",
    "new ai model boosts productivity for software developers",
    "automaker announces electric vehicle production expansion",
    "retail sales rise as consumers shift to online shopping",
    "energy sector gains on natural gas supply disruption",
    "health officials track outbreak with new vaccine rollout",
    "airline shares fall after fuel cost guidance update",
]

These factors can affect latency, so results may change with a larger dataset, a different region, a self-hosted setup, or production-scale traffic.

Conclusion

This benchmark shows that latency can vary widely across vector databases, especially in API-based setups. In this test, Moss delivered the lowest end-to-end latency, while Milvus and Qdrant showed moderate performance, and Pinecone recorded higher response times.

The accuracy comparison also showed that Moss and Qdrant performed almost equally well, so the final choice should not depend on accuracy alone. Choose based on your latency needs, deployment model, scalability requirements, and infrastructure preferences.

Frequently Asked Questions (FAQ)

What is a vector database? 

A vector database stores numerical representations of data (called embeddings) and retrieves results based on semantic similarity rather than exact keyword matching. This makes it well-suited for AI applications like semantic search, chatbots, and recommendation engines.

What does "latency" mean in this benchmark? 

Latency here is the full end-to-end time, from submitting a query to receiving the top-k results. It includes embedding generation, vector search, and result retrieval, not just the database search step alone.

What are P50 and P90? 

P50 (median) is the latency that 50% of queries fall under; it reflects typical performance. P90 means 90% of queries are completed within that time, making it a reliable indicator of tail latency and worst-case behavior under normal load.

Why is Pinecone slower in this benchmark? 

Pinecone is a fully managed serverless service, so every query travels over the network to a remote API. The latency reflects that round-trip overhead rather than raw search speed. In exchange, it offers elastic scalability and zero infrastructure management.

Which vector database was fastest in this benchmark?

Moss delivered the lowest latency in this API-based benchmark, with a P50 of 4.24 ms and P90 of 5.53 ms.