VectorVault

A from-scratch vector search engine: HNSW and IVF-Flat indexing for approximate nearest neighbor search, crash-safe persistence with write-ahead logging, and optional distributed queries across sharded servers.

This is a learning project demonstrating how a vector database works under the hood — the graph traversal, storage, and distribution logic are all implemented here, not wrapped from external libraries.

Architecture overview

┌─────────────────────────────────────────────────────────────────┐
│                           Clients                               │
│        Python SDK • REST API • curl • External apps             │
└────────────────────┬───────────────────────────┬────────────────┘
                     │                           │
    ┌────────────────▼──────┐           ┌────────▼────────-┐
    │  Single-Node Mode     │           │ Distributed Mode │
    │  (Embedded/Server)    │           │  (Cluster)       │
    └────────────┬──────────┘           └────────┬─────────┘
                 │                               │
         ┌───────▼────────┐           ┌──────────▼────────────┐
         │ Shard Server   │           │    Coordinator        │
         │   (C++ gRPC)   │           │  • Consistent hash    │
         └───────┬────────┘           │  • Scatter-gather     │
                 │                    │  • Result merge       │
         ┌───────▼────────┐           └──────-───┬────────────┘
         │  HNSW Index    │           ┌──────────┬─────-──────┐
         │  (Graph+SIMD)  │     ┌─────────┐ ┌─────────┐ ┌─────────┐
         └───────┬────────┘     │ Shard 0 │ │ Shard 1 │ │ Shard 2 │
                 │              └────┬────┘ └────┬────┘ └────┬────┘
         ┌───────▼────────┐          │           │           │
         │   Storage      │     ┌────▼────┐ ┌────▼────┐ ┌────▼────┐
         │ WAL • mmap     │     │  PVC 0  │ │  PVC 1  │ │  PVC 2  │
         │ SQLite • snap  │     │  data   │ │  data   │ │  data   │
         └────────────────┘     └─────────┘ └─────────┘ └─────────┘

Key features:

• HNSW graph index: L2, cosine, and dot product metrics with SIMD-optimized distance kernels
• IVF-Flat index: Inverted-file clustering for tunable recall vs speed tradeoff (HNSW is default)
• Durability: Write-ahead log + periodic snapshots; survives crashes
• Distributed mode: Consistent-hash routing + parallel scatter-gather search across shards
• Multiple interfaces: Python SDK (gRPC or HTTP), REST API, embedded library
• Kubernetes ready: StatefulSet deployment with persistent storage
• Prometheus metrics: REST gateway exposes /metrics (scrapable counters and histograms) and a live /dashboard UI

What it does (and doesn't)

Capability	Status
Approximate nearest neighbor search (ANN)	✅ HNSW with configurable M and ef parameters
Multiple distance metrics	✅ L2, cosine, dot product
Crash recovery	✅ WAL replay + graph snapshots
Distributed sharding	✅ Static shard count with consistent hashing
Python SDK	✅ gRPC and HTTP clients
REST API	✅ FastAPI gateway
Kubernetes deployment	✅ StatefulSet + coordinator + gateway
Prometheus monitoring	✅ /metrics endpoint + live dashboard on gateway
Benchmarks vs hnswlib	✅ SIFT10K/100K/1M with recall@k and QPS
Alternative indexes (IVF, PQ, disk-native)	✅ IVF-Flat (tunable recall/speed); HNSW default
Filtered ANN (metadata predicates in graph)	❌ Post-filter only
Replication / HA	❌ Single copy per shard
Dynamic rebalancing	❌ Static shard assignment

Full architecture details: docs/architecture.md

Quick start

pip install -e ".[dev]"

# Build C++ core, pybind11 module, shard binary, and hnswlib bench
make build

# Fast test suite (skips crash-recovery timing test)
make test

# Full SIFT benchmarks (SIFT10K + SIFT100K)
make bench

# SIFT1M (long build; often 20+ minutes)
make bench-1m

Manual build:

cmake -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j$(nproc)
python3 scripts/generate_proto.py
pytest tests/ -v

Usage examples

Embedded (Python)

from vectorvault import PersistentCollection
import numpy as np

# Create a collection with durable storage
coll = PersistentCollection("embeddings", dim=128, metric="cosine", path="./data")

# Insert vectors
vectors = np.random.randn(1000, 128).astype(np.float32)
for i, vec in enumerate(vectors):
    coll.insert(i, vec, metadata=f'{{"doc": "page_{i}"}}')

# Search
query = np.random.randn(128).astype(np.float32)
results = coll.search(query, k=10)
print(results)  # [(id, distance), ...]

HTTP REST API

# Start a shard (gRPC server binary)
./build/vectorvault_shard 0.0.0.0:50051

# Start REST gateway (shard or coordinator) with Prometheus metrics + live dashboard
./scripts/run_gateway.sh
# Dashboard:  http://127.0.0.1:8080/dashboard
# Prometheus: http://127.0.0.1:8080/metrics

# Create collection and search via curl
curl -X POST localhost:8080/collections \
  -H 'Content-Type: application/json' \
  -d '{"name":"demo","dim":4,"metric":"l2"}'

curl -X POST localhost:8080/collections/demo/vectors \
  -H 'Content-Type: application/json' \
  -d '{"vectors":[{"id":1,"values":[1.0, 0.0, 0.0, 0.0]}]}'

curl -X POST localhost:8080/collections/demo/search \
  -H 'Content-Type: application/json' \
  -d '{"query":[1.0, 0.0, 0.0, 0.0],"k":1}'

Distributed cluster (local)

make build
make cluster  # 3 shards + coordinator + gateway on localhost

Python SDK:

from vectorvault import VectorVault

with VectorVault.connect("localhost:7000") as vv:
    coll = vv.create_collection("demo", dim=4, metric="l2")
    coll.insert(1, [1.0, 0.0, 0.0, 0.0])
    print(coll.search([1.0, 0.0, 0.0, 0.0], k=1))

Kubernetes deployment

3-shard StatefulSet + coordinator + REST gateway on kind or minikube:

bash scripts/kind-up.sh      # create local kind cluster
make docker-build            # build shard + coordinator images
make k8s-deploy              # apply manifests, wait for pods
curl -s http://localhost:30080/health

Full guide: docs/kubernetes.md

Benchmark results

Single-node HNSW core (embedded)

These benchmarks measure the C++ HNSW index directly via pybind11 — no gRPC, no sharding, no coordinator. They showcase the core search engine performance.

Benchmarks run on a 4-vCPU provisioned instance. Raw data: bench/results/m6_all.json (local file, not in git). Reproduce:

bash bench/download_datasets.sh
make bench          # SIFT10K + SIFT100K
make bench-1m       # SIFT1M (long build; often 20+ minutes)

SIFT dataset results:

Dataset	Implementation	Build (s)	Search QPS	recall@10	p50 (ms)	p99 (ms)
SIFT10K	C++ HNSW	4.6	10,194	0.998	0.094	0.135
SIFT10K	hnswlib	0.5	12,875	0.987	0.070	0.105
SIFT100K	C++ HNSW	79.2	3,830	0.983	0.259	0.381
SIFT100K	hnswlib	12.7	5,292	0.990	0.189	0.275
SIFT1M	C++ HNSW	2,753	2,355	0.968	0.429	0.573
SIFT1M	hnswlib	344	501	0.967	0.512	72.73

At ef_search=64, VectorVault C++ outperforms hnswlib search QPS by 4.7× (2,355 vs 501 QPS) on SIFT1M while matching recall@10 at 0.968. Build time is ~8× slower (2,753s vs 344s) — the tradeoff favors VectorVault for read-heavy workloads. See docs/recall-and-build-notes.md for build-time analysis.

10K direct comparison:

Head-to-head on SIFT10K using identical test harness (bench/cpp_bench.py):

make cpp-hnswlib-bench

Implementation	Build (s)	Search QPS	recall@10
VectorVault C++	7.3	11,841	0.998
hnswlib C++	12.2	13,498	0.997

See bench/results/cpp_native_comparison_10k.json (local file, not in git).

Visualization (SIFT1M):

Full analysis: docs/recall-and-build-notes.md

---

Distributed mode (coordinator + shards)

Separate benchmarks measure coordinator overhead and shard scaling. Running make bench includes a 3-shard coordinator scaling test on SIFT10K. Results and plots in bench/results/shard_scaling.png.

Key finding: Scatter-gather search across N shards adds coordinator merge overhead and reduces per-shard QPS. See architecture.md for distributed query flow.

Project structure

Directory	Purpose
core/	C++ HNSW implementation (distance, graph, WAL, storage)
bindings/	pybind11 Python bindings
server/	C++ shard binary (gRPC ShardService)
cluster/	Python coordinator (routing, scatter-gather, merge)
gateway/	FastAPI REST gateway
vectorvault/	Python SDK and local PersistentCollection
proto/	gRPC service definitions
bench/	SIFT benchmark harness + plot generation
deploy/	Dockerfiles and Kubernetes manifests
tests/	pytest unit and integration tests
docs/	Architecture, performance, Kubernetes guides

Documentation

• architecture.md — System design, data model, request flows
• kubernetes.md — Kubernetes deployment guide (kind, minikube, operations)
• prometheus-metrics.md — Prometheus /metrics, dashboard, and Grafana setup
• recall-and-build-notes.md — HNSW recall analysis and build-time tradeoffs
• performance-bottlenecks.md — Performance journey: bottlenecks discovered and how they were fixed

Why this project exists

This is a learning project demonstrating vector database internals:

• Understand HNSW, WAL, distributed queries, gRPC, and Kubernetes StatefulSets by implementing them
• Show end-to-end system design skills — C++ core, Python services, deployment, benchmarks
• Discuss design tradeoffs, performance optimization, distributed systems patterns

Not for production use. This codebase prioritizes clarity and educational value over feature completeness.

License

MIT License — see LICENSE