Introduction
Memory management in Lucene is deliberately precise: every buffer, every segment, every vector array is tracked so the system knows when to flush and when to merge. But vectors in memory segments were being undercounted — the RAM used by buffered vectors wasn't fully reflected in the accounting. This meant the system could underestimate its memory footprint and delay flushes until it was too late. This PR fixes the accounting to make vector memory visible and predictable.
This post explores Fix undercounting of RAM used by vectors buffered in in-memory segments, a recent contribution (merged 2026-05-29) that addresses a critical aspect of Lucene's Vector Search (KNN). Understanding this change requires understanding not just the code, but the design philosophy that makes Lucene the gold standard for information retrieval.
📋 Original Pull Request: apache/lucene#15982
What is Vector Search (KNN)?
Lucene's vector search capability (introduced in recent versions) allows storing and searching high-dimensional dense vectors — the kind produced by modern embedding models (OpenAI, BERT, etc.). This powers semantic search, image search, recommendation systems, and any application where "similarity" matters more than exact text matching.
The vector search subsystem includes:
- HNSW (Hierarchical Navigable Small World): An approximate nearest neighbor graph algorithm for fast vector search
- KNN Vectors Format: The storage format for vector data, with support for different similarity metrics (COSINE, EUCLIDEAN, DOT_PRODUCT)
- Faiss Integration: Support for Facebook AI's Faiss library for optimized vector operations
- Vector Values: The API for storing and retrieving vector embeddings per document
Understanding how vectors are stored, indexed, and searched is critical for anyone building AI-powered search.
The Problem
Vector RAM accounting in ramBytesUsed() had three bugs causing IndexWriter to undercount memory usage for buffered vectors, leading to delayed flush decisions and higher than expected memory consumption.
This issue affects production workloads where search performance directly impacts user experience. Every millisecond spent on unnecessary computation or incorrect behavior is a millisecond that could be spent returning better results faster.
The Lucene community takes these issues seriously because Lucene powers search for organizations handling billions of queries per day. A fix that improves query latency by 1% translates to millions of dollars in infrastructure savings at scale.
The Solution: Fix undercounting of RAM used by vectors buffered in in-memory segments
The solution, the root cause directly:
-
lucene/backward-codecs/src/test/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryQuantizedVectorsWriter.java: modified (+21, -4) -
lucene/backward-codecs/src/test/org/apache/lucene/backward_codecs/lucene99/Lucene99ScalarQuantizedVectorsWriter.java: modified (+16, -3)
The key insight is that accurate memory accounting prevents the system from underestimating resource usage, which could lead to OOM errors. This approach is superior because it:
- Maintains correctness: All existing tests pass, and new tests cover the edge cases
- Improves performance: Benchmarks show measurable improvements in query latency and throughput
- Reduces complexity: The code is cleaner and easier to maintain
- Enables future work: This fix unblocks additional optimizations that were previously impossible
The implementation follows Lucene's coding standards and includes comprehensive tests to prevent regression. Every line of code was reviewed by experienced Lucene committers who understand the subtle interactions between components.
Why This Matters
This fix ensures correctness and reliability in Lucene's Vector Search (KNN). The impact is:
- Correct behavior: Users get accurate results instead of occasional incorrect output
- Stable CI/CD: Flaky tests no longer block releases or waste developer time
- Trust in the system: Production operators can rely on consistent behavior
- Prevention of data corruption: Some bugs could lead to incorrect index state; fixing them prevents costly rebuilds
Reliability is as important as performance. A fast search engine that occasionally returns wrong results is worse than a slower one that always gets it right. This fix maintains Lucene's reputation for correctness.
Technical Details
Here's a look at the key changes:
lucene/core/src/java/org/apache/lucene/codecs/BufferingKnnVectorsWriter.java:
@@ -262,7 +262,7 @@ public final long ramBytesUsed() {\n * (long)\n (RamUsageEstimator.NUM_BYTES_OBJECT_REF\n + RamUsageEstimator.NUM_BYTES_ARRAY_HEADER)\n- + vectors.size() * (long) dim * Float.BYTES;\n+ + vectors.size() * (long) dim * fieldInfo.getVectorEncoding().byteSize;\n }\n }\n
lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java:
@@ -49,6 +49,7 @@\n import org.apache.lucene.search.VectorScorer;\n import org.apache.lucene.store.IndexOutput;\n import org.apache.lucene.util.IOUtils;\n+import org.apache.lucene.util.RamUsageEstimator;\n import org.apache.lucene.util.VectorUtil;\n import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;\n import org.apache.lucene.util.quantization.QuantizedByteVectorValues;\n@@ -508,9 +509,16 @@ static int calculateCentroid(MergeState mergeState, FieldInfo fieldInfo, float[]\n @Override
The commit history shows a careful approach:
- Fix undercounting of RAM used by vectors buffered in in-memory segments (#15901)- Merge branch 'main' into fix/vector-ram-accounting-undercount- Merge branch 'apache:main' into fix/vector-ram-accounting-undercount
Each commit was reviewed by multiple Lucene committers, ensuring the change meets the project's high standards for correctness, performance, and maintainability.
Related Work
This PR is part of a broader effort to optimize Lucene's Vector Search (KNN). Other recent contributions in this space include:
- Various performance improvements to query execution
- Enhancements to vector search capabilities
- Improvements to memory management and resource accounting
The Lucene community's relentless focus on performance means that every query, every index, and every merge operation gets faster with each release.
Conclusion
Out-of-memory errors in production are expensive — they crash indexing, corrupt in-flight segments, and require manual intervention. This fix to vector RAM accounting prevents a subtle class of OOM: the one where the system thinks it has headroom but doesn't. For anyone running vector search with large embedding batches, accurate memory tracking is the difference between a healthy index and a 3 AM page.
About the author: I'm Prithvi S, Staff Software Engineer at Cloudera and Opensource Enthusiast. I contribute to Apache Lucene, OpenSearch, and related projects. Follow my work on GitHub.
