Apache Lucene: Everything You Need to Know About the World's Most Used Search Engine

Document 1: "The cat sat on the mat"
Document 2: "The dog sat on the log"  
Document 3: "The cat chased the dog"

"cat"    → [Doc 1, Doc 3]
"dog"    → [Doc 2, Doc 3]
"sat"    → [Doc 1, Doc 2]
"mat"    → [Doc 1]
"log"    → [Doc 2]
"chased" → [Doc 3]
"the"    → [Doc 1, Doc 2, Doc 3]

"cat" → [
  {doc: 1, freq: 1, positions: [2]},    // "cat" appears once in Doc 1, at word position 2
  {doc: 3, freq: 1, positions: [2]}     // "cat" appears once in Doc 3, at word position 2
]

"sat" → [
  {doc: 1, freq: 1, positions: [3]},
  {doc: 2, freq: 1, positions: [3]}
]

"the" → [
  {doc: 1, freq: 2, positions: [1, 5]},  // "the" appears TWICE in Doc 1
  {doc: 2, freq: 2, positions: [1, 5]},
  {doc: 3, freq: 2, positions: [1, 5]}
]

Document {
  "title": "Lucene in Action",
  "author": "Michael McCandless",
  "year": 2024,
  "body": "Lucene is a search library..."
}

Field "title" = Text (analyzed, searchable, stored)
Field "author" = String (exact match, not analyzed)
Field "year" = IntPoint (numeric, for range queries)
Field "price" = NumericDocValues (for sorting/faceting)

Input:  "The quick brown foxes jump!"
Output: ["quick", "brown", "fox", "jump"]

Document doc = new Document();
doc.add(new TextField("title", "Lucene in Action", Field.Store.YES));
doc.add(new TextField("body", "Lucene is a search library", Field.Store.YES));
writer.addDocument(doc);

┌─────────────────────────────────────┐
│  IndexWriter.addDocument(doc)       │
│        ↓                            │
│  ┌─────────────────────────────┐    │
│  │  DWPT (Thread 1)            │    │
│  │  - RAM Buffer (16MB default)│    │
│  │  - Postings Hash Map        │    │
│  │  - Stored Fields Buffer     │    │
│  │  - DocValues Buffer         │    │
│  └─────────────────────────────┘    │
│        ↓                            │
│  [Other DWPTs for other threads]   │
└─────────────────────────────────────┘

Input: "Lucene is a search library"

StandardAnalyzer:
  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
  │ StandardTokenizer │→│ LowerCaseFilter │→│ StopFilter   │
  └──────────────┘   └──────────────┘   └──────────────┘
       ↓                   ↓                   ↓
  ["Lucene", "is",       ["lucene",           ["lucene",
   "a", "search",          "is", "a",           "search",
   "library"]              "search",            "library"]
                           "library"]

Before adding doc:
  "lucene"  → []  
  "search"  → []
  "library" → []

After adding doc (docID=0):
  "lucene"  → [{doc: 0, freq: 1, pos: 1}]
  "search"  → [{doc: 0, freq: 1, pos: 3}]
  "library" → [{doc: 0, freq: 1, pos: 4}]

Stored Fields Buffer (row-oriented):
┌───────┬─────────────────────────────┐
│ docID │ field data (compressed)     │
├───────┼─────────────────────────────┤
│ 0     │ title="Lucene in Action"    │
│       │ body="Lucene is a search..."│
└───────┴─────────────────────────────┘

DocValues Buffer (column-oriented):
┌──────────────┬──────────────┐
│ docID (all)  │ year (none)  │
├──────────────┼──────────────┤
│ 0            │ (no year field)│
└──────────────┴──────────────┘

BEFORE FLUSH:
┌─────────────────────────┐
│  DWPT In-Memory         │
│  ├─ Postings Hash       │
│  ├─ Stored Fields       │
│  ├─ DocValues           │
│  └─ Field Infos         │
└─────────────────────────┘
           ↓ serialize
AFTER FLUSH:
┌─────────────────────────┐
│  Segment Files on Disk  │
│  ├─ _0.fdt (stored data)│
│  ├─ _0.fdx (stored index)│
│  ├─ _0.tim (term dictionary)│
│  ├─ _0.tip (term index)│
│  ├─ _0.doc (postings)   │
│  ├─ _0.pos (positions)   │
│  ├─ _0.dvd (docvalues)   │
│  └─ _0.si (segment info) │
└─────────────────────────┘

_0.si   → Segment metadata (version, diagnostics, number of docs)
_0.fnm  → Field names and types (field "title" is TextField, etc.)
_0.fdx  → Stored fields index (quick lookup into .fdt)
_0.fdt  → Stored fields data (compressed document content)
_0.tim  → Term dictionary (FST - compressed term map)
_0.tip  → Term index (in-memory FST for fast term lookup)
_0.doc  → Postings lists (doc IDs and frequencies)
_0.pos  → Positions (where each term occurs in each doc)
_0.pay  → Payloads and offsets (optional per-occurrence data)
_0.dvd  → DocValues data (columnar numeric/sorted fields)
_0.dvm  → DocValues metadata
_0.nvd  → Norms (1 byte per doc per field for length normalization)
_0.nvm  → Norms metadata

Before commit:
  segments_1 → [Segment_0, Segment_1, Segment_2]

After commit (new docs added):
  segments_2 → [Segment_0, Segment_1, Segment_2, Segment_3, Segment_4]

  (segments_1 is kept as backup until segments_2 is fsync'd)

Before merge:
  [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB]  ← 11 segments

After merge (TieredMergePolicy):
  [20MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB] [2MB]  ← 1 big + 10 small

Eventually:
  [200MB] [50MB] [20MB] [5MB] [2MB]  ← logarithmic tiering

Query query = new TermQuery(new Term("body", "search"));
TopDocs results = searcher.search(query, 10);

User input: "lucene AND search OR library"
         ↓
QueryParser:
  ┌─────────────────────────────────┐
  │ BooleanQuery                    │
  │  ├── MUST: TermQuery("lucene")  │
  │  └── SHOULD: BooleanQuery       │
  │       ├── TermQuery("search")   │
  │       └── TermQuery("library")  │
  └─────────────────────────────────┘

Before rewrite:
  BooleanQuery
    ├── MUST: TermQuery("body", "search")
    └── MUST: MatchAllDocsQuery

After rewrite:
  TermQuery("body", "search")  ← MatchAllDocsQuery is redundant, removed

Weight weight = query.createWeight(searcher, ScoreMode.TOP_SCORES, 1.0f);

IndexReader has 3 segments: [Seg_0, Seg_1, Seg_2]

Weight.scorer(Seg_0) → Scorer_0
Weight.scorer(Seg_1) → Scorer_1
Weight.scorer(Seg_2) → Scorer_2

Collector (min-heap of top 10)
    ↑
    │  collect(doc, score)
    │
Scorer for Segment 0:
  postings = [doc=5, freq=3, pos=[...]]
  postings = [doc=12, freq=1, pos=[...]]
  postings = [doc=23, freq=2, pos=[...]]

  for each posting:
    score = BM25Score(freq, docLength, avgLength, idf)
    if score > minHeap.min():
      add to heap
      update minCompetitiveScore

For each block of 64 docs:
  blockMaxScore = precomputed maximum score for this block
  if blockMaxScore < minCompetitiveScore:
    SKIP ENTIRE BLOCK (64 docs)
    continue
  else:
    score each doc individually

IDF("search") = log(1 + (N - n(q) + 0.5) / (n(q) + 0.5))
              = log(1 + (1000 - 50 + 0.5) / (50 + 0.5))
              = log(1 + 950.5 / 50.5)
              = log(1 + 18.82)
              = log(19.82)
              = 2.99

IDF = log(1 + (1000 - 5 + 0.5) / (5 + 0.5))
    = log(1 + 995.5 / 5.5)
    = log(181.9)
    = 5.20

tfComponent = (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * |D|/avgDL))

For Document 5 (freq=3, |D|=100, avgDL=200):
  = (3 * (1.2 + 1)) / (3 + 1.2 * (1 - 0.75 + 0.75 * 100/200))
  = (3 * 2.2) / (3 + 1.2 * (0.25 + 0.375))
  = 6.6 / (3 + 1.2 * 0.625)
  = 6.6 / (3 + 0.75)
  = 6.6 / 3.75
  = 1.76

Score = IDF * tfComponent
      = 2.99 * 1.76
      = 5.26

tfComponent = (3 * 2.2) / (3 + 1.2 * (0.25 + 0.75 * 50/200))
            = 6.6 / (3 + 1.2 * (0.25 + 0.1875))
            = 6.6 / (3 + 1.2 * 0.4375)
            = 6.6 / (3 + 0.525)
            = 6.6 / 3.525
            = 1.87

Score = 2.99 * 1.87 = 5.59  ← Higher score! Shorter docs get boost.

Collector's min-heap (top 10 by score):
  1. Doc 5, Score: 5.26
  2. Doc 12, Score: 4.89
  3. Doc 23, Score: 4.71
  ...
  10. Doc 89, Score: 3.42

TopDocs {
  totalHits: 156 (how many docs matched total)
  scoreDocs: [ScoreDoc(5, 5.26), ScoreDoc(12, 4.89), ...]
}

Raw Text
   ↓
CharFilter (optional)  ← e.g., HTML strip, mapping chars
   ↓
Tokenizer              ← Split into tokens
   ↓
TokenFilter            ← Transform tokens (lowercase, stop, stem)
   ↓
TokenFilter            ← More transformations
   ↓
Terms → Index

Analyzer analyzer = new StandardAnalyzer();

// What it does:
// 1. StandardTokenizer: Unicode-aware word boundaries
//    "Hello, world! Check out https://example.com"
//    → ["Hello", "world", "Check", "out", "https://example.com"]
//
// 2. LowerCaseFilter: "Hello" → "hello"
//
// 3. StopFilter: Removes "the", "is", "a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with"

Analyzer ecommerceAnalyzer = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName) {
        // Tokenize on whitespace (preserves product codes like "ABC-123")
        Tokenizer tokenizer = new WhitespaceTokenizer();

        TokenStream stream = new LowerCaseFilter(tokenizer);

        // Synonyms: "laptop" = "notebook" = "portable computer"
        SynonymMap.Builder builder = new SynonymMap.Builder(true);
        builder.add(new CharsRef("laptop"), new CharsRef("notebook"), true);
        builder.add(new CharsRef("laptop"), new CharsRef("portable computer"), true);
        SynonymMap synonymMap = builder.build();
        stream = new SynonymGraphFilter(stream, synonymMap, true);

        // Stemming: "running" → "run", "shoes" → "shoe"
        stream = new PorterStemFilter(stream);

        // Edge n-grams for autocomplete: "lap" → "laptop"
        stream = new EdgeNGramTokenFilter(stream, 2, 10);

        return new TokenStreamComponents(tokenizer, stream);
    }
};

// English
Analyzer english = new EnglishAnalyzer();  // Standard + English stopwords + Porter stemmer

// French
Analyzer french = new FrenchAnalyzer();      // French stopwords + French stemming

// Chinese/Japanese/Korean
Analyzer cjk = new CJKAnalyzer();            // Bigram tokenization (no spaces in CJK)

// ICU (International Components for Unicode) - handles all languages
Analyzer icu = new ICUNormalizer2CharFilterFactory();  // NFKC normalization

// See EXACTLY what your analyzer produces
String text = "The quick brown foxes jump!";

try (TokenStream stream = analyzer.tokenStream("field", text)) {
    CharTermAttribute termAttr = stream.addAttribute(CharTermAttribute.class);
    PositionIncrementAttribute posAttr = stream.addAttribute(PositionIncrementAttribute.class);
    OffsetAttribute offsetAttr = stream.addAttribute(OffsetAttribute.class);
    TypeAttribute typeAttr = stream.addAttribute(TypeAttribute.class);

    stream.reset();
    while (stream.incrementToken()) {
        System.out.printf("term=%s, pos=%d, offset=%d-%d, type=%s%n",
            termAttr.toString(),
            posAttr.getPositionIncrement(),
            offsetAttr.startOffset(),
            offsetAttr.endOffset(),
            typeAttr.type());
    }
    stream.end();
}

// Output:
// term=quick, pos=2, offset=4-9, type=<ALPHANUM>
// term=brown, pos=1, offset=10-15, type=<ALPHANUM>
// term=fox, pos=1, offset=16-21, type=<ALPHANUM>
// term=jump, pos=1, offset=22-27, type=<ALPHANUM>

IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());

// RAM buffer size - controls how much memory before flush
config.setRAMBufferSizeMB(256.0);  // Default 16MB. More = faster indexing, more memory.

// Max buffered docs - alternative trigger (whichever comes first)
config.setMaxBufferedDocs(10000);  // Flush after 10,000 docs

// Merge policy - controls segment merging
config.setMergePolicy(new TieredMergePolicy());

// Merge scheduler - controls how merges run
config.setMergeScheduler(new ConcurrentMergeScheduler());  // Default, merges in background threads
// config.setMergeScheduler(new SerialMergeScheduler());   // For testing, merges in foreground

// Open mode
config.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
// CREATE = overwrite existing
// APPEND = add to existing
// CREATE_OR_APPEND = create if none, append if exists

// Index deletion policy - controls commit history
config.setIndexDeletionPolicy(new KeepOnlyLastCommitDeletionPolicy());
// KeepOnlyLastCommitDeletionPolicy = only keep last commit (default)
// SnapshotDeletionPolicy = allow snapshotting commits for backup

// Similarity - scoring model
config.setSimilarity(new BM25Similarity());
// config.setSimilarity(new ClassicSimilarity());  // TF-IDF (legacy)

// Codec - on-disk format
config.setCodec(new Lucene99Codec());
// Lucene99Codec = current default (Lucene 9.x)
// Lucene95Codec = older format
// You can write custom codecs!

// Info stream - debug logging
config.setInfoStream(System.out);  // See EVERYTHING IndexWriter does

// Flush on close - ensure all docs are flushed before closing
config.setCommitOnClose(true);

IndexWriter writer = new IndexWriter(FSDirectory.open(Paths.get("/index")), config);

┌─────────────────────────────────────────────────────────────┐
│                    TIMELINE                                  │
├─────────────────────────────────────────────────────────────┤
│  addDocument() → addDocument() → addDocument() → ...       │
│       │              │              │                        │
│       └──────┬───────┴──────┬──────┘                        │
│              │              │                                │
│              ▼              ▼                                │
│        [RAM Buffer fills]  [Or commit() called]              │
│              │              │                                │
│              ▼              ▼                                │
│         FLUSH ──────────── FLUSH                             │
│              │              │                                │
│              ▼              ▼                                │
│      New Segment created  New Segment created                │
│              │              │                                │
│              └──────┬──────┘                                │
│                     │                                       │
│                     ▼                                       │
│              [Background Thread]                             │
│                     │                                       │
│                     ▼                                       │
│                   MERGE                                     │
│                     │                                       │
│                     ▼                                       │
│           Small segments → Large segments                   │
└─────────────────────────────────────────────────────────────┘

// Option 1: Commit + reopen (durable, ~1 second latency)
writer.commit();
DirectoryReader newReader = DirectoryReader.openIfChanged(reader);

// Option 2: NRT - no commit needed! (~100ms latency)
DirectoryReader nrtReader = DirectoryReader.open(writer);
IndexSearcher nrtSearcher = new IndexSearcher(nrtReader);
TopDocs results = nrtSearcher.search(query, 10);

// Reopen to see newer docs (still no commit!)
DirectoryReader newNrtReader = DirectoryReader.openIfChanged(nrtReader);

// 1. DirectoryReader - reads from disk (FSDirectory)
DirectoryReader reader = DirectoryReader.open(FSDirectory.open(Paths.get("/index")));

// 2. NRT Reader - reads uncommitted segments from IndexWriter
DirectoryReader nrtReader = DirectoryReader.open(writer);

// 3. MultiReader - reads multiple indices as one
IndexReader multiReader = new MultiReader(reader1, reader2, reader3);

// 4. SlowCompositeReaderWrapper - flattens multi-segment to single (slower, for compatibility)
IndexReader slowReader = new SlowCompositeReaderWrapper(reader);

// 5. FilterDirectoryReader - wraps another reader with filtering

// Basic search
IndexSearcher searcher = new IndexSearcher(reader);
TopDocs results = searcher.search(query, 10);

// Search with sorting
Sort sort = new Sort(new SortField("price", SortField.Type.INT, false));  // ascending
TopFieldDocs sortedResults = searcher.search(query, 10, sort);

// Search with filter (cheaper than query for reusable filters)
Query filter = new TermQuery(new Term("status", "active"));
Query filteredQuery = new BooleanQuery.Builder()
    .add(query, BooleanClause.Occur.MUST)
    .add(filter, BooleanClause.Occur.FILTER)  // FILTER doesn't score
    .build();
TopDocs filteredResults = searcher.search(filteredQuery, 10);

// Search with collector manager (for concurrent search across segments)
CollectorManager<TopScoreDocCollector, TopDocs> manager = 
    TopScoreDocCollector.createSharedManager(10, null);
TopDocs concurrentResults = searcher.search(query, manager);

IndexReader  → Thread-safe for reading, NOT for reopening
IndexSearcher → Thread-safe for searching
  (Internally, one IndexSearcher per thread for concurrent segment search)

// Find documents where "field" contains exactly "value"
Query q = new TermQuery(new Term("title", "lucene"));

// Use case: Finding documents by ID, category, exact keyword

// "machine learning" must appear as consecutive words
Query q = new PhraseQuery("body", "machine", "learning");

// With slop (allows words in between)
PhraseQuery.Builder builder = new PhraseQuery.Builder();
builder.add(new Term("body", "machine"), 0);
builder.add(new Term("body", "learning"), 1);
builder.setSlop(2);  // Allow up to 2 words between
Query q = builder.build();

// "machine [anything] [anything] learning" matches

Query q = new BooleanQuery.Builder()
    .add(new TermQuery(new Term("title", "lucene")), BooleanClause.Occur.MUST)      // AND
    .add(new TermQuery(new Term("body", "search")), BooleanClause.Occur.SHOULD)     // OR (boosts score)
    .add(new TermQuery(new Term("status", "draft")), BooleanClause.Occur.MUST_NOT)   // NOT
    .add(new TermQuery(new Term("year", "2024")), BooleanClause.Occur.FILTER)       // AND (no score contribution)
    .build();

// MUST = required (AND)
// SHOULD = optional (OR), contributes to score
// MUST_NOT = excluded (NOT)
// FILTER = required, no scoring (faster than MUST for caching)

// Integer range (uses BKD tree, NOT inverted index)
Query q = IntPoint.newRangeQuery("price", 10, 100);

// Long range (timestamps)
Query q = LongPoint.newRangeQuery("timestamp", 
    Instant.parse("2024-01-01T00:00:00Z").toEpochMilli(),
    Instant.parse("2024-12-31T23:59:59Z").toEpochMilli());

// Double range
Query q = DoublePoint.newRangeQuery("rating", 4.0, 5.0);

// Prefix: "aut*" matches "auto", "automobile", "autumn"
Query q = new PrefixQuery(new Term("title", "aut"));

// Wildcard: "*cat*" matches "cat", "catch", "concatenate"
Query q = new WildcardQuery(new Term("title", "*cat*"));

// Regex: "[a-z]+cat[a-z]*" matches words containing "cat"
Query q = new RegexpQuery(new Term("title", "[a-z]+cat[a-z]*"));

// "lucne" matches "lucene" with edit distance 1
Query q = new FuzzyQuery(new Term("title", "lucne"), 1);

// "lucne" matches "lucene" with edit distance 2
Query q = new FuzzyQuery(new Term("title", "lucne"), 2);

// Fuzzy + prefix: "lucn*" must match prefix, then fuzzy within
Query q = new FuzzyQuery(new Term("title", "lucne"), 2, 3);  // prefix length 3

// Match everything (useful for testing, or with filters)
Query q = new MatchAllDocsQuery();

// Match nothing (useful for degenerate cases)
Query q = new MatchNoDocsQuery();

// Default parameters
BM25Similarity similarity = new BM25Similarity(1.2f, 0.75f);
// k1 = 1.2 (term frequency saturation)
// b = 0.75 (length normalization)

k1 = 0.0:  No term frequency saturation. All occurrences weighted equally.
k1 = 1.2:  Default. Diminishing returns after ~3-5 occurrences.
k1 = 10.0: Nearly linear TF weighting. Very sensitive to frequency.

Example: Document with 1 vs 10 occurrences of "lucene"
- k1=0.0:   Score ratio = 1:1 (no difference!)
- k1=1.2:   Score ratio ≈ 1:2 (10x frequency → 2x score)
- k1=10.0:  Score ratio ≈ 1:7 (10x frequency → 7x score)

b = 0.0:  No length normalization. Short and long docs equal.
b = 0.75: Default. Moderate length penalty.
b = 1.0:  Full length normalization. Long docs heavily penalized.

Example: Title match vs Body match
- "lucene" in title (5 words) vs "lucene" in body (500 words)
- b=0.0:   Title score = Body score (same TF)
- b=0.75:  Title score ≈ 2.5x Body score (shorter = better)
- b=1.0:   Title score ≈ 5x Body score (extreme preference for short)

// Combine BM25 with a custom function (e.g., boost by popularity)
Query baseQuery = new TermQuery(new Term("title", "lucene"));

DoubleValuesSource popularityBoost = DoubleValuesSource.fromIntField("popularity");

Query boostedQuery = new FunctionScoreQuery(baseQuery, popularityBoost) {
    @Override
    protected float score(float docScore, double funcScore) {
        return docScore * (float) Math.log1p(funcScore);  // BM25 * log(popularity)
    }
};

// Pre-Lucene 6 scoring
ClassicSimilarity similarity = new ClassicSimilarity();

score = tf * idf * norm
  where tf = sqrt(termFrequency)
  where idf = 1 + log(numDocs / (docFreq + 1))
  where norm = 1 / sqrt(docLength)

Terms: "cat", "cats", "dog", "dogs", "door", "dorm"

FST (simplified):
    c ──→ a ──→ t ──→ $  (cat)
                  │
                  └─→ s ──→ $  (cats)
    d ──→ o ──→ g ──→ $  (dog)
                  │
                  ├─→ s ──→ $  (dogs)
                  │
                  ├─→ o ──→ r ──→ $  (door)
                  │
                  └─→ r ──→ m ──→ $  (dorm)

$ = final state (valid term end)
Arc output = postings pointer + doc frequency

Points: [(1, 2), (3, 4), (5, 6), (7, 8), (2, 3), (4, 5), (6, 7), (8, 9)]

Step 1: Sort by dimension 0, find median
  Sorted: [(1,2), (2,3), (3,4), (4,5), (5,6), (6,7), (7,8), (8,9)]
  Median: (4,5) and (5,6) → split at 4.5

Step 2: Left half [(1,2), (2,3), (3,4), (4,5)]
  Sort by dim 1: [(1,2), (2,3), (3,4), (4,5)]
  Split at 2.5

Step 3: Recurse until ≤ 1024 points per leaf

On disk:
  .dii = index tree (inner nodes)
  .dim = leaf blocks (packed points, doc IDs, min/max bounds)

Range query: [x: 3-7, y: 4-8]
  Traverse tree:
    Root bounds: [1-8, 2-9] → intersects → go deeper
    Left child: [1-4, 2-5] → intersects → go deeper
    Right child: [5-8, 6-9] → intersects → go deeper
    ...
    Leaf [3-4, 4-5] → check each point individually
    Prune branches that don't intersect!

Documents: 1000
Field "price" values: [10, 25, 10, 50, 25, 10, 100, 50, ...]

Row-oriented (Stored Fields): 
  Doc 0: {title: "A", price: 10}  ← must read entire doc to get price
  Doc 1: {title: "B", price: 25}
  Doc 2: {title: "C", price: 10}
  ...

Column-oriented (DocValues):
  price.dvd: [10, 25, 10, 50, 25, 10, 100, 50, ...]
  ← sequential read, cache-friendly, no decompression of unrelated fields

// One byte per document per field
// Stores: 1 / (1 + b * (fieldLength / avgFieldLength - 1))

// Encoding: 256 discrete values (log-scaled)
// ~0.4% relative error

// For a document with fieldLength = 100, avgFieldLength = 200:
norm = 1 / (1 + 0.75 * (100/200 - 1))
     = 1 / (1 + 0.75 * (-0.5))
     = 1 / (1 - 0.375)
     = 1 / 0.625
     = 1.6

// Encoded to 1 byte: ~230 (out of 256)

// "quick" within 5 positions of "fox"
SpanQuery quick = new SpanTermQuery(new Term("body", "quick"));
SpanQuery fox = new SpanTermQuery(new Term("body", "fox"));
SpanQuery near = new SpanNearQuery(new SpanQuery[]{quick, fox}, 5, true);
// "quick brown fox" matches (distance 2)
// "quick ... fox" matches (distance up to 5)
// "fox quick" doesn't match (ordered=true requires quick before fox)

// "quick" NOT within 10 positions of "lazy"
SpanQuery quick = new SpanTermQuery(new Term("body", "quick"));
SpanQuery lazy = new SpanTermQuery(new Term("body", "lazy"));
SpanQuery notNear = new SpanNotQuery(quick, lazy, 10);

// Complex: "a" near "b" near "c" within 20 positions
SpanQuery a = new SpanTermQuery(new Term("body", "a"));
SpanQuery b = new SpanTermQuery(new Term("body", "b"));
SpanQuery c = new SpanTermQuery(new Term("body", "c"));
SpanQuery ab = new SpanNearQuery(new SpanQuery[]{a, b}, 10, false);
SpanQuery abc = new SpanNearQuery(new SpanQuery[]{ab, c}, 20, false);

// Index time: store custom payload with each term occurrence
TokenStream stream = analyzer.tokenStream("body", text);
PayloadAttribute payloadAttr = stream.addAttribute(PayloadAttribute.class);
// ... for each token, set payload = "POS=NOUN" or "WEIGHT=0.9" ...

// Search time: boost by payload
PayloadScoreQuery query = new PayloadScoreQuery(
    new SpanTermQuery(new Term("body", "lucene")),
    new MaxPayloadFunction()  // or AveragePayloadFunction, MinPayloadFunction
);

// Boost by a numeric field value
Query base = new TermQuery(new Term("title", "lucene"));
DoubleValuesSource recency = DoubleValuesSource.fromLongField("timestamp");
DoubleValuesSource popularity = DoubleValuesSource.fromIntField("views");

// Recency boost: newer = higher score
DoubleValuesSource recencyBoost = new ReciprocalDoubleValuesSource(recency);

// Combine: BM25 * recency * log(popularity)
DoubleValuesSource combined = new ProductDoubleValuesSource(
    new ProductDoubleValuesSource(recencyBoost, popularity)
);

Query functionQuery = new FunctionScoreQuery(base, combined);

Query base = new TermQuery(new Term("title", "lucene"));

CustomScoreQuery customQuery = new CustomScoreQuery(base) {
    @Override
    protected CustomScoreProvider getCustomScoreProvider(LeafReaderContext context) {
        return new CustomScoreProvider(context) {
            @Override
            public float customScore(int doc, float subQueryScore, float[] valSrcScores) {
                // Custom scoring logic:
                // subQueryScore = BM25 score from base query
                // valSrcScores = scores from value sources (if any)

                float popularity = getPopularity(doc);  // your custom field
                return subQueryScore * (1 + popularity / 100.0f);
            }
        };
    }
};

// Find documents similar to document 42
MoreLikeThis mlt = new MoreLikeThis(reader);
mlt.setFieldNames(new String[]{"title", "body"});
mlt.setMinTermFreq(1);
mlt.setMinDocFreq(1);
mlt.setAnalyzer(analyzer);

Query likeQuery = mlt.like(42);  // Generate query from document 42
TopDocs similarDocs = searcher.search(likeQuery, 10);

// Or from text directly:
Reader textReader = new StringReader("This is the reference text...");
Query likeQuery = mlt.like(textReader);

// Levenshtein distance 2, prefix length 3
FuzzyQuery fuzzy = new FuzzyQuery(new Term("title", "lucne"), 2, 3);

// Internally:
// 1. Build Levenshtein automaton for "lucne" with max edits=2
// 2. Intersect automaton with FST term dictionary
// 3. Collect matching terms: "lucene" (1 edit), "lance" (2 edits), "lucie" (2 edits)
// 4. Rewrite to BooleanQuery(TermQuery("lucene"), TermQuery("lance"), TermQuery("lucie"))
// 5. Execute with max_expansions limit (default 50)

// Fast regex: starts with literal prefix
RegexpQuery fast = new RegexpQuery(new Term("title", "lucene[0-9]+"));
// FST can find "lucene" prefix, then check regex on remaining characters

// Slow regex: no literal prefix
RegexpQuery slow = new RegexpQuery(new Term("title", "[a-z]+lucene"));
// Must scan entire term dictionary

# Heap size: Enough for FSTs, query caches, but leave room for OS page cache
-Xms16g -Xmx16g

# G1GC is generally best for Lucene workloads
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

# Disable biased locking (not helpful for Lucene's concurrency)
-XX:-UseBiasedLocking

# Large pages for heap (if supported by OS)
-XX:+UseLargePages

# Disable explicit GC calls (some libraries call System.gc())
-XX:+DisableExplicitGC

// MMapDirectory - memory-mapped files (default, fastest for most cases)
Directory dir = new MMapDirectory(Paths.get("/index"));
// Pros: OS page cache, no JVM heap, fast random access
// Cons: 64-bit only, may have issues with very large files on some OS

// NIOFSDirectory - NIO FileChannel
Directory dir = new NIOFSDirectory(Paths.get("/index"));
// Pros: Works on all platforms, predictable
// Cons: Slower than MMap, more system calls

// SimpleFSDirectory - plain RandomAccessFile
Directory dir = new SimpleFSDirectory(Paths.get("/index"));
// Pros: Simple, no dependencies
// Cons: Slow, not recommended for production

// RAMDirectory - in-memory (deprecated, use ByteBuffersDirectory instead)
Directory dir = new RAMDirectory();
// Use for: testing, small temporary indices

TieredMergePolicy mergePolicy = new TieredMergePolicy();
mergePolicy.setMaxMergeAtOnce(10);          // Max segments to merge at once
mergePolicy.setSegmentsPerTier(10.0);       // Target segments per size tier
mergePolicy.setMaxMergedSegmentMB(5000);    // Max segment size (5GB)
mergePolicy.setFloorSegmentMB(2);           // Minimum segment size for tiering
mergePolicy.setForceMergeDeletesPctAllowed(10); // Merge segments with >10% deleted docs

config.setMergePolicy(mergePolicy);

// Throttle merges to avoid impacting search
ConcurrentMergeScheduler scheduler = new ConcurrentMergeScheduler();
scheduler.setMaxMergesAndThreads(3, 2);  // 3 max merges, 2 threads
config.setMergeScheduler(scheduler);

// 1. Increase RAM buffer (more docs in memory before flush)
config.setRAMBufferSizeMB(256.0);  // Default 16MB. Try 256MB-512MB.

// 2. Use multiple indexing threads (DWPT handles this automatically)
// Each thread gets its own DWPT. Just index from multiple threads.

// 3. Disable unnecessary features
// If you don't need positions:
FieldType type = new FieldType(TextField.TYPE_STORED);
type.setIndexOptions(IndexOptions.DOCS_AND_FREQS);  // No positions
// Saves ~30-50% index size and indexing time

// 4. Bulk document addition
List<Document> docs = loadBatch();  // 100-1000 docs
writer.addDocuments(docs);  // Bulk add (slightly more efficient)

// 5. Disable norms if not needed
FieldType type = new FieldType(TextField.TYPE_STORED);
type.setOmitNorms(true);  // Saves 1 byte per doc per field

// 6. Use stored fields sparingly
// Only store fields you need to retrieve. Everything else = index only.

// 1. Warm the index
// On startup, run typical queries to warm OS page cache
for (Query warmQuery : typicalQueries) {
    searcher.search(warmQuery, 1);  // Don't care about results, just load data
}

// 2. Use filters for caching
Query filter = new TermQuery(new Term("status", "active"));
Query constantScore = new ConstantScoreQuery(filter);  // Cacheable

// 3. Limit wildcard/prefix queries
// Set max expansions
PrefixQuery query = new PrefixQuery(new Term("title", "a"));
query.setRewriteMethod(new MultiTermQuery.TopTermsRewrite(100));  // Max 100 terms

// 4. Use query cache
LRUQueryCache queryCache = new LRUQueryCache(1000, 100_000_000);  // 1000 queries, 100MB
IndexSearcher searcher = new IndexSearcher(reader);
searcher.setQueryCache(queryCache);

// 5. Collector-level optimization
// If you only need count, don't score
TotalHitCountCollector countCollector = new TotalHitCountCollector();
searcher.search(query, countCollector);
int totalHits = countCollector.getTotalHits();

// 6. Parallel search across segments
ExecutorService executor = Executors.newFixedThreadPool(4);
IndexSearcher parallelSearcher = new IndexSearcher(reader, executor);

// Query cache (caches query results per segment)
LRUQueryCache queryCache = new LRUQueryCache(
    1000,    // Max cached queries
    100_000_000,  // Max cache size in bytes (100MB)
    context -> true,  // Cache all queries (or filter by cost)
    1.0f     // Cache ratio (1.0 = cache everything eligible)
);

// Field cache (for DocValues, automatically managed)
// No configuration needed - DocValues are memory-mapped

// Filter cache (caches filter bitsets)
// Use CachingWrapperFilter or LRUQueryCache with filters

// Backup using SnapshotDeletionPolicy
SnapshotDeletionPolicy snapshotPolicy = new SnapshotDeletionPolicy(
    new KeepOnlyLastCommitDeletionPolicy()
);
config.setIndexDeletionPolicy(snapshotPolicy);

// Take a snapshot
IndexCommit commit = snapshotPolicy.snapshot();
Collection<String> fileNames = commit.getFileNames();
// Copy all fileNames to backup location

// Release snapshot when done
snapshotPolicy.release(commit);

// Restore: just copy files back and open
Directory restoredDir = FSDirectory.open(Paths.get("/restored"));
// Verify with CheckIndex

# CheckIndex tool - verify and optionally fix index
java org.apache.lucene.index.CheckIndex /path/to/index

# Output shows:
# - Segment integrity
# - File checksums
# - Doc count verification
# - Orphaned file detection

# Fix index (removes corrupt segments)
java org.apache.lucene.index.CheckIndex /path/to/index -fix

// Programmatic check
CheckIndex checkIndex = new CheckIndex(dir);
CheckIndex.Status status = checkIndex.checkIndex();
if (status.clean) {
    System.out.println("Index is clean!");
} else {
    System.err.println("Index has problems: " + status.segmentsChecked);
}

// Lucene 8 to 9: Use Lucene's IndexUpgrader
IndexUpgrader upgrader = new IndexUpgrader(dir, new Lucene99Codec(), true);
upgrader.upgrade();

// Or simply open with newer version and optimize (force merge)
IndexWriterConfig config = new IndexWriterConfig(analyzer);
config.setCodec(new Lucene99Codec());
IndexWriter writer = new IndexWriter(dir, config);
writer.forceMerge(1);  // Merge all segments to new format
writer.close();

// IndexWriter metrics
IndexWriter writer = new IndexWriter(dir, config);

// Number of segments
int numSegments = writer.getDirectory().listAll().length;

// Merge statistics
MergeScheduler mergeScheduler = config.getMergeScheduler();
// Use MergeScheduler.Info to get running merges

// Segment info
List<SegmentCommitInfo> segments = writer.getSegmentInfos().asList();
for (SegmentCommitInfo seg : segments) {
    System.out.println("Segment: " + seg.info.name);
    System.out.println("  Docs: " + seg.info.maxDoc());
    System.out.println("  Size: " + seg.sizeInBytes());
    System.out.println("  Del: " + seg.getDelCount());
}

// Searcher metrics
IndexSearcher searcher = new IndexSearcher(reader);
// Track query latency, cache hit rates, segment counts externally

┌─────────────────────────────────────────────┐
│  HOT TIER (SSD, Recent Data)                │
│  ├── Last 7 days of logs                    │
│  ├── Active products                        │
│  └── Frequent queries → cached in RAM       │
├─────────────────────────────────────────────┤
│  WARM TIER (SSD, Older Data)                │
│  ├── Last 30 days of logs                   │
│  ├── Seasonal products                      │
│  └── Less frequent queries                  │
├─────────────────────────────────────────────┤
│  COLD TIER (HDD/S3, Archive)                │
│  ├── Historical data                        │
│  └── Force-merged to 1 segment              │
│  └── On-demand loading                      │
└─────────────────────────────────────────────┘

Elasticsearch Cluster
  ├── Node 1
  │     ├── Shard 0 (Primary) → Lucene Index
  │     │     ├── Segment 1
  │     │     ├── Segment 2
  │     │     └── Segment 3
  │     └── Shard 1 (Replica) → Lucene Index
  ├── Node 2
  │     ├── Shard 0 (Replica) → Lucene Index
  │     └── Shard 1 (Primary) → Lucene Index
  └── Node 3
        ├── Shard 2 (Primary) → Lucene Index
        └── Shard 2 (Replica) → Lucene Index

ES adds:
  - Distributed architecture (cluster coordination)
  - REST API
  - Document-level operations (CRUD)
  - Mapping/schema management
  - Aggregations framework (on top of DocValues)
  - Replication and failover
  - Index lifecycle management
  - Machine learning integration
  - Ingest pipelines

Solr Core
  ├── Lucene Index (same as above)
  ├── Schema (managed, with field types)
  ├── Request Handlers (/select, /update, etc.)
  ├── Update Processors (custom indexing logic)
  ├── Search Components (faceting, highlighting, grouping)
  └── Replication (master-slave or SolrCloud)

Solr adds:
  - XML/JSON config-based schema
  - Rich search components (facet, stats, cluster, etc.)
  - SolrCloud (ZooKeeper-based distributed coordination)
  - Built-in faceting (more mature than early ES)

// Index vectors
float[] vector = embeddingModel.embed("query text");
doc.add(new KnnVectorField("embedding", vector, VectorSimilarityFunction.COSINE));

// Search vectors
Query knnQuery = new KnnVectorQuery("embedding", queryVector, 100);
TopDocs results = searcher.search(knnQuery, 10);

// Future: Concurrent indexing with virtual threads
// IndexWriter will use virtual threads for concurrent DWPT flushes
// IndexSearcher will use virtual threads for per-segment concurrent search
// No more thread pool management!

// Future: Java Vector API for BM25 scoring
// 2-5x speedup for scoring-heavy queries
// Multiple document scores computed in parallel using SIMD instructions

// Enable debug logging to see everything
config.setInfoStream(System.out);

// Or use a file for analysis
config.setInfoStream(new PrintStream("/tmp/lucene-indexing.log"));

# Clone and build
git clone https://github.com/apache/lucene.git
cd lucene
./gradlew assemble

# Run all tests (takes hours!)
./gradlew test

# Run specific module tests
./gradlew :lucene-core:test

# Run specific test class
./gradlew :lucene-core:test --tests "TestIndexWriter"

# Run specific test method
./gradlew :lucene-core:test --tests "TestIndexWriter.testCommit"

# Run with random seed (for reproducibility)
./gradlew :lucene-core:test --tests "TestIndexWriter" -Dtests.seed=DEADBEEF

lucene/
├── lucene/                          # Main code modules
│   ├── core/                        # Core indexing and search
│   ├── analysis/                    # Analyzers and tokenizers
│   ├── codecs/                      # Codec implementations
│   ├── demo/                        # Demo applications
│   ├── facet/                       # Faceting module
│   ├── group/                       # Result grouping
│   ├── highlighter/                 # Highlighting implementations
│   ├── join/                        # Parent/child joins
│   ├── memory/                      # Memory-based indices
│   ├── misc/                        # Miscellaneous utilities
│   ├── queries/                     # Additional query types
│   ├── queryparser/                 # Query parsers
│   ├── suggest/                     # Autocomplete/suggest
│   ├── benchmark/                   # Performance benchmarks
│   └── test-framework/              # Testing utilities
├── gradle/                          # Gradle build files
├── dev-docs/                        # Developer documentation
└── versions.lock                    # Dependency versions

   IndexWriter.addDocument() [index/IndexWriter.java]
     → DocumentsWriter.updateDocument() [index/DocumentsWriter.java]
       → DocumentsWriterPerThread.updateDocument() [index/DocumentsWriterPerThread.java]
         → DefaultIndexingChain.processDocument() [index/DefaultIndexingChain.java]
           → FreqProxTermsWriterPerField.addTerm() [index/FreqProxTermsWriter.java]

   IndexSearcher.search() [search/IndexSearcher.java]
     → createWeight() [search/IndexSearcher.java]
       → Query.createWeight() [search/Query.java]
         → Query.rewrite() [search/Query.java]
     → Weight.scorer() [search/Weight.java]
       → TermWeight.scorer() [search/TermQuery.java inner class]
         → TermScorer constructor [search/TermScorer.java]
     → BulkScorer.score() [search/BulkScorer.java]
       → TopScoreDocCollector.collect() [search/TopScoreDocCollector.java]

   Lucene99Codec [codecs/lucene99/Lucene99Codec.java]
     → Lucene99PostingsFormat [codecs/lucene99/Lucene99PostingsFormat.java]
       → BlockTreeTermsWriter [codecs/blocktree/BlockTreeTermsWriter.java]
         → FST [util/fst/FST.java]
       → Lucene99PostingsWriter [codecs/lucene99/Lucene99PostingsWriter.java]
     → Lucene99DocValuesFormat [codecs/lucene99/Lucene99DocValuesFormat.java]
       → Lucene99DocValuesConsumer [codecs/lucene99/Lucene99DocValuesConsumer.java]
     → Lucene99StoredFieldsFormat [codecs/lucene99/Lucene99StoredFieldsFormat.java]

// Check segment count
int segmentCount = writer.getSegmentInfos().size();
if (segmentCount > 50) {
    // Force merge to reduce segments (do during low traffic!)
    writer.forceMerge(10);  // Target 10 segments max
}

// Or increase merge aggressiveness
TieredMergePolicy policy = new TieredMergePolicy();
policy.setSegmentsPerTier(5.0);  // Default 10, lower = more aggressive merging
config.setMergePolicy(policy);

// Throttle merges
ConcurrentMergeScheduler scheduler = new ConcurrentMergeScheduler();
scheduler.setMaxMergesAndThreads(2, 1);  // Max 2 merges, 1 thread
config.setMergeScheduler(scheduler);

// Or use a rate limiter
scheduler.setMaxMergesAndThreads(3, 2);
// Additional: setMaxMergeCount() and setMaxThreadCount() per your I/O capacity

// Reduce RAM buffer
config.setRAMBufferSizeMB(64.0);  // Down from 256MB

// Or use max buffered docs instead
config.setMaxBufferedDocs(10000);
config.setRAMBufferSizeMB(IndexWriterConfig.DISABLE_AUTO_FLUSH);  // Disable RAM trigger

// Limit concurrent DWPTs (threads)
// Lucene limits this automatically based on RAM buffer / (maxThreadStates * 2)
// But you can control: config.setIndexerThreads(Math.min(8, Runtime.getRuntime().availableProcessors()));

// WRONG:
doc.add(new TextField("price", "49.99", Field.Store.YES));  // String comparison! Slow!

// RIGHT:
doc.add(new IntPoint("price", 4999));  // BKD tree! Fast!
doc.add(new StoredField("price", 4999));  // Store original for retrieval

// Force merge segments with many deleted docs
writer.forceMergeDeletes(true);  // Merge segments with > 10% deleted

// Or configure merge policy
TieredMergePolicy policy = new TieredMergePolicy();
policy.setForceMergeDeletesPctAllowed(5);  // Merge when 5% deleted
config.setMergePolicy(policy);

// Always use the same analyzer at index and query time
Analyzer analyzer = new StandardAnalyzer();

IndexWriterConfig config = new IndexWriterConfig(analyzer);  // Index time
QueryParser parser = new QueryParser("body", analyzer);       // Query time

// Verify with token analysis
// "Foxes" at index time → ["fox"] (PorterStemFilter)
// "Foxes" at query time → ["fox"] (same analyzer) ✓
// If query time was KeywordAnalyzer → ["foxes"] ✗ (no match!)

// Use NativeFSLockFactory (default, most reliable)
Directory dir = new NIOFSDirectory(Paths.get("/index"), NativeFSLockFactory.INSTANCE);

// Or for single-process, in-memory lock:
Directory dir = new NIOFSDirectory(Paths.get("/index"), new SingleInstanceLockFactory());

// Check for stale locks (if JVM crashed)
// Remove write.lock file manually if you're sure no other process is using the index

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.*;
import org.apache.lucene.index.*;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.search.*;
import org.apache.lucene.store.*;
import java.nio.file.Paths;

public class BasicSearchExample {
    public static void main(String[] args) throws Exception {
        // 1. Setup
        Directory dir = new MMapDirectory(Paths.get("/tmp/lucene-index"));
        Analyzer analyzer = new StandardAnalyzer();
        IndexWriterConfig config = new IndexWriterConfig(analyzer);
        IndexWriter writer = new IndexWriter(dir, config);

        // 2. Index documents
        Document doc1 = new Document();
        doc1.add(new TextField("title", "Lucene in Action", Field.Store.YES));
        doc1.add(new TextField("body", "Lucene is a powerful search library", Field.Store.YES));
        doc1.add(new IntPoint("year", 2024));
        doc1.add(new StoredField("year", 2024));
        writer.addDocument(doc1);

        Document doc2 = new Document();
        doc2.add(new TextField("title", "Search Engine Architecture", Field.Store.YES));
        doc2.add(new TextField("body", "Building search systems with Lucene and Elasticsearch", Field.Store.YES));
        doc2.add(new IntPoint("year", 2023));
        doc2.add(new StoredField("year", 2023));
        writer.addDocument(doc2);

        writer.commit();
        writer.close();

        // 3. Search
        DirectoryReader reader = DirectoryReader.open(dir);
        IndexSearcher searcher = new IndexSearcher(reader);
        QueryParser parser = new QueryParser("body", analyzer);
        Query query = parser.parse("lucene search");

        TopDocs results = searcher.search(query, 10);
        System.out.println("Total hits: " + results.totalHits);

        for (ScoreDoc scoreDoc : results.scoreDocs) {
            Document doc = searcher.doc(scoreDoc.doc);
            System.out.printf("Score: %.2f, Title: %s%n", 
                scoreDoc.score, doc.get("title"));
        }

        reader.close();
        dir.close();
    }
}

import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.core.*;
import org.apache.lucene.analysis.en.*;
import org.apache.lucene.analysis.miscellaneous.*;
import org.apache.lucene.analysis.synonym.*;
import org.apache.lucene.analysis.standard.*;

public class CustomAnalyzerExample {
    public static Analyzer createEcommerceAnalyzer() {
        return new Analyzer() {
            @Override
            protected TokenStreamComponents createComponents(String fieldName) {
                Tokenizer tokenizer = new StandardTokenizer();
                TokenStream stream = new LowerCaseFilter(tokenizer);
                stream = new StopFilter(stream, EnglishAnalyzer.ENGLISH_STOP_WORDS_SET);

                // Synonyms
                SynonymMap.Builder builder = new SynonymMap.Builder(true);
                builder.add(new CharsRef("laptop"), new CharsRef("notebook"), true);
                builder.add(new CharsRef("phone"), new CharsRef("mobile"), true);
                try {
                    SynonymMap synonymMap = builder.build();
                    stream = new SynonymGraphFilter(stream, synonymMap, true);
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }

                // Stemming
                stream = new PorterStemFilter(stream);

                // Edge n-grams for autocomplete (2-10 chars)
                stream = new EdgeNGramTokenFilter(stream, 2, 10);

                return new TokenStreamComponents(tokenizer, stream);
            }
        };
    }

    public static void main(String[] args) throws Exception {
        Analyzer analyzer = createEcommerceAnalyzer();
        String text = "The quick brown foxes jump over the laptop!";

        try (TokenStream stream = analyzer.tokenStream("body", text)) {
            CharTermAttribute termAttr = stream.addAttribute(CharTermAttribute.class);
            stream.reset();
            while (stream.incrementToken()) {
                System.out.println("Token: " + termAttr.toString());
            }
            stream.end();
        }
        // Output: qu, qui, quic, quick, br, bro, brow, brown, etc.
    }
}

import org.apache.lucene.index.*;
import org.apache.lucene.search.*;
import org.apache.lucene.util.*;

// A custom query that boosts documents containing a term in the first 100 positions
public class EarlyPositionBoostQuery extends Query {
    private final Term term;
    private final int maxPosition;
    private final float boost;

    public EarlyPositionBoostQuery(Term term, int maxPosition, float boost) {
        this.term = term;
        this.maxPosition = maxPosition;
        this.boost = boost;
    }

    @Override
    public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {
        Weight innerWeight = new TermQuery(term).createWeight(searcher, scoreMode, boost);

        return new Weight(this) {
            @Override
            public Scorer scorer(LeafReaderContext context) throws IOException {
                Scorer innerScorer = innerWeight.scorer(context);
                if (innerScorer == null) return null;

                return new Scorer(this) {
                    @Override
                    public DocIdSetIterator iterator() {
                        return innerScorer.iterator();
                    }

                    @Override
                    public float getMaxScore(int upTo) throws IOException {
                        return innerScorer.getMaxScore(upTo) * EarlyPositionBoostQuery.this.boost;
                    }

                    @Override
                    public float score() throws IOException {
                        int doc = innerScorer.docID();

                        // Check if term appears in first 100 positions
                        PostingsEnum postings = context.reader().postings(term, PostingsEnum.POSITIONS);
                        if (postings != null && postings.advance(doc) == doc) {
                            for (int i = 0; i < postings.freq(); i++) {
                                int pos = postings.nextPosition();
                                if (pos < maxPosition) {
                                    return innerScorer.score() * EarlyPositionBoostQuery.this.boost;
                                }
                            }
                        }
                        return innerScorer.score();
                    }

                    @Override
                    public int docID() {
                        return innerScorer.docID();
                    }
                };
            }

            @Override
            public boolean isCacheable(LeafReaderContext ctx) {
                return false;
            }

            @Override
            public Explanation explain(LeafReaderContext context, int doc) throws IOException {
                return innerWeight.explain(context, doc);
            }
        };
    }

    @Override
    public String toString(String field) {
        return "EarlyPositionBoost(" + term + ", pos<" + maxPosition + ", boost=" + boost + ")";
    }

    @Override
    public boolean equals(Object other) {
        return sameClassAs(other) && term.equals(((EarlyPositionBoostQuery) other).term);
    }

    @Override
    public int hashCode() {
        return classHash() ^ term.hashCode();
    }

    @Override
    public void visit(QueryVisitor visitor) {
        visitor.visitLeaf(this);
    }
}

import org.apache.lucene.search.*;

// A custom scorer that boosts recent documents
public class RecencyBoostScorer extends Scorer {
    private final Scorer innerScorer;
    private final long currentTime;
    private final float halfLifeDays;
    private final NumericDocValues timestampValues;

    public RecencyBoostScorer(Weight weight, Scorer innerScorer, 
                               NumericDocValues timestampValues, 
                               float halfLifeDays) {
        super(weight);
        this.innerScorer = innerScorer;
        this.timestampValues = timestampValues;
        this.currentTime = System.currentTimeMillis();
        this.halfLifeDays = halfLifeDays;
    }

    @Override
    public DocIdSetIterator iterator() {
        return innerScorer.iterator();
    }

    @Override
    public float getMaxScore(int upTo) throws IOException {
        return innerScorer.getMaxScore(upTo) * 2.0f;  // Max possible boost
    }

    @Override
    public float score() throws IOException {
        float baseScore = innerScorer.score();
        int doc = docID();

        if (timestampValues.advanceExact(doc)) {
            long docTime = timestampValues.longValue();
            long ageMs = currentTime - docTime;
            double ageDays = ageMs / (1000.0 * 60 * 60 * 24);
            double decay = Math.pow(0.5, ageDays / halfLifeDays);  // Exponential decay
            return baseScore * (float)(1.0 + decay);  // Recent docs get up to 2x boost
        }
        return baseScore;
    }

    @Override
    public int docID() {
        return innerScorer.docID();
    }
}

import org.apache.lucene.facet.*;
import org.apache.lucene.facet.sortedset.*;

// Setup: index with facets
Directory dir = new MMapDirectory(Paths.get("/tmp/facet-index"));
FacetFields facetFields = new FacetFields(taxoWriter);

Document doc = new Document();
doc.add(new TextField("title", "Product A", Field.Store.YES));

// Add facets as drill-down paths
List<FacetField> facets = new ArrayList<>();
facets.add(new FacetField("category", "Electronics", "Computers"));
facets.add(new FacetField("price_range", "100-200"));
facets.add(new FacetField("brand", "Apple"));
doc.add(new FacetField("category", "Electronics", "Computers"));

writer.addDocument(doc);

// Search with faceting
DirectoryReader reader = DirectoryReader.open(dir);
IndexSearcher searcher = new IndexSearcher(reader);

// Facet configuration
FacetsConfig facetConfig = new FacetsConfig();
facetConfig.setMultiValued("category", true);
facetConfig.setHierarchical("category", true);

// Search and collect facets
FacetsCollector facetsCollector = new FacetsCollector();
TopDocs results = FacetsCollector.search(searcher, query, 10, facetsCollector);

// Get facet counts
Facets facets = new SortedSetDocValuesFacetCounts(state, facetsCollector);
FacetResult categoryResult = facets.getTopChildren(10, "category");
FacetResult priceResult = facets.getTopChildren(10, "price_range");

// Print facet counts
System.out.println("Categories:");
for (LabelAndValue lv : categoryResult.labelValues) {
    System.out.println("  " + lv.label + ": " + lv.value);
}
// Output:
//   Electronics: 150
//   Electronics/Computers: 80
//   Electronics/Phones: 70

import org.apache.lucene.search.highlight.*;

// Setup
QueryParser parser = new QueryParser("body", analyzer);
Query query = parser.parse("lucene search");

// Using UnifiedHighlighter (recommended)
UnifiedHighlighter highlighter = new UnifiedHighlighter(searcher, analyzer);
String[] snippets = highlighter.highlight("body", query, results, 3);  // 3 snippets

for (String snippet : snippets) {
    System.out.println(snippet);
}
// Output: "<b>Lucene</b> is a powerful <b>search</b> library"

// Using classic Highlighter (more control)
QueryScorer scorer = new QueryScorer(query);
Fragmenter fragmenter = new SimpleSpanFragmenter(scorer, 100);
Highlighter classicHighlighter = new Highlighter(scorer);
classicHighlighter.setTextFragmenter(fragmenter);

TokenStream tokenStream = analyzer.tokenStream("body", doc.get("body"));
String snippet = classicHighlighter.getBestFragment(tokenStream, doc.get("body"));

import org.apache.lucene.search.spell.*;
import org.apache.lucene.index.*;

// Build spell index from existing index
Directory spellIndexDir = new MMapDirectory(Paths.get("/tmp/spell-index"));
SpellChecker spellChecker = new SpellChecker(spellIndexDir);

// Index the dictionary from the main index
Dictionary dictionary = new LuceneDictionary(reader, "title");
spellChecker.indexDictionary(dictionary, new IndexWriterConfig(analyzer), true);

// Suggest corrections
String word = "lucne";
int numSuggestions = 5;
String[] suggestions = spellChecker.suggestSimilar(word, numSuggestions);
// Output: ["lucene", "lucien", "lune", ...]

// Did-you-mean suggestion for a full query
String userQuery = "lucne serch";
String[] words = userQuery.split(" ");
StringBuilder didYouMean = new StringBuilder();
for (String w : words) {
    String[] similar = spellChecker.suggestSimilar(w, 1);
    didYouMean.append(similar.length > 0 ? similar[0] : w).append(" ");
}
System.out.println("Did you mean: " + didYouMean.toString().trim());
// Output: "Did you mean: lucene search"

import org.apache.lucene.queries.mlt.MoreLikeThis;

// Find documents similar to document 42
MoreLikeThis mlt = new MoreLikeThis(reader);
mlt.setFieldNames(new String[]{"title", "body"});
mlt.setAnalyzer(analyzer);
mlt.setMinTermFreq(1);   // Ignore terms that appear less than this in source doc
mlt.setMinDocFreq(1);    // Ignore terms that appear in less than this many docs
mlt.setMaxQueryTerms(25); // Max terms to include in generated query

Query likeQuery = mlt.like(42);  // Generate query from doc 42
TopDocs similarDocs = searcher.search(likeQuery, 10);

System.out.println("Documents similar to #42:");
for (ScoreDoc sd : similarDocs.scoreDocs) {
    if (sd.doc != 42) {  // Exclude the source document
        Document doc = searcher.doc(sd.doc);
        System.out.printf("  Score: %.2f, Title: %s%n", sd.score, doc.get("title"));
    }
}

// Or from external text:
Reader textReader = new StringReader("Apache Lucene is a search library...");
Query fromTextQuery = mlt.like(textReader);

import org.apache.lucene.document.KnnVectorField;
import org.apache.lucene.search.KnnVectorQuery;
import org.apache.lucene.index.VectorSimilarityFunction;

// Index documents with vector embeddings
float[] docVector = embeddingModel.embed("Lucene search library document");
Document doc = new Document();
doc.add(new TextField("title", "Lucene Guide", Field.Store.YES));
doc.add(new KnnVectorField("embedding", docVector, VectorSimilarityFunction.COSINE));
writer.addDocument(doc);

// Search by vector similarity
float[] queryVector = embeddingModel.embed("search engine library");
Query knnQuery = new KnnVectorQuery("embedding", queryVector, 100);

// Combine with text query for hybrid search
Query textQuery = new TermQuery(new Term("title", "lucene"));
Query hybridQuery = new BooleanQuery.Builder()
    .add(knnQuery, BooleanClause.Occur.SHOULD)
    .add(textQuery, BooleanClause.Occur.SHOULD)
    .build();

TopDocs results = searcher.search(hybridQuery, 10);

// Results ordered by combined vector + text relevance
for (ScoreDoc sd : results.scoreDocs) {
    Document doc = searcher.doc(sd.doc);
    System.out.printf("Score: %.4f, Title: %s%n", sd.score, doc.get("title"));
}

import org.apache.lucene.index.*;
import org.apache.lucene.search.*;

// Setup
Directory dir = new MMapDirectory(Paths.get("/tmp/nrt-index"));
IndexWriterConfig config = new IndexWriterConfig(analyzer);
IndexWriter writer = new IndexWriter(dir, config);

// Add initial document
writer.addDocument(doc1);
writer.commit();  // Durable commit

// Open initial NRT reader
DirectoryReader nrtReader = DirectoryReader.open(writer);
IndexSearcher searcher = new IndexSearcher(nrtReader);

// Search sees doc1
TopDocs results1 = searcher.search(new MatchAllDocsQuery(), 10);
System.out.println("Hits after initial: " + results1.totalHits);  // 1

// Add new document WITHOUT commit!
writer.addDocument(doc2);
// No commit! But NRT reader can still see it after reopen

// Reopen NRT reader (sees uncommitted doc2!)
DirectoryReader newReader = DirectoryReader.openIfChanged(nrtReader);
if (newReader != null) {
    nrtReader.close();
    nrtReader = newReader;
    searcher = new IndexSearcher(nrtReader);
}

// Search now sees doc2
TopDocs results2 = searcher.search(new MatchAllDocsQuery(), 10);
System.out.println("Hits after NRT reopen: " + results2.totalHits);  // 2

// doc2 is NOT durable yet! If JVM crashes now, doc2 is lost.
writer.commit();  // Now doc2 is durable