There are no promises for the exact timing (it's done when it's done!), but we already have a volunteer release manager (thank you Anshum!).
A major release in Lucene means all deprecated APIs (as of 4.10.x) are dropped, support for 3.x indices is removed while the numerous 4.x index formats are still supported for index backwards compatibility, and the 4.10.x branch becomes our bug-fix only release series (no new features, no API changes).
5.0.0 already contains a number of exciting changes, which I describe below, and they are still rolling in with ongoing active development.
Stronger index safety
Many of the 5.0.0 changes are focused on providing stronger protection against index corruption.
access now uses
NIO.2 APIs, giving us better error handling
Files.delete returns a meaningful exception) along
with atomic rename for
commits, reducing the risk of hideous "your entire index is gone"
Lucene's replication module, along with distributed servers on top of Lucene such as Elasticsearch or Solr, must copy index files from one place to another. They do this for backup purposes (e.g., snapshot and restore), for migrating or recovering a shard from one node to another or when adding a new replica. Such replicators try to be incremental, so that if the same file name is present, with the same length and checksum, it will not be copied again.
Unfortunately, these layers sometimes have subtle bugs (they are complex!). Thanks to checksums (added in 4.8.0), Lucene already detects if the replicator caused any bit-flips while copying, and this revealed a long standing nasty bug in the compression library Elasticsearch uses.
With 5.0.0 we take this even further and now detect if whole files
were copied to the wrong file name, by assigning
id to every segment and commit (
Each index file now records
id in its header, and then these ids are cross-checked when the
index is opened.
The new Lucene50Codec also includes further index corruption detection.
CorruptIndexException itself is improved! It will
now always refer to the file or resource where the corruption was
detected, as this is now a required argument to its constructors.
When corruption is detected higher up (e.g., a bad field number in the
field infos file), the resulting
will now state whether there was also a checksum mismatch in the file,
helping to narrow the possible source of the corruption.
Finally, during merge,
IndexWriter now always checks the
incoming segments for corruption before merging. This can mean, on
upgrading to 5.0.0, that merging may uncover long-standing latent
corruption in an older 4.x index.
Reduced heap usage
5.0.0 also includes several changes to reduce heap usage during indexing and searching.
If your index has 1B docs, then caching a
FixedBitSet-based filter in
costs a non-trivial 125 MB of heap! But with 5.0.0, Lucene now
supports random-writable and advance-able sparse bitsets
so the heap required is in proportion to how many bits are set, not how
many total documents exist in the index. These bitsets also greatly
MultiTermQuery is rewritten (no
CONSTANT_SCORE_AUTO_REWRITE_METHOD), and they
provide faster advance implementations than
linear scan. Finally, they provide a more
cost() implementation, allowing Lucene to make
better choices about how to drive the intersection at query time.
Heap usage during
IndexWriter merging is
lower with the new Lucene50Codec, since doc values and norms for
the segments being merged are no longer fully loaded into heap for all
fields; now they are loaded for the one field currently being merged, and then
The default norms format now uses sparse encoding when appropriate, so indices that enable norms for many sparse fields will see a large reduction in required heap at search time.
An explain API for heap usage
If you still find Lucene using more heap than you expected, 5.0.0 has a new API to print a tree structure showing a recursive breakdown of which parts are using how much heap. This is analogous to Lucene's explain API, used to understand why a document has a certain relevance score, but applied to heap usage instead.
It produces output like this:
_cz(5.0.0):C8330469: 28MB postings [...]: 5.2MB ... field 'latitude' [...]: 678.5KB term index [FST(nodes=6679, ...)]: 678.3KBThis is a much faster way to see what is using up your heap than trying to stare at a Java heap dump.
There is a long tail of additional 5.0.0 changes; here are some of them:
- Old experimental postings formats
Sep/Fixed/VariableIntPostingsFormat) have been removed. PulsingPostingsFormat has also been removed, since the default postings format already pulses unique terms.
FieldCacheis gone (moved to a dedicated
miscmodule). This means when you intend to sort on a field, you should index that field using doc values, which is much faster and less heap consuming than
Analyzers no longer require
NormsFormatnow gets its own dedicated
- Simplifications to
FieldInfo(Lucene's "low schema"): no more
normType(it is always a
DocValuesType.NUMERIC), no more
- Compound file handling is simpler, and is now under codec control.
SortedSetSortField, used to sort on a multi-valued field, is promoted from sandbox to Lucene's core
PostingsFormatnow uses a "pull" API when writing postings, just like doc values. This is powerful because you can do things in your postings format that require making more than one pass through the postings such as iterating over all postings for each term to decide which compression format it should use.
- Version is no longer required on init to classes
IndexWriterConfigand analysis components.