We had two projects this summer; here I'll describe Han Jiang's project (I was his mentor). The second project, separating
IndexableField, is also very exciting and wrapping up now.
The goal of Han's project was to create a compelling replacement int-block postings format, improving on the aging variable-int postings format that Lucene has used forever. The project was definitely a success! This morning I merged over the feature branch we had been using all summer, committing to both Lucene's trunk (future 5.x) and Lucene 4.x branch (to be 4.0 GA). This means Lucene 4.0 GA will have the new
The new postings format shows incredible search performance gains versus the current default postings format. I tested with luceneutil on the full May 2012 English Wikipedia index (33.3 million documents, 15 GB index):
|Task||QPS base||StdDev base||QPS packed||StdDev packed||% change|
The gains are awesome! After this change has baked for a while I expect we'll make it the default postings format, maybe in time for 4.1.
This effort has been a long time coming: ever since the creation of flexible indexing, allowing for full pluggability of all files in the index, we have wanted to improve on Lucene's postings format. Early int-block prototypes showed great performance.
The new format stores postings data in fixed blocks of 128 ints. Each block is encoded using a simple packed ints format. The
.docfile interleaves docID delta and term frequency blocks, and also includes skip data, while the
.posfile holds position deltas. Offsets and payloads, if present, are stored in the
.payfile. On the full English Wikipedia index used in the above test this resulted in a nice reduction of disk usage: 13.7 GB (new) vs 14.7 GB (current). That's ~7% smaller.
The format only uses packed int blocks for high-frequency terms: low frequency terms (that appear in less than 128 documents), and the final partial-block of a high frequency term, are stored in the same variable-int format as today's default. Skip points are stored at the start of each block (except the first block).
Han built and tested a number of fun ideas, including skipping within blocks (with partial block decoding), using PForDelta instead of simple packed ints, wrapping with
PulsingPostingsFormat, all sorts of different ways to decode packed ints, different values for
skipMultiplier, encoding all-single-values very compactly, etc. We spent lots of electricity running all sorts of benchmarks. Some ideas worked and some didn't, and things did not go according to the original plan! Such is the healthy unpredictable, zig-zag, iterative nature of open-source development/exploration, yet in the end we find ourselves with a great new postings format.
Thank you Han Jiang for such an important improvement to Lucene, and thank you Google for sponsoring this effort through Google Summer of Code.