tag:blogger.com,1999:blog-8623074010562846957.post8384420114320174537..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: Lucene's new BlockPostingsFormat, thanks to Google Summer of CodeMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-8623074010562846957.post-38279087747556671612019-01-16T05:57:18.060-05:002019-01-16T05:57:18.060-05:00HI Mike:
I have a lucene performance question, my ...HI Mike:<br />I have a lucene performance question, my embedded hardware have a bad random io performance. So i want to enlarge the block the postingsFormat block size to load more data in one io, do you think it is resonable.<br />While based on lucene api introduce(http://lucene.apache.org/core/4_10_2/core/org/apache/lucene/index/IndexWriterConfig.html#setTermIndexInterval%28int%29), Large values cause less memory to be used by IndexReader, but slow random-access to terms.<br /><br />Thanks<br />Carl ChenAnonymoushttps://www.blogger.com/profile/04826422158253831105noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-91755623598891187142012-11-23T09:24:57.657-05:002012-11-23T09:24:57.657-05:00Hi Ramprakash,
Yes, BlockPostingsFormat is availa...Hi Ramprakash,<br /><br />Yes, BlockPostingsFormat is available in 4.0, however it's not the default format, and I think the format has changed from 4.0 to 4.1 so if you index with it you'll have to re-index after upgrading.<br /><br />As of 4.1 it's the default and will have full back compat ...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-68020702651572679392012-11-23T06:41:31.804-05:002012-11-23T06:41:31.804-05:00Hey Mike and Han, Kudos to the great work.
Just a...Hey Mike and Han, Kudos to the great work.<br /><br />Just a clarification, does this block posting codec comes along with the 4.0 GA? Ramprakashhttps://www.blogger.com/profile/09798950437818378941noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-54495068056903809292012-09-17T20:37:04.559-04:002012-09-17T20:37:04.559-04:00Hi Daniel,
VERY interesting! Thank you for writi...Hi Daniel,<br /><br />VERY interesting! Thank you for writing the paper and sharing your source code under the generous Apache Software License 2. I'll read the paper ... it looks very thorough.<br /><br />I would love to do our packed bit decoding in C w/ SIMD instructions...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-60778679664554389542012-09-17T18:49:34.115-04:002012-09-17T18:49:34.115-04:00We just published a paper where we use exactly the...We just published a paper where we use exactly the same scheme (as far as I can tell). We call it SIMD-BP128. <br /><br />You might be interested in the following blog post:<br /><br />Fast integer compression: decoding billions of integers per second<br />http://lemire.me/blog/archives/2012/09/12/fast-integer-compression-decoding-billions-of-integers-per-second/<br /><br />You will find a link to our paper as well as our C++ in the blog post.<br /><br /><br />Though we wrote our software in C++, I ported some schemes in Java:<br /><br />https://github.com/lemire/JavaFastPFOR<br /><br />Daniel Lemirehttps://www.blogger.com/profile/01566622051558391310noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-89246292096085523562012-08-21T23:38:31.182-04:002012-08-21T23:38:31.182-04:00Thanks Mike! It is always enjoyable to discuss wit...Thanks Mike! It is always enjoyable to discuss with you! I'll be glad to contribute if we need any further optimization!Han Jianghttps://www.blogger.com/profile/02844724203689373576noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-40159193985048863802012-08-21T16:11:50.386-04:002012-08-21T16:11:50.386-04:00Nice one Mike and great to hear a pure success not...Nice one Mike and great to hear a pure success not only with the code but also with the mentoring of the project as a whole. I suspect that your student Han Jiang will be sticking around in the Lucene circle for a while ;0) I'll look forward to your follow up post if/when the code makes it into 4.1!Lewis John McGibbneyhttps://www.blogger.com/profile/17401964450486120371noreply@blogger.com