tag:blogger.com,1999:blog-8623074010562846957.post5673949079038022071..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: Building a new Lucene postings formatMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-8623074010562846957.post-84152817593671858712017-11-21T10:02:36.194-05:002017-11-21T10:02:36.194-05:00Hi Igor, I suggest starting with Lucene's &quo...Hi Igor, I suggest starting with Lucene's "demo" module, specifically IndexFiles.java and SearchFiles.java. And then send questions to Lucene's users list (java-user@lucene.apache.org).Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-74880018475254254542017-11-18T02:01:03.419-05:002017-11-18T02:01:03.419-05:00Hello Mike,
I'm starting to study Lucene, and ...Hello Mike,<br />I'm starting to study Lucene, and I'm curious to see how the inverted index of the document gets after indexing some files.<br />I would like to know what steps I should take and which classes to use.<br />Thanks if you can respond.Anonymoushttps://www.blogger.com/profile/05575192824370435926noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-49280627108942437902014-07-07T06:40:56.929-04:002014-07-07T06:40:56.929-04:00Hi Mike,
Thanks a lot for your reply. I could solv...Hi Mike,<br />Thanks a lot for your reply. I could solve some of the easier issues listed above. Will put a new question based on the remaining issues.Aditya Tripathihttps://www.blogger.com/profile/02229285858723837318noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-52720945138080316862014-07-06T06:31:04.658-04:002014-07-06T06:31:04.658-04:00Hi Aditya,
Sorry for the slow response here; if y...Hi Aditya,<br /><br />Sorry for the slow response here; if you haven't already, could you ask this question on the Lucene users list instead? (java-user@lucene.apache.org).Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-26688775823412930092014-06-16T06:06:48.150-04:002014-06-16T06:06:48.150-04:00Hi Mike,
We were trying to use a configurable PerF...Hi Mike,<br />We were trying to use a configurable PerFieldPostingsFormat to get an updateable field with a key-value store. Something similar to what you have linked in your post(Flax). However, I have a doubt that an updateable field is possible using codecs for the following reasons:<br /><br />Assuming the key for this key-value store is: segmentName_fieldName_term and value is the postingsArray: say just array of docIds.<br /><br />Assume there is a field PK (id kind of field) which is stored and uses default codec.<br /><br />#1. fieldConsumers of PostingsFormat are called only at flush time. With this design handling an update for a document not yet flushed seems difficult for a field writing to a key-value store.<br />So if I add a document with say PK:1 and then add another partial update document for PK:1 then to update this, we first need to figure out the segmentName for this document, only then we can update it in key-value store. But this doc has no segmentName as of now. Because of DWPTs we can't even assume that the document will be in this same segment.<br /><br />Note: Since the custom PostingsFormat is only called at flush time, we write the postings data directly to the key-value store.<br /><br />#2:Merged segments are just checkpointed, they are not committed. So, for the custom TermsConsumer.merge() call we can not write the new merged state to the key-value store directly. Note that merge() and flush() both use same methods like startTerm(),startDoc(),finishTerm() etc. And since in these methods the design is to write directly to key-value store, we do not use TermsConsumer.merge() function. We actually put the new merged state in-memory but have no good signal to flush it to the key-value store. We are flushing this in-memory merge state at the next flush. However, Lucene can commit the checkpointed merged segment without any document to be flushed. So we get into a state where Lucene Index Dir has new merged segments, the key-value store is not yet updated with that. If we write the merged state directly, then we get into a state where key-value store has the new merged segment but not the Lucene Directory.<br /><br />#3 - Dropped Segments. Again can't handle this as the only time our custom consumers can remove deleted docs is at merge time. But Lucene can drop segments during normal commits.<br /><br />#4 - Replication: We do take index files and key-value store files for replication. We can not keep both of them in sync.<br /><br />#5 - Applying updates to a reindexed document when reindexed document is taken by a DWPT which is flushed after the partial update document. Updates are lost in this case.<br /><br />Currently, I plan to try adding SegmentInfosFormat(May be LiveDocsFormat too) and a DirectoryWrapper but as if now I do not have any idea if it will solve all problems. <br /><br />It will be great to know your thoughts on this on the feasibility of providing updateable fields using codecs.Aditya Tripathihttps://www.blogger.com/profile/02229285858723837318noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-44992762301259110992014-04-03T15:53:57.520-04:002014-04-03T15:53:57.520-04:00I will convert and run all span queries as conjunc...I will convert and run all span queries as conjunction queries and then use the output as filter query for the original span query. That might reduce the scope of span query substantially. <br /><br />Thanks for the idea.Gopalhttps://www.blogger.com/profile/14245097139079980069noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-40018188989903397952014-04-02T15:12:44.072-04:002014-04-02T15:12:44.072-04:00Span queries are unfortunately very costly. Have ...Span queries are unfortunately very costly. Have you tried a sloppy PhraseQuery instead? They will still be costly but perhaps less-so.<br /><br />However, even better: do a simpler (non-positional) query and then use the new rescoring APIs to re-sort the top e.g. 500 hits with your positional queries.<br /><br />DirectPostingsFormat is a very RAM-heavy, but fast, postings format.<br /><br />You don't need termOffsets for span queries; that's typically used for highlighting with PostingsHighlighter.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-21386072449203170912014-04-02T13:29:48.631-04:002014-04-02T13:29:48.631-04:00Thanks, so is there a better postingsformat to cho...Thanks, so is there a better postingsformat to choose from while using span queries in solr 4.6 or solr4.7? Since, span queries utilize all the parts of postings like terms, documents, frequencies, positions and offsets.<br /><br />When I am debugging the lucene 4.6 test cases for span queries, it is showing that for above nextdoc() call it is utilizing DirectPostingsFormat.<br /><br />Does having termPositions=true and termOffsets=true help?Gopalhttps://www.blogger.com/profile/14245097139079980069noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-74988878790560141202014-04-02T13:15:53.420-04:002014-04-02T13:15:53.420-04:00Yes, Lucene41PF is still the default, so it's ...Yes, Lucene41PF is still the default, so it's used for all queries to iterate docs/positions. MemoryPF isn't really a good match: it's best for primary key fields (each term matches just one document). Term vectors are unrelated: they are not used during searching, and they are quite slow to retrieve per document.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-73546873374395379022014-04-01T17:37:16.869-04:002014-04-01T17:37:16.869-04:00Does lucene 4.7 use Lucene41PostingsFormat for Pos...Does lucene 4.7 use Lucene41PostingsFormat for Postings.nextdoc() while executing the span queries? My requirement is to run multiple span queries like [cat dog]~2 on 2 TB of index and I am worried about the performance as I have to collect all the docs in results. Is there a better postings format to choose from like memory postings while using span queries in solr 4.7?? Does having termvectors help me in this regard?Gopalhttps://www.blogger.com/profile/14245097139079980069noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-11383335219312123242013-08-21T07:34:37.681-04:002013-08-21T07:34:37.681-04:00I don't think there's any documentation fo...I don't think there's any documentation for this besides the source code itself; could you ask on the dev list (dev@lucene.apache.org)?Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-58957054683181300002013-08-20T16:26:25.969-04:002013-08-20T16:26:25.969-04:00I read http://lucene.apache.org/core/3_0_3/filefor...I read http://lucene.apache.org/core/3_0_3/fileformats.html but not sure how things work when a query arrives.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-90257482782924065372013-08-20T16:22:33.241-04:002013-08-20T16:22:33.241-04:00Is there documentation about how lucene works inte...Is there documentation about how lucene works internally when a one does a query say "title:Foo" ? e.g. how the tii, tis, prx, fdt, etc.. lookuped and in what order to get the information.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-6048532296787677692013-05-28T07:47:25.935-04:002013-05-28T07:47:25.935-04:00Thanks Gora, the only IR book is a wonderful resou...Thanks Gora, the only IR book is a wonderful resource!Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-55714439765369856912013-05-28T07:29:49.926-04:002013-05-28T07:29:49.926-04:00Some advice to all of those asking about what post...Some advice to all of those asking about what postings are: have a look at the online IR Book at http://www-nlp.stanford.edu/IR-book/ . The first few chapters will help you understand the theory behind postings and inverted index. After reading the IR Book, this blog post makes much more sense! :-) Thanks Mike.Anonymoushttps://www.blogger.com/profile/18427571745324544409noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-60841988276700215432013-05-10T15:42:13.812-04:002013-05-10T15:42:13.812-04:00Hi John,
That's a good question ... I'm n...Hi John,<br /><br />That's a good question ... I'm not sure? If any of the codec APIs changed (which is likely: they are marked experimental) then you'd need to upgrade lucene core as well. Try it and report back ;)Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-36887449904427443012013-05-10T09:22:02.158-04:002013-05-10T09:22:02.158-04:00Thanks Mike. Lot clear to me now :)
I see that Lu...Thanks Mike. Lot clear to me now :)<br /><br />I see that Lucene 4 used Lucene40 Postings format by default. Would it be possible to use BlockPostingsFormat and still be on Lucene4 or do we need to upgrade to Lucene4.1?<br /><br />-JohnAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-61194938978592202162013-05-09T06:38:28.760-04:002013-05-09T06:38:28.760-04:00Hi Anonymous,
A Codec holds many formats: posting...Hi Anonymous,<br /><br />A Codec holds many formats: postings, term vectors, deletions, etc. PostingsFormat just covers how postings (= the inverted part of the index, i.e. fields, terms, docs, positions, freqs, offsets) are written.<br /><br />There are very different performance characteristics for each PostingsFormat ... I would say the best one is the current default one (BlockPostingsFormat): it's fast for frequent terms because it bulk encodes/decodes the integers.<br /><br />I don't think there's any good document that describes the steps in building a PostingsFormat. I would start by looking at the PostingsFormat impls in the Lucene sources? Typically one builds a PostingsBaseFormat, which plugs into the terms dictionary, because building a new terms dictionary is challenging ... the PostingsBaseFormat only needs to encode the docs/freqs/positions/offsets.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-72843087462283576782013-05-09T05:08:11.621-04:002013-05-09T05:08:11.621-04:00Mike
Lucene newbie here. Sorry if my question sou...Mike<br /><br />Lucene newbie here. Sorry if my question sounds silly.<br />Still trying to understand the terminology related to Postings/PostingFormat/Codec. Are they all same?<br /><br />From what you said, if custom postingsformat controls how the data is stored in index, would it be right to say that for the same dataset being loaded into index, different postingformats have different performance times?<br /><br />What would be the best postingformat to use if speed is important.<br /><br />Also can you point me to documentation/example to build custom postingformat using java?<br /><br />Thanks<br />JohnAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-31668137312404455212013-01-03T07:03:52.345-05:002013-01-03T07:03:52.345-05:00Thanks Mike Sir for answering.Thanks Mike Sir for answering.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-7533454335736629312012-10-13T11:21:12.528-04:002012-10-13T11:21:12.528-04:00The PostingsFormat controls how the inverted index...The PostingsFormat controls how the inverted index (terms mapping to list of documents that contain that term, plus things like frequency, positions, offsets) is stored in the index.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-79052064163330966792012-10-13T10:48:39.218-04:002012-10-13T10:48:39.218-04:00what is Posting format ?
Is it something related t...what is Posting format ?<br />Is it something related to the way string is sent to the searcher ??Anonymousnoreply@blogger.com