tag:blogger.com,1999:blog-8623074010562846957.post7527378770560092483..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: 265% indexing speedup with Lucene's concurrent flushingMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-8623074010562846957.post-30464728447610821352015-06-11T11:22:14.211-04:002015-06-11T11:22:14.211-04:00Thanks for the work Michael, this was very good to...Thanks for the work Michael, this was very good to know since I am now working in Petabytes of data.....BDD_1970https://www.blogger.com/profile/17409734887115083705noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-23335764188114287522014-07-07T11:38:54.455-04:002014-07-07T11:38:54.455-04:00Close it when your application needs to shutdown.Close it when your application needs to shutdown.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-48844423930406904002014-07-06T08:52:01.390-04:002014-07-06T08:52:01.390-04:00tks for ur reply ! Michael ~ ,
The Lucene doc...tks for ur reply ! Michael ~ , <br /> The Lucene docs suggest to re-use IndexWriter instance because it's costly close() operation . i agreed with this , but i'm wondering when should close it properly ?Anonymoushttps://www.blogger.com/profile/12332248369833565551noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-55342875759352015452014-07-06T08:47:38.485-04:002014-07-06T08:47:38.485-04:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/12332248369833565551noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-76831166080155948192014-07-02T05:28:39.266-04:002014-07-02T05:28:39.266-04:00Using a thread pool to do indexing is currently no...Using a thread pool to do indexing is currently not done by IndexWriter, i.e. it's up to the application. But I agree a simple Utility class to do this would be a nice addition to Lucene ... maybe you can open a Jira issue and attach an initial patch?Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-44901625411458448602014-07-01T12:30:52.413-04:002014-07-01T12:30:52.413-04:00i've been reading ur post these days , now i&#...i've been reading ur post these days , now i've got a better understanding what the concurrent flushing in Lucene and i made a experiment about it , the speed of Indexing just improved for about 9 times !! here is what i've done , i overwrite the IndexWriter class's addDocument method by bind each addDocument job to a Runnable task , and make a ThreadPoolExecutor to run these tasks . I was thinking Lucene have done these internally ... so , did the process in ur post did the same thing as i do or there's a better way ? Tks advance ~! :DAnonymoushttps://www.blogger.com/profile/12332248369833565551noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-76909159561578161472014-06-27T17:39:39.983-04:002014-06-27T17:39:39.983-04:00No, I mean multiple threads sharing a single Index...No, I mean multiple threads sharing a single IndexWriter...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-79923944527081021392014-06-27T01:13:46.020-04:002014-06-27T01:13:46.020-04:00do u mean mutipul interWriter write to the same in...do u mean mutipul interWriter write to the same index path concurrently ?<br />Anonymoushttps://www.blogger.com/profile/12332248369833565551noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-19671711140179113572012-08-01T05:53:11.732-04:002012-08-01T05:53:11.732-04:00Hi, please ask those questions on the solr-user@lu...Hi, please ask those questions on the solr-user@lucene.apache.org list. Thanks.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-69982731820933833682012-08-01T04:37:48.942-04:002012-08-01T04:37:48.942-04:00hi, i wonder if we can configure the number of ind...hi, i wonder if we can configure the number of indexing threads through solr4 ? <br />also would you mind explaining more on how RAM buffer affects the indexing rate? many thanks!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-51591829486583430332012-06-30T12:36:16.744-04:002012-06-30T12:36:16.744-04:00Hi Elisha,
This is in the upcoming Lucene 4.0 .. ...Hi Elisha,<br /><br />This is in the upcoming Lucene 4.0 .. the alpha release should be out any day now!Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-30322002864963885702012-06-28T10:14:37.380-04:002012-06-28T10:14:37.380-04:00That sounds great - in which lucene version was th...That sounds great - in which lucene version was this feature developed?Elishahttps://www.blogger.com/profile/14731416651965463861noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-42126097629739437372011-05-28T09:22:21.759-04:002011-05-28T09:22:21.759-04:00Wow! Amazing job on this one. I once had to inde...Wow! Amazing job on this one. I once had to index 6MM document and had a goal to make it happen in less than 10 minutes for 14GB of data. While running solr, I saw the same problem and it was the single thing that prevented me from having a single process hit my goal.<br /><br />I'm thrilled to check this out - thanks.Kristianhttps://www.blogger.com/profile/11346238503805824800noreply@blogger.com