tag:blogger.com,1999:blog-8623074010562846957.post2119311654280007305..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: Lucene and fadvise/madviseMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-8623074010562846957.post-25576222213781087112014-02-06T06:53:38.497-05:002014-02-06T06:53:38.497-05:00That's really quite strange. Maybe turn on In...That's really quite strange. Maybe turn on IndexWriter's infoStream (IndexWriterConfig.setInfoStream)? It could shed some light about why you have too much merging happening. Also try posting to java-user@lucene.apache.org with the details?Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-15621369191179609772014-02-06T04:16:15.356-05:002014-02-06T04:16:15.356-05:00Hello Michael
I am using lucene 4.X on CentOS 6.
...Hello Michael<br /><br />I am using lucene 4.X on CentOS 6.<br />I have an application which deals with tons of data (i receive half million per week, and keep them for 90 days). My usual index size is around (35 to 40G) I use NIODirectory<br />and all indexes are on SSDs. I use TieredMergePolicy with all default settings. ConcurrentMergeScheduler with default settings.<br /><br />I see very suspicious behaviour in my application regarding search and indexing.<br /><br />When i start application first time, with index(~20G) or without index, it works fine for around 10 days or so<br />- it returns million of results in search,<br />- indexing is good 40 document/sec.<br /><br />But suddenly after 10 days or so it gets slower (Searching and Indexing, Queries take 5x time more), I profiled application in that state and is see ~50% cpu is utilized mainly by merge threads (short lived), with no thread leaks. Now interesting part is when i undeploy application cpu usage drops to 20% there are no threads related to lucene. Now i again deploy application and try to search and searching/indexing is still slow. So undeploy didn't help.<br /><br />But after that, i restart server and application runs fine even with huge index. This behaviour remains same, after every 10 days or so i see this, situation.<br /><br />Is it anything in lucene which is bound to the application server process ? As only after restart situation gets better, undeploy also dosen't work for me.Jigarhttps://www.blogger.com/profile/13461575627688089524noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-83619617856624866102014-02-06T03:21:02.787-05:002014-02-06T03:21:02.787-05:00This comment has been removed by the author.Jigarhttps://www.blogger.com/profile/13461575627688089524noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-25784459190738987372013-05-30T12:40:07.402-04:002013-05-30T12:40:07.402-04:00Hi Matteo,
JNA is a good idea! Next time I need ...Hi Matteo,<br /><br />JNA is a good idea! Next time I need to play with native stuff I'll try it. Thanks.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-39140768008769625332013-05-29T08:17:00.045-04:002013-05-29T08:17:00.045-04:00Hi, what about jna to enable O_DIRECT flag? I wrot...Hi, what about jna to enable O_DIRECT flag? I wrote a couple of classes, available on my blog.Matteo Catenahttp://www.nicecode.eunoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-79926124778030930402012-10-08T12:51:49.457-04:002012-10-08T12:51:49.457-04:00Hi Mark,
That's very interesting! I would li...Hi Mark,<br /><br />That's very interesting! I would like to re-benchmark. But I can't tell whether anything was committed / which kernel releases have this improvement?Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-85094885863823278882012-10-07T17:35:27.834-04:002012-10-07T17:35:27.834-04:00Hi,
Since your post, things have been "evolv...Hi, <br />Since your post, things have been "evolving" in the Linux kernel regarding proper support of POSIX_FADV_NOREUSE for fadvise<br /><br />http://thread.gmane.org/gmane.linux.kernel.mm/73853<br /><br />What about retesting with a kernel where this patch is obviously included, while using NOREUSE (pun intended) ? Anonymoushttps://www.blogger.com/profile/18151911876893511453noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-75535900139492660702011-04-23T19:19:57.349-04:002011-04-23T19:19:57.349-04:00Hi Brian,
That patch looks great! So now Linux w...Hi Brian,<br /><br />That patch looks great! So now Linux will prefer to evict pages loaded due to SEQUENTIAL reading, and it looks like if the page is also accessed through non-SEQUENTIAL, it's not aggressively evicted. This does sound great for our use case....<br /><br />Though... really, it'd be better if this logic was under the NOREUSE flag instead, I think. Ie, Lucene should pass SEQUENTIAL when reading the postings, since we seek to one spot and then read potentially many bytes from there. Yet, really we want the OS to keep such pages in the buffer cache, ie for "hot" (frequently searched terms), caching would help us.<br /><br />This is also good timing: we have a potential GSoC project (http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/varunthacker1989/1) that will make it possible for native dir impls to customize the IO flags depending on whether the file is for merging, searching, etc. Maybe we can test this better SEQUENTIAL behavior with this project...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-25721066016006422542011-04-19T13:51:37.682-04:002011-04-19T13:51:37.682-04:00Very interesting to read about your experiments.
...Very interesting to read about your experiments.<br /><br />The MADV_SEQUENTIAL behavior has gotten smarter in recent kernels, I wonder if your mmap+madvise() based test results would improve.<br /><br />Prior to 2.6.29, MADV_SEQUENTIAL did nothing but increase the kernel's readahead window. In 2.6.29, the following patch was merged:<br /><a href="http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.29.y.git;a=commit;h=4917e5d0499b5ae7b26b56fccaefddf9aec9369c" rel="nofollow">http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.29.y.git;a=commit;h=4917e5d0499b5ae7b26b56fccaefddf9aec9369c</a><br />AFAICS this would implement the behavior you'd want for the lucene merger.Brian Bloniarznoreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-73338616415866600782010-07-14T08:57:54.944-04:002010-07-14T08:57:54.944-04:00It should be noted that CFQ will also reduce io pr...It should be noted that CFQ will also reduce io priority on processes with normal nice (CPU) priority, but ioprio_set does add more control.<br /><br />I believe several OSes has O/I priorities and this is one place where Linux is lagging the commercial OSes quite a bit, but a problem is that it is not standardized and rarely used by applications<br /><br />It is getting a while ago and my memory is fuzzy, it is not entirely I/O priorities, but GRIO - Guaranteed Rate I/O, was one of the big differentiators claimed by SGI on XFS already 15+ years ago. Since I never had much interest in multimedia type processing and the original GRIO stuff was not all that useful on busy multiuser setups, I did not pay all that much attention to this stuff back then.<br /><br />Being a bit lazy today, cannot find a clear answer if GRIO is supported on Linux XFS, so I have no idea there.<br /><br />It also seems like the io priority stuff introduced starting windows vista works reasonably well. Vista also seems relatively good at detecting streaming type I/O and not flushing caches/not swapping out processes to free memory for caching.<br /><br />I used to process a lot of data on my home computer around the time Vista was introduced, and it behaved way better than XP and a fair bit better than standard Linux distributions when churning through large amounts of data (given enough memory that Vista performs in the first place).<br /><br />Would be interesting to know how vista/7 handles lucene index optimizing.Unknownhttps://www.blogger.com/profile/07561956586014509575noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-3801144109284406712010-07-14T05:16:33.831-04:002010-07-14T05:16:33.831-04:00Terje: nice! I did not know ioprio_get/set could ...Terje: nice! I did not know ioprio_get/set could operate per thread.<br /><br />I haven't done any testing with IO prioritization, but I'd love to. Lucene could very much use this, to decrease priority of IO required for merging.<br /><br />This is nicely orthogonal to not polluting the buffer cache. The combination of the two should mean that merging has very little impact on ongoing searches, unlike today, where users have noted sometimes catastrophic performance loss due to ongoing merging.<br /><br />The problem is... IO prioritization is in its infancy. Really all OS's should support it well, but (I think?) it's only Linux, and even then only if you use the CFQ IO scheduler. OS/IO is still in the dark ages!!<br /><br />Note that we could emulate this in Lucene. It's hackish, but, a Directory implementation could track when open IndexInputs are being used for searching and forcefully pause, or rate limit to some low floor, any ongoing merge-only IO. Such an implementation could work well in practice...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-51935475674790631132010-07-13T12:14:10.501-04:002010-07-13T12:14:10.501-04:00Just thinking about something now... I read somew...Just thinking about something now... I read somewhere long time saying that ioprio_set() could be applied to threads and not just process ids like the man page says. I have only used it on whole processes myself, so I cannot confirm this.<br /><br />A search on google brings up a few postings which seems to confirm this as well.<br /><br />Assuming that the optimize routine in lucene is done in a separate thread (and it is possible to get the right ID from that thread in java), have you considered applying that?<br /><br />As I said, I have only used this on separate processes. It does tend to help, but not overly much so on the cache problem. Anyway, I guess search responsiveness is what matters here, so anything that can be added to reduce the effect on search helps?<br /><br />Be aware that you need a linux kernel I/O scheduler which support i/o priorities as well. This may be a problem in itself as I think it is still just the CFQ scheduler which supports this and it is not always the best choice....Unknownhttps://www.blogger.com/profile/07561956586014509575noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-69777403228058446972010-07-13T06:50:42.202-04:002010-07-13T06:50:42.202-04:00Terje: no, I would definitely NOT use O_DIRECT for...Terje: no, I would definitely NOT use O_DIRECT for searching (performance would be horrible). It should only be used for reading & writing during indexing.<br /><br />This is easy to do in Lucene: when you create your IndexWriter, use DirectIOLinuxDirectory; when you create your IndexReader/Searcher, use one of the other core Directory implementations (NIOFSDirectory, MMapDirectory).<br /><br />During searching we very much want to tap into the OS's buffer cache, we want the OS to do readahead for us, etc. (However, we also want the OS to never swap out the data structures we carefully loaded in RAM; this is why I always set <a href="http://kerneltrap.org/node/3000" rel="nofollow">swappiness</a> to 0 when I'm on Linux).<br /><br />Really we need an O_DIRECT-like option that still does write caching and read-ahead (but quickly drops pages from the buffer cache). Either that, or, the kernel should actually respect the NOEREUSE flag (it's a no-op now!).Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-16300894868554649212010-07-13T06:25:10.323-04:002010-07-13T06:25:10.323-04:00This mean that you use DirectIO for the reader as ...This mean that you use DirectIO for the reader as well (that is, queries)?<br /><br />If so, I would expect search performance to drop overall, especially if you got a lot of cache. You loose the ability the OS have to cache group and order I/O operations across multiple treads and processes and this tend to slow things down on newer machines (where memory bandwidth is rarely a bit problem vs. I/O unlike in old days when DIRECTIO was originally made)Unknownhttps://www.blogger.com/profile/07561956586014509575noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-91645957198198521622010-07-12T08:45:17.944-04:002010-07-12T08:45:17.944-04:00Jonathan: yes.
It's been committed to Lucene&...Jonathan: yes.<br /><br />It's been committed to Lucene's contrib/misc package, under org.apache.lucene.store, DirectIOLinuxDirectory. This is on both Lucene stable (to be 3.1) and trunk (to be 4.0).Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-27746769925440348142010-07-12T08:28:07.726-04:002010-07-12T08:28:07.726-04:00Is your O_DIRECT code posted anywhere?Is your O_DIRECT code posted anywhere?Jonathan Ellishttps://www.blogger.com/profile/11003648392946638242noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-87579722033690610472010-07-02T06:12:02.375-04:002010-07-02T06:12:02.375-04:00I did not try mmap for searching and normal file I...I did not try mmap for searching and normal file IO for merging... but I imagine that'll still evict hot searching pages. FADV_SEQUENTIAL looks like it's only about increasing the heuristic readahead that the kernel does, and not about quickly evicting the pages you've read.<br /><br />Really MADV/FADV_SEQUENTIAL need to somehow mark a given page for aggressive eviction only if that page was not already resident due to a non-sequential/no-reuse read. Or perhaps maintain two buffer caches: one "transient" one for sequential/noreuse-only reads (which'd in general be tiny), and a second one for "normal" (random-like) file/mmap io.<br /><br />Or... O_DIRECT is actually an OK match, except, I'd still want the OS to somehow do the readahead and write caching.<br /><br />I had already set swappiness to 0 for all of these tests, so OS would make no effort to swap "idle" process pages out.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-6809318970153308302010-07-02T03:19:00.936-04:002010-07-02T03:19:00.936-04:00did you try using mmap to map the index for search...did you try using mmap to map the index for searching and standard file io for merging? the index would then be cached with the rest of the process' address space and would not be evicted by sequentially scanning the file using fread and fadvise with FADV_SEQUENTIAL. you may need to play with the swappiness kernel parameter to tune performance.<br /><br />i think MADV_SEQUENTIAL actively pages out memory after you read further into the file whether that page was previously mapped or not, so that's why using it killed your performance.Jeff Plaisancehttps://www.blogger.com/profile/05937289313470162260noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-72075185420126414852010-06-16T05:17:05.565-04:002010-06-16T05:17:05.565-04:00No, I just used Lucene's default MergePolicy; ...No, I just used Lucene's default MergePolicy; BalancedSegmentMergePolicy wouldn't have changed anything since my test simply calls IndexWriter.optimize(). I wanted a merge that was large enough to force eviction of the IO cache; optimize did the trick (it burns through 23.1G (= 7.7G * 3) bytes).Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-57644830033757969972010-06-15T15:00:31.960-04:002010-06-15T15:00:31.960-04:00Hi Mike:
Was wondering if you did any testin...Hi Mike:<br /><br /> Was wondering if you did any testing with the MegePolicy we contributed.<br /><br />Thanks<br /><br />-JohnJohn Wanghttps://www.blogger.com/profile/00124403675406755385noreply@blogger.com