tag:blogger.com,1999:blog-8623074010562846957.post5484895994330610464..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: Lucene's RAM usage for searchingMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-8623074010562846957.post-61722260035255400302016-10-25T09:30:30.890-04:002016-10-25T09:30:30.890-04:00Hi Murali,
In fact, the default postings format (...Hi Murali,<br /><br />In fact, the default postings format (Lucene50PostingsFormat) takes two parameters (a min and a max int) saying how many terms should be written into each on-disk block. They default to 25 and 48, but if you increase them then you will see the same effect of increasing the terms index divisor from older Lucene releases.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-1666250786880950062016-10-25T07:17:34.203-04:002016-10-25T07:17:34.203-04:00Hi Michael, looks like terms index divisor is no ...Hi Michael, looks like terms index divisor is no longer supported. Is there some other way of controlling what is loaded into memory?Murali Krishna Phttps://www.blogger.com/profile/00397199918385471282noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-55234333793994988852014-02-13T08:10:19.098-05:002014-02-13T08:10:19.098-05:00Thanks Swami; it's spooky that it's gettin...Thanks Swami; it's spooky that it's getting harder in Windows to have it NOT swap out your process's RAM.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-42168894176359705822014-02-13T07:25:00.921-05:002014-02-13T07:25:00.921-05:00Hi Michael the method you provide for turning down...Hi Michael the method you provide for turning down the IO Caching of the OS for windows doesn't seem to exist starting Windows 2008 upwards. The current option on Windows 2008 upwards is only the ->System Properties -> Advanced -> Performance -> Advanced -> Adjust for Performance of "Programs/ Background Services" which actually would only control the processor scheduling.<br /><br />However Microsoft does seem to provide a Dynamic Cache Service available for download for Windows 2008 http://support.microsoft.com/kb/976618 & for Windows 2008 R2 it can be got only via a MSDN ticket.<br />Swamihttps://www.blogger.com/profile/07150203826019394002noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-80039633255539600772013-06-30T06:14:58.009-04:002013-06-30T06:14:58.009-04:00cgroups looks great! Thanks for sharing Stephen.cgroups looks great! Thanks for sharing Stephen.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-64451619507052848932013-06-29T19:45:08.958-04:002013-06-29T19:45:08.958-04:00There's actually a way to set swappiness per p...There's actually a way to set swappiness per process if you're on a new-ish Linux kernel (2.6.24 or above?). Check out the memory controller for cgroups: https://www.kernel.org/doc/Documentation/cgroups/memory.txt<br /><br />You can also set lots of other things as well, like RAM used for RSS + cache, etc., and there are other controllers for things like CPU priority.Anonymoushttps://www.blogger.com/profile/00322945868687057199noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-50019356274690725782013-05-18T06:13:26.442-04:002013-05-18T06:13:26.442-04:00Hi Ashish,
I mean heap memory, i.e. allocated Jav...Hi Ashish,<br /><br />I mean heap memory, i.e. allocated Java objects. But the terms index RAM usage in 4.x is now a tiny fraction of what it used to be...Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-54776264821241677642013-05-17T19:42:30.022-04:002013-05-17T19:42:30.022-04:00hi Mike, So when you say -"The terms dict ind...hi Mike, So when you say -"The terms dict index requires substantial RAM per indexed term". Does RAM indicate heap memory ? Or are you referring to non-heap memory?Ashishhttps://www.blogger.com/profile/07770165081855690389noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-174349067763381092013-04-28T08:10:21.676-04:002013-04-28T08:10:21.676-04:00Hi kbros,
You should stil worry about OS swappine...Hi kbros,<br /><br />You should stil worry about OS swappiness even when using MMapDir, if you care about search latency.<br /><br />Lucene loads certain structures into RAM (deleted docs, norms, terms index, field cache / doc values) and if the OS swaps those out it will cause latency spikes in your searching.<br /><br />I turn it off (set swappiness to 0) on every Linux box I touch... swapping is a poor abstraction.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-790443082172650692013-04-27T15:21:11.596-04:002013-04-27T15:21:11.596-04:00In case I use MMapDirectory, should I worry about ...In case I use MMapDirectory, should I worry about the O.S swappiness?kbroshttps://www.blogger.com/profile/15698030213270606806noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-47995715383021879642013-03-11T14:16:30.650-04:002013-03-11T14:16:30.650-04:00Hi Gili,
Maybe there is some way, but I don't...Hi Gili,<br /><br />Maybe there is some way, but I don't know about it! That sure would be nice.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-18001646154101865592013-03-11T11:51:33.689-04:002013-03-11T11:51:33.689-04:00FS Cache Vs Java heap - Would have been nice if th...FS Cache Vs Java heap - Would have been nice if the swappiness factor could have been provided per process. Though if you're running a dedicated Search server then this is less of a problem.Gili Nachumhttps://www.blogger.com/profile/16996461994293511014noreply@blogger.com