tag:blogger.com,1999:blog-8623074010562846957.post7285315674629168414..comments2023-09-01T03:38:08.236-04:00Comments on Changing Bits: Lucene's near-real-time segment index replicationMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-8623074010562846957.post-2395631083028608652018-04-24T18:39:14.905-04:002018-04-24T18:39:14.905-04:00Hi Eduardo,
Best bet is to send an email to Lucen...Hi Eduardo,<br /><br />Best bet is to send an email to Lucene's user list (java-user@lucene.apache.org).<br /><br />MikeMichael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-53297427275618724742018-04-13T16:18:11.538-04:002018-04-13T16:18:11.538-04:00Hi Michael,
I'm facing an issue when using Se...Hi Michael,<br /><br />I'm facing an issue when using SearchManager in Lucene and I'd like to know where I can get some help. Can you point me a good forum? Thanks.Eduardo Lopeshttps://www.blogger.com/profile/12959219721291711732noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-67219613850471223102017-09-17T18:41:56.541-04:002017-09-17T18:41:56.541-04:00Hi Mark,
Thanks! That's a great point, and i...Hi Mark,<br /><br />Thanks! That's a great point, and it applies to Lucene indexing and searching on a single box as well, i.e. the tension of working hard during indexing (merging) to make fewer segments so searching is faster.<br /><br />I haven't done any specific analysis of this wrt NRT replication, but I am curious! However, if you have many replicas, then the 1X indexing cost sees N*X benefits (N = number of replicas) which make it more compelling than the single indexing + searching machine.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-25846464291006564592017-09-13T11:43:44.406-04:002017-09-13T11:43:44.406-04:00Nice write-up, Mike. (and Hi, hope you're doin...Nice write-up, Mike. (and Hi, hope you're doing well!)<br />I worry that the search-efficiency objectives might conflict with replication-efficiency ones? One objective is about aiming for consolidation into a few big files and the other is minimising data transfer by replicating smaller files so they are fundamentally opposed?<br />Has there been any analysis of the overheads involved in repeated-replication of re-written segments?<br />Anonymoushttps://www.blogger.com/profile/05294769488164092601noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-67208747796467659452017-09-05T05:19:41.861-04:002017-09-05T05:19:41.861-04:00Hi Michael,
To offer durable writes with NRT repl...Hi Michael,<br /><br />To offer durable writes with NRT replication you really need to use your own transaction log, or e.g. consume from a Kinesis or Kafka stream which lets you push durability back into that channel. Another option is to indeed delay confirming the writes until a refresh + commit has happened, but that delay could be long, e.g. on the order of 30s or 60s or whatever your refresh + commit interval is. Plus, if you want to be durable to more than one node crashing you'll need to commit in 2 (or more) nodes.Michael McCandlesshttps://www.blogger.com/profile/04277432937861334672noreply@blogger.comtag:blogger.com,1999:blog-8623074010562846957.post-38932260211111691282017-09-04T20:40:11.872-04:002017-09-04T20:40:11.872-04:00Cool! Having just re-read the post you linked to a...Cool! Having just re-read the post you linked to about losing updates in ES, I'm wondering about how one would ensure that writes to a cluster are durable and consistent. In Lucene's NRT replication, is the primary aware of the replicas' states? EG if I want a durable cluster that doesn't lose any writes, the server should only acknowledges writes to its client once a quorum of replicas has received them, and I'm wondering if this system will provide any support for that?Michael Sokolovhttps://www.blogger.com/profile/04786063298116782267noreply@blogger.com