tag:blogger.com,1999:blog-86230740105628469572024-03-13T12:29:57.081-04:00Changing BitsMichael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.comBlogger129125tag:blogger.com,1999:blog-8623074010562846957.post-87432485705154859942021-03-19T16:59:00.002-04:002021-03-22T19:45:58.015-04:00 Open-source collaboration, or how we finally added merge-on-refresh to Apache Lucene<p>The <a href="https://en.wikipedia.org/wiki/Open-source-software_movement" target="_blank">open-source software movement</a> is clearly a powerful phenomenon.</p>
A diverse (in time, geography, interests, gender (<a href="https://www.wired.com/2017/06/diversity-open-source-even-worse-tech-overall/">hmm not really, not yet, hrmph</a>), race, skills, use-cases, age, corporate employer, motivation, IDEs (or, <a href="https://www.google.com/search?q=emacs&oq=emacs&aqs=chrome..69i57.505j0j1&sourceid=chrome&ie=UTF-8">Emacs</a> (with all of its recursive parens)), operating system, ...) group of passionate developers work together, using surprisingly primitive digital tooling and asynchronous communication channels, devoid of emotion and ripe for misinterpreting intents, to jointly produce something incredible, one tiny “progress not perfection” change at a time.<div><br /></div><div>With enough passion and enough time and enough developers, a strong community, the end result is in a league all its own versus the closed source alternatives. This, despite developers <a href="https://markmail.org/thread/3sqrctjhsjgdwc7h">coming</a> and <a href="https://markmail.org/thread/6z5x5ig33eyg3mai">going</a>, passionate <a href="https://en.wikipedia.org/wiki/Law_of_triviality">“bike shedding” battles</a> emerging and eventually fizzing out, major disruptions like <a href="https://markmail.org/thread/cmbek34vfycb6sfc">joining the development of two related projects</a>, and a decade later, <a href="https://markmail.org/thread/25p7tu5m3mbt322h">doing just the opposite</a>, or the <a href="https://www.theregister.com/2016/11/14/datastax_versus_asf_staxeit/">Apache board stepping in when one corporation has too much influence on the Project Management Committee (PMC)</a>.<br />
<br />
Many changes are simple: a developer notices a typo in javadoc, code comments or an exception message and pushes a fix immediately, not needing synchronous review. Others begin as a <a href="https://markmail.org/message/73acypgxzmrya35v">surprising spinoff while discussing how to fix a unit-test failure</a> over email and then iterate over time to something remarkable, such as <a href="http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html">Lucene’s now powerful randomized unit-testing infrastructure</a>. Some changes mix energy from one developer with strong engagement from others, such as the recent <a href="https://issues.apache.org/jira/browse/LUCENE-8982"></a><div class="separator" style="clear: both; text-align: center;"><a href="https://issues.apache.org/jira/browse/LUCENE-8982"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge2eNOmMn4b5N1OXV3jB5pMEjU82iEbhUrTvZYrQt_8k5vs6qxhPj08xQ3GEPllt3p_4zABs6pf9DZ3JV71hZzPVpbh87QKXl6L0D9qJdtG02YhDgvDiLsdJg6vxzspJZ2ScTlgrSWdFFQ/s512/bfeabvstni1kgrjsn1ef5eb6sk.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="512" data-original-width="512" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge2eNOmMn4b5N1OXV3jB5pMEjU82iEbhUrTvZYrQt_8k5vs6qxhPj08xQ3GEPllt3p_4zABs6pf9DZ3JV71hZzPVpbh87QKXl6L0D9qJdtG02YhDgvDiLsdJg6vxzspJZ2ScTlgrSWdFFQ/w200-h200/bfeabvstni1kgrjsn1ef5eb6sk.png" width="200" /></a></div><a href="https://issues.apache.org/jira/browse/LUCENE-8982" target="_blank">pure-Java re-implementation of our Direct IO Directory implementation</a> to reduce the impact of large backround merges to concurrent searching. Some problems are <a href="https://issues.apache.org/jira/browse/LUCENE-3418">discovered and fixed thanks to massive hurricanes</a>!</div><div><br /></div><div>Vital collaboration sometimes happens outside the main project sources, such as the <a href="https://github.com/mikemccand/luceneutil/issues/77">recent addition of “always on“ low-overhead Java Flight Recorder (JFR) profiling</a> and <a href="https://markmail.org/message/5zsmiy2e2vhf5rkc">flame charts</a> to Lucene’s <a href="http://blog.mikemccandless.com/2011/04/catching-slowdowns-in-lucene.html">long running nightly benchmarks</a>, now running on a <a href="http://blog.mikemccandless.com/2021/01/apache-lucene-performance-on-128-core.html">the very concurrent 64/128 core AMD Ryzen 3990X Threadripper CPU</a>. Some proposed changes are <a href="https://issues.apache.org/jira/browse/LUCENE-8947">carefully rejected for good reasons</a>. Still others, too many unfortunately, seem to <a href="https://issues.apache.org/jira/browse/LUCENE-602">quietly die on the vine for no apparent reason</a>.<br />
<br />
And then there are truly exotic examples, like the new <a href="https://issues.apache.org/jira/browse/LUCENE-8962"><code>merge-on-refresh</code> feature in Lucene 8.7.0</a>, rare even to me and my <a href="https://markmail.org/thread/bxxzyshgdzg4wosy">14+ years since joining the Apache Lucene developer community</a>. One long scroll through all the comments on that linked issue (<a href="https://issues.apache.org/jira/browse/LUCENE-8962">LUCENE-8962</a>) should give you a quick, rough, from-a-distance appreciation for the strange collaborative magic that produced this impactful new feature, including a big initial GitHub pull-request, many subsequent iterations, three attempts to commit the feature and two reverts due to unanticipated yet clear problems, the many random test failures, and finally one subtle, critical and nearly show-stopper bug and its clever solution.<br />
<br />
The full story of this change, and the quiet impact of this feature, is so fascinating that I feel compelled to explain it here and now. Not least because this impressive collaboration happened right under our noses, as a collaboration between employees of at least two very different companies, largely as asynchronous emails and pull requests flying across our screens, buried in the 100s of other passionate Lucene related emails at the time.<br />
<br />
It is hard to see this particular forrest from the trees. Let’s reconstruct!</div><br /><div><br /></div><h2 style="text-align: left;">Setting the stage</h2><div style="text-align: left;"><br /></div><div style="text-align: left;">To begin, we must first learn a bit about Lucene to understand the context of this new feature. A Lucene index consists of multiple <i>write-once segments.</i> New documents, indexed into in-memory thread-private segments, are periodically written to disk as small initial segments. Each segment is its own self contained miniature Lucene index, consisting itself of multiple on-disk files holding the diverse parts of a Lucene index (inverted index postings, doc values or “forward index”, <a href="https://www.elastic.co/blog/lucene-points-6.0">dimensional points</a>, stored fields, <a href="https://www.elastic.co/blog/lucenes-handling-of-deleted-documents">deleted documents</a>, etc.), read and written by <a href="https://www.elastic.co/blog/what-is-an-apache-lucene-codec">Lucene’s <code>Codec</code> abstraction</a>. Over time, too many segments inevitably sprout up like mushrooms, so Lucene periodically, nearly continuously, merges such segments into a <a href="http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">larger and larger logarithmic staircase of segments</a> in the background.</div><div>
<br />
At search time, each query must visit all live segments to find and rank its matching hits, either sequentially or, more often these days thanks to <a href="https://www.cpubenchmark.net/high_end_cpus.html">massively concurrent hardware the CPU creators</a> keep releasing, <a href="http://blog.mikemccandless.com/2019/10/concurrent-query-execution-in-apache.html">concurrently</a>. This concurrent search, where multiple threads search for matches for your query, keeps our (<a href="https://amazon.com" target="_blank">Amazon</a>'s customer-facing product search’s) long-pole query latencies nice and low so you get your search results quickly! Unfortunately, segments naturally add some search CPU, HEAP and GC cost: the more segments in your index, the more cost for the same query, all else being equal. This is why Lucene users with mostly static indices might consider <a href="https://blog.trifork.com/2011/11/21/simon-says-optimize-is-bad-for-you/">force-merging their whole index down to a single segment</a>.<br />
<br />
If you are continuously indexing a stream of documents and would like to <a href="http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html">search those recently indexed documents in near-real-time</a>, this segmented design is particularly brilliant: thank you <a href="https://en.wikipedia.org/wiki/Doug_Cutting">Doug Cutting</a>! In our case there is a relentless firehose of high-velocity catalog updates and we must make all of those updates searchable, quickly. The segmented design works well, providing an application-controlled compromise between indexing throughput, search performance, and the delay after indexing until documents become near-real-time searchable.<br />
<br />
The per-segment query-time cost breaks down into two parts: 1) a small fixed cost for each segment, such as initializing a <code>Scorer</code> for that query and segment, looking up terms in the segment’s term dictionary, allocating objects, cloning classes for IO, etc., and also 2) a variable cost in proportion to how many documents the query matches in the segment. At Amazon, where we have <a href="https://www.youtube.com/watch?v=EkkzSLstSAE">now migrated 100% of customer-facing product search queries onto Apache Lucene</a>, we have very high and peaky query rates, so the small fixed cost of even tiny segments can add up. We have already heavily invested in reducing the number of segments, including <a href="https://www.elastic.co/blog/lucenes-handling-of-deleted-documents">aggressively reclaiming deleted documents</a>, by <a href="http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">carefully tuning <code>TieredMergePolicy</code></a>.<br />
<br />
We happily accept higher indexing costs in exchange for lower search time costs because we use <a href="http://blog.mikemccandless.com/2017/09/lucenes-near-real-time-segment-index.html">Lucene’s efficient Segment Replication feature</a> to quickly propagate index updates across many replicas running on a great many AWS EC2 instances. With this design, each shard needs only a single indexer, regardless of how many replicas it has. This feature enables physical isolation of the processes and servers doing indexing from the replicas searching that index, and greatly lowers the total CPU cost of indexing relative to the CPU cost of searching. Heavy indexing events, like a long-running large merge or a sudden burst of documents to re-index, have near-zero impact on searching. This also gives us freedom to separately fine tune optimal <a href="https://aws.amazon.com/ec2/instance-types/">AWS EC2 instance types</a> to use for indexing versus searching, and yields a stream of incremental index snapshots (backups) stored in <a href="https://aws.amazon.com/s3/">AWS S3</a> that we can quickly roll back to if disaster strikes.</div><div><br />
<h2>An idea is born</h2><div><br /></div>
Necessity is the mother of invention! The idea for <code>merge-on-commit</code> came from <a href="https://www.linkedin.com/in/msfroh/">Michael Froh</a>, a long-time developer now working with me on Amazon's product search team. Michael, staring at our production metrics one day, noticed that each new index snapshot, incrementally replicated out to many replicas via <a href="https://aws.amazon.com/s3/">AWS S3</a>, contained quite a few minuscule segments. This is expected, because of Lucene <code>IndexWriter</code>’s highly concurrent “one indexing thread per segment” design: if you use eight concurrent indexing threads, for higher overall indexing throughput, each refresh will then write eight new segments. If you refresh frequently, e.g. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-refresh.html">Elasticsearch defaults to every second</a>, these new segments will usually be very small and very numerous.<br />
<br />
Lucene will typically merge away these small segments, after <code>commit</code> finishes, and after those segments were already replicated for searching. But Michael’s simple idea was to modify <code>IndexWriter</code> to instead quickly merge such tiny segments <i>during</i> its <code>commit</code> operation, such that after <code>commit</code> finishes, the commit point will reference already merged tiny segments, substantially reducing the segment count replicated for searching. <code>commit</code> is already a rather costly operation, so adding, say, up to five seconds (configurable via <code><a href="https://lucene.apache.org/core/8_8_0/core/org/apache/lucene/index/IndexWriterConfig.html#setMaxFullFlushMergeWaitMillis-long-">IndexWriterConfig</a></code>) for these tiny segments to merge, is an acceptable latency price to pay if it means those eight newly flushed segments are merged down to one, reducing our per-query segment fixed cost. So we <a href="https://issues.apache.org/jira/browse/LUCENE-8962">opened an issue (LUCENE-8962)</a> in Lucene’s Jira to get a discussion started and to explore the idea.<br />
<br />
Unfortunately, <code>IndexWriter's</code> concurrency is especially confusing: multiple complex classes, each with multiple shared concurrency constructs, make changes risky. We have a <a href="https://issues.apache.org/jira/browse/LUCENE-5006">long-standing issue to improve the situation</a>, but there has been little progress over the years (patches welcome!). After many pull request (PR) iterations, internal to Amazon, we settled on an implementation, reviewed and tested it carefully, pushed it to our world-wide production search fleet, and saw a substantial (~25%) reduction in average segment counts searched per-query, along with a big reduction in segment count variance, yay!:<br />
<br />
<img alt="LUCENE-8962-segment-counts.png" height="335" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxYAAAGLCAYAAABa2QQTAAAgAElEQVR4XuzdCbjt1fgH8Pfce263ORoU/kLSiIrQoBKFFBJJUlxDSMYMZSghZUjIWOYpUilUtwENRCkJZUjGRBmaU/fec8//+fx+vbdt22cP5+wzr/U857n3nL1/67fWu9611vf7DmsNDA8PD0cpRQJFAkUCRQJFAkUCRQJFAkUCRQJFAmOQwEAhFmOQXnm0SKBIoEigSKBIoEigSKBIoEigSKCSQCEWRRGKBIoEigSKBIoEigSKBIoEigSKBMYsgUIsxizCUkGRQJFAkUCRQJFAkUCRQJFAkUCRQCEWRQeKBIoEigSKBIoEigSKBIoEigSKBMYsgUIsxizCUkGRQJFAkUCRQJHA1JfAkUceGZdddtmyhr7jHe+Ihz3sYZPe8M997nNx+umnL2vHy172sth5550nvV2lAUUCRQK9S6AQi95lVp4oEigSmEAJXHnllXHNNdfE4x//+FhppZXil7/8ZVx77bWxww47xAorrBBz586dwNaUVxUJTF8JPO1pT4vvfOc7yzrw3e9+N57whCdMeode97rXxYc+9KFl7fjUpz4V+++//6S3qzSgSKBIoHcJFGLRu8zKE0UCRQITJIGhoaH47Gc/G9/+9rfj4x//eKy++urxmc98Ji688ML46Ec/Gve+971j3rx5E9Sa8pqxSMDJ5tddd13cdtttsXjx4qqq9ddfP5ZffvmxVFue7UEChVj0IKzy1SKBIoFRSaAQi1GJrTxUJFAkMBESuPnmm0OYxPe///34+te/HnfccUccd9xxccUVV8RXv/rVGBgYmIhmlHf0QQLIxGGHHRY//vGP429/+1ssXbo0TjvttNhoo436UHupohsJFGLRjZTKd4oEigTGIoFCLMYivfJskUCRwLhK4Gc/+1mcffbZ8fe//z2OOuqoKgzqzDPPjNtvvz3e8573jOu7S+X9lcBdd90Vz3jGM+Liiy8O/1955ZUrwrjpppv290WlthElUIhFUY4igSKB8ZZAIRbjLeFSf5HAOEpgyZIlccMNN8TVV18df/zjH+Mf//hHCB8SHrTWWmtVoSabb755zJ8/P+bMmRN//etfY+HChfHvf/+7Ckd50IMeFE95ylNitdVWW5arwJJ86623hvjrP/3pT/F///d/sdtuu1X5DIp3/utf/6qSQL0TSFxjjTVi2223rd6PDKhfEb+91VZbjVoCJ510Uvz617+OddZZJ174wheG33/3u9/FAx7wgHjBC14wqnrJ4Pe//33V9n/+85+VHMhG/sb973//eOhDH1r9u+qqqy6r33d8V77HX/7yl7jpppti0aJFMTg4GPe9733j4Q9/eKy33npVHY2FrMji8ssvr975n//8p5LV1ltvHbwxP/3pT6vPhQltt9121U+W733ve5V1P8uTnvSkeOADHxi/+MUvqvEWUmRc73e/+8VjH/vYKiyM/H/+859X79I+Y2b8Hve4x1Vj3Bg2ZtzI4De/+U2Vs4KsaQfA75lHPvKRlQ55R5ZvfetbFbnL8qxnPav6Prn89re/jTvvvLOS20Me8pBq3L2fbP/whz/El770pTj++OOrdyk+E0d/n/vcpyIXSEcW8vWMsabfPFXattxyy1XyfvCDH1zJmyy79VrdcsstVZ3yda6//vpKfuokE/qVda655ppVmxVzgZyuuuqqZc9pi3cKy9PPLbbYIlZZZZX/yfW58cYb489//nM1VnROPepdccUVq3F8xCMeUb23cUwuuuiiOO+885bJgT7w6OSY0hntJPeci8bcHCFX7zQG9NLYbbLJJtX3tVVpRSysEcZPO+nMve51r2oOPOYxj1m2bjTqtO/wOP3qV7+q5oK1gp4bG7povTE/c73IZz1n3UAsPWed0kbrxnvf+9745Cc/uew1JcdiVEtbeahIYEpIoBCLKTEMpRFFAqOTAAB26qmnVpbfSy+9tAJOQLC4dcBLgvPzn//82HjjjSvwgwy8+c1vjp/85CcBaAGc73vf+yqQk6AYAAA03va2t1UkweksH/zgBytwAgwAB0jH1772teqdANq6664bL3nJSyrwJbwFoFIAhje96U1dd077ASVtUBAJwAXw3HLLLePEE0+sABogD/gqAA0wqL/tSpIwbU95JTGSAA6Yew8wh2w9+tGPXgbetOtHP/pRdXKNMCzACknwbsAP8dpll13iUY96VAUwAU+gFNDzPmFcABWZA10LFiyovDD6R1YArhN6hAplMU7GJsvb3/72Sgbk/sMf/rAiOsZMm1/0ohdV/UcSvvGNb1TvAoCRCcD3la98ZUVmyMm7yEI/tA2BQVa0VZuNM33Yc889Y8cdd6zIZybI77vvvvHlL395WZuOPfbYCtzrxw9+8INqrBAFMvROwBbR4HV68pOfPOLw7L333lVom0K/ELGzzjqrkjmd0jbtBlbJG+h94hOfWAHlJAHtxt7zwDlSrU4gGqGjz+rcYIMNqvFGhNWJTJGF+UVOxj2fU5d3GsdtttmmAvnAMbk1tuX888+v3kcu5hPi5nN6Rnf32muvan6aO1l44d761rcu+/0Vr3hFNf/ozyWXXFIRIgTzm9/8ZjWGCCTySb7GELkwH+klfdhpp53i6U9/evUepZlYfPGLX6zqUR+dojPIAdkifcYPEcriu3RMjpPxIRvz1Zyki9aZZz7zmVUb6SWCo5CzOXPuuedW+mMszHHE7MUvfnHlhWxMKi/Eousls3yxSGDKSaAQiyk3JKVBRQLdS4ClEmiwuQNezQWQdJrS4YcfXoGEZsvxhhtuWIFcPwChAvwCHJKmAcqXvvSlsc8++1TAAThCJgDZtMLmO1lfgQXPAyBKr8TiDW94QwUwASilsU/AevPvvgMI7bHHHvHhD3+4reBYez//+c9XORuATSt5pfX7kEMOiSOOOKKqD7iUKC55HBlofs4zSBuAefTRR1dAmtyANGCPbAFA4CsLWQNb2pSy6kQsEAMgGClqbAMQDBgDuuTGC9L4ufYgFx/4wAcqYoJ4Ascvf/nLq7q0oblPADCg/K53vasiprwSSjOxAEDpBCLTWAdLtM8OPfTQCmz2QiwQU+Pk32Yd8HuOEXmw7neTvA/gG3ekq12djl5Vp74jjgi4o0+RP783yyk9RuYL2TYmonuOh2ak9/HoAO88cVmaiQXPAQ+COZfv5tlBBIzb+9///mqumosj6fOBBx4YH/nIR6pXNBOLpz71qZVOI3KNz/PYIdfILO+KkkTrne98ZzVHEdtWcwGZ1XdziO4ZL2SHrukr8pnJ+/TMXKBD/p6lEIu2S1n5sEhgSkugEIspPTylcUUC7SUgpMBRjZtttll1Hj3gKSRFWIFQBZZLFuWTTz55WfgNKyNQ6XOfCaMBnIVMKEjKa1/72sqSzTL87ne/u7ISA3CePeGEEyqQnZ4KxAX4POOMM+Kcc86prLMJOHolFqzTQl+AOEBDP4TlPPe5z60Iyyc+8YmqLRkKor3AHQIlpKNdAb4QJMBb3UJqgB9WU3LzN22/4IILYtdddw2ATAFGAV3WVmQAkQPQECnWaNZkQJ2sPOMzRENdPvvYxz5WkRNgjSWfrIAsYLuR4HQiFsbHuwE+Y8TyC9ADZ9qPWGy//fYVkfQ5UCtsidUYONUPn7Fq+z/wRt5JPpEzstcuIU/G+3nPe14FBj3XiliQOQLB80V++qrfxgS58B4AmnyA41e/+tVV2I4CdCKD+gVcIiDKwQcfXMk7yeUBBxxQvSM9Zp7Xb4DVCWFpFW839ghSEkPfQ4x5mLQRsePBSC/epz/96crTQ08AaHJCEpFw3oPdd9+98l6QkWdY9M3B5zznOf91J8QxxxxTzTFjRlfoH88Hr5vCc/GqV72qIv1ZmomFseE9QgDoPXKvzXTWnNZ/4VOAOvlog7Ey3sgQPUN4zGelmViok3fBd+ioNjM+pMfT+DFIKMC/372T/K0t3sUjYrwRMt408uQpQS68D8kmp6985SuVfphD5oo+IYdIkrllDLIUYlF2viKB6SuBQiym79iVlhcJVGBOWIKwBwDcZs8CKf/ARg98AZ5IAtAJgAgB8bnQBx4GzwFywnh8FzEBJgG7Zz/72RWxYLFWF2DE8u1ZdQEQSAorqlANBAXxyNIrscjnADlx7azMCBNLOcCqnYhQo5W3WzVAwoQsAab6DdgB04AfYMXzoc8+B9JZ+RWhXKecckoFwsnXJWP6i5Sx9PLo+AxABjqRFeAPWJJwjnzwVngPWfkOgoY06U+WTsQCSCMH4Na48+4kSFWH0BrECchHwrQL0UsdEG7iWV4UfQIOAX7jzgotrEe9wGGGrwF+ZJ13CjR7LBA+YUxCgcT2+xyY9U6AUqgVvVN87v9ApgIICxdqvqANOSMXQFYRQuVHW3yX/ma+ir93EwqFrAC1yLAi5AlB02d1kpewOOAa0AaUjTkPlFAj1nryRQQ8Y47QbfqJ2CAp2u3ZLHSDHiNMgL52I3tvectbqq9o92te85oqzDBLM7EQiua9xo1+pkfEOAmVMx8RCPNT6BoiCLQjOzwK5hA9T11uJhbG1hjSAbIx34VLZg6JdQOxUejKG9/4xioECjlFlrRX28wL6w2dI0thUNqjrwgqQiJszlql0CckE1kzP8jAkdJZCrHodlUr3ysSmHoSKMRi6o1JaVGRQNcSyORgFmqgiKUfCJIfkQBOZTb9jLMGqoFfoSYAMeu6kAoWRCDAd1//+tdXFn2WXbHyWb7whS9UzwLg3iOvQrgLcsISCaCyDmcZLbEAwlgyJY8D/azefmf1BWLbxeuPJDxtPuiggyrgK95b0V9AEUECeoUUAWKAUhbeEQAdmPN9YIoVF6AEooAtoNRzSA+LN+8Eb4X3IXpkA3STDdJCdmSeINO7OhELzxoL71HInvU4C+CMWLC2KwhMhhP5HSAEfAFlfQI86YJ4fyEvQCtACeghk4r4//3222/ZmDYTC0f/ApCAs0KHEJgsjXrXLbFAXBELoJjni5wBUCBb+8keqG1OlG83achGW5GqrBMZRzAyHwBwz5AvdQG35JIhP/qp/0iHOrQRCUMQ6I3vAvZZ6IZn/SBJ5ioi0Biyx5PAS5ClmVggeAhEJl/n98x1pBiZV/faa69d6QLy1fzdRrk0Ews6imDxHjEUWCN4YxR6hPjxhiq8Wd6ZhMk7hQuSGxLJ02EuZJ4NuWqTucTggJC7x0Shy9YRz3mveYJ4ZCnEop02l8+KBKa2BAqxmNrjU1pXJDCiBIAbAINlXBgSwGhzb4zlbwXwgFwWREALyAYoAVJAVUgNYCzMRjiQ/ABhKFkANGAo8wIACVb5tBoL6wCQs3RLLPQFwAC4FQAE8BZjzgIP+Pid9dTveUQpIAx4dnP7NtIFNKlHv4HqDNkCUlmuESrhSkBT9kn/AeRs20gDoh1CVMTzS3hmzWaZTVmxaAOVWa/EbKAxSydiIU6exTxLM7EQtsN7kGUkYiHUC5HSrlZx+Y39oxtCtzJXoJlYSNpuDEvrB7HgzZE3wKODjKTckTpeIuSKxwjB6Gbc9Ye3iycNiDc/1KnvnlcnSztrvUMBsk43QdPvHL+Rxp0Omjf0StiWYo4B/cic0DIkiUfA3xtLJ2KBqCMszQXxppdC6fQFsafT9K/dKVntjpvtRCwQZERM8nanuYBMIDnmAh1CKhxMkLkVCEkjqS43b5eNrkhg5kigEIuZM5alJ7NMArwNCAUikDcaA02twGKj5djnnmU1BOAAYhZXpACAZyUV6sHrIVSCRbKRWAjfSIAEeDWeXDRaYsGbAFgLQ2KBzSMtgZI8ypOXQYiH3/MoS4mlLOoAU6cCDAF3TroSHw8UC+nQlzwGNI+1FTcOZCrdEgt1IBSInrAoxAJgTVkhFo1W2elALOgGYqEvykQQC+FDQp1Yy1m8HTnLCp4kEuljDecd4MXoJhQKqeT94o1RJ6DPo5B1Cl0DmvVVnYBxt8SCXJAeYVOInzlIBxyv632SrNOT2Dw3ZyqxQM5474TCmcOIBGKRc6EQi06rVfm8SGD6SqAQi+k7dqXls1wCrKE8ApkrIWQBOEIEgKdMkiWmRmLhd+ANsGUVFuLAYi/mWSiIUBShLcJhhD403mMgREHYQt4r4P88FgAoayQLa4bReE+3HgvAUQiNtgD/Qp7EuCMMyIP8CMfqso4D7wpQKCzIMaHNcfrtVMO7JIry8Eh8ZfUVpw4AKupjFc97MshAKJR26SciwzLcKJdsD2ImZEYYCdKmnrwfgjUbWAVCyYqcewmF6pfHgtdGn1i988hPno5WdzEA7XIEhMtMFLHwHp4KZDPHSNw/susoWgRRGBOPDP3qJnlbnQC/kEG5JcZdMrk5ok6AV9gXL506ycLcQK7znhGeMp6exnCp1DNyksgsl4EeGVdhT8iMuhzBqs2S6RHaLKMlFjyV5oK+ZCgUfZNfkWFprebAWDwW5iaZI+aIkveYC8h4Y+hgzgWhlOaC8bFGMBykN5UBQt6L50oo1CzfyEr3Z5wECrGYcUNaOjRbJCDfgfeA5ZUlFDgUWuMkmUyiTVkAxiyxjaEjLKrCoVhVlbTUSrCUkIk0NF7Y5jvCbeRUyCkAxuRgOHcf+BduAjindRvwB9J4NXopgJz8kAQjwp4ATKEmfoCn0RSgRow9oMdCDfAArEKD8s4E9SJZ5JghKGSM1CAfSAFPD6CNXPjuSKEniBHwJDxNOA0LO+JFVsJJhJU55SqBGItuu3ss+kUshBohkf4FgoFhfZXA63SmxiNTm+U8Vo8F8C1EiL4CpvJmMtk434X0Ia48E/IFeBcQAdZvdx0gQ0idnBjgvZvjZnlA9BUIVieSgmBJTlYncA7kkoE6EVb5B/SXnJAZoVKS7+Wk0KGRCI2cCgBeLgKdA7DNMwSNTjTe68KLxdORwLw5x2KkUCjk2LzisUS6EVmhcQhj3knj3TxyxlOolDIWYmFe8siQmbAo46N9xgEp40UcaS7QdfM5L0ekf0iVfCNzUqhUHgWsnSXHYjQrXHmmSGBqSKAQi6kxDqUVRQI9S4BXghWcxQ9QkycBqABRvBEAURbWTPHv4smzAFYZy+5vCQwAEYCYlbX5CFfgTiImQAPgseIKHxE65O8AOCCopKUSQemlAH3ixVlGHVkJRAqL4VnQvzz+spc6fRcYIi+JwJJUhTrpKyAonIwHQQE8gTYyUMgZieMZUoSNscQ6YYoVujEUx/95LfyLoCEDgCoLr9AQwJys3BHiuFCgKmWFsDXG0zdfkNcvYoEUyplw/CeyBCSTKVlrI3LRCBCB3pHuseg1xwIho3d0R508AhLx6Vm+Q5gMciHcCekgT0SDdR545jnyXUQYUO/GYyG3gofCu/KWcl4Rc0SdrPHC7HymTqTHeBsn+UbmGCBN3+XN0IE8KS310POpTzwbvBPIN2KhHuTS/PD/DIli8We9B8zpTLfEQtvljNDNvJ2d3iL65q1QPHpNx9Sd3rexEAtypzcMEuYmvSAHfRAuiCQ0zgXjQrfokkMEHPcr30QhH0Tbc4wRyF05brbXFa18v0hgakqgEIupOS6lVUUCHSXA2guMCucAWlhZhUEBRciFEI+8wRq4AJTd5JsFmAI8gHUWWaAACACgeCbSMtvYEEAUqXDUKSsviz1A451IC+up/A0lj2aVEN1L4UEBRIBNQFR9AIlEbr87cWk0RdsBS5ZXoBAI1GeyIythMgrLtNuAM/xHOAtQBMT5HqKjvwCvPjaCcLJzWhALsXAu7UasHNUJeDbKSs4IEpWyMpaNty6PF7EA0nlRjAuQ73d9Mn7a2OwBQKBYlJWxeCyAc/2jt/ImyJ68gE/J2Jmrg+AgLMZIe3jZ6Kd20i//RzgAWh6EbhK46StA711ZJ3CP8KmTDmy++eZVnd7vO8YaOZRYbKy819/Jibya32u8kGxjjXwgw3SKPIFuzwHniB1i4P3mLIKEvCD23RILbVG3wwDMVToKyNMv3gt6qU+8NPrE86KMhVioz3voAvIlHIscvNNYNYdDCU80F3yHh86YIpKK7+u7tlqz6IM1K0vxWIxmhSvPFAlMDQkUYjE1xqG0okigZwkApZKQhR4BzYCyzVw4BCAk5ISVNnMHhC3Y2BMIIx2STIX6AFdCJ2z0wkGEbshlaAZPgBgQwMIvJAJIUp+wIJdzeR/QoahL2FDjSUbddJJVlzVWfoKwDmFKzrgXPuFCL4B+NAWRQq54VCS7+x3Y1X5EQX+BSx4FeRYszQq5CGvhrWHFzkv8gMTmE4MQCoCSJwN4TO+AZz0HTJKV+HiXpzmuVNEnoLsxTGa8iAVQqk/kqq3CwRANhKvxFKaUMbDMSj1WYuG9vA70A+Fioc4TmoTxAcgKCzYdoMN5uzu50W8glE6weLN2I5/tTkFqBKpCm7LOvEUbGFYn7xWAn94Tem9sebnICdnlgTBPyKnViVq8IsJ7yFC+ELKAmAhbQux5DHmDzFuhiTwK+o/AGwdgu1tioV+8ITwH5qEf4B3Z0DdgPvUZUebFGCuxMAbWDHOAR5GnBBnI43SbT7wyh4QZGjdzhcfRukImxsHfjR+y6vQoBotCLEazspVnigSmlgQKsZha41FaUyTQtQQAGDHLLkljSWTxBJAAY5s9YuGHBVVJgNEIxICctHgCOQA2ciJMYaQkUOAFIM6TdQAiAIFHgTVaeJV3CJNwVn2GYXTbMfH0gJbn5EIAMgiLE6q0qZvQl1bvYvFWLxBNbsBNEgtWV4AfGBIWlHdBZD15ShVgTGaeBRibiYV6eB4ASIXsyQooZeFl8U5ZAVN5jCvZ8WywlmcxLkJEspBH4/0dQJqk/Cw8T+7kyCJ2H2nIglTlZWf+BoDydiFNmRidpxc1ys+9IXmXiVj5DAnzHUC6MeclcxLyeaFdeaO2v5GjsTTG3p1ExtGkSUB9LkQMqUCY07oPiLL8u9eEnvt/twUAF/Zk3JGDPHwAsVAPcE82eTBA1osMkYmcH21CFM2tVsQCQHZRnmfkPSEkwDdij2gaOx4aBNPYI7bmnHfSGf3jqUOAsgi5S1LQqq/epT/eYz4iQvqmLvpsbPLeD8/LUTLeWZDZPLo576HJkCTriXbJCWos2kwOxone5IELzcRCn+UMpQeMHJALxgxjq43CEnlREG8/WYSnOYiilCKBIoHpJ4FCLKbfmJUWFwlMqgRYKYFgSZti0oUmAVqs2oAREAf8O0EGUGZdnq0FgCIrXiDgGYglK2QDCQP8hQQhg46mZdUupUigSKBIoEigSGC6SqAQi+k6cqXdRQKTJAHWd1ZGllwW3/QgsNLmJWBCe1jMWWkzpGiSmjupr0XCHGHKc9EoK94OsuIxkafCkux7o80fmdROlpcXCRQJFAkUCRQJ3C2BQiyKKhQJFAn0JAH3C0jKlPib+QpCXIQ8ZLy6Y0VZ31nom+976Oll0/zL4vHlwAj9cRyvUJyUlaNPhfQ4oYqshMTMZllN86EuzS8SKBIoEigScMfUcKtreotoigSKBIoERpAAYCz5WciTOHn/9zcWed4JMdtbbbVVdepLN3cMzGRB80iQj9hyydpJLMhKvL34dyFScjOaT9WZyXIpfSsSKBIoEigSmJkSKMRiZo5r6VWRQJFAkUCRQJFAkUCRQJFAkcCESqAQiwkVd3lZkUCRQJFAkUCRQJFAkUCRQJHAzJRAIRYzc1xLr4oEigSKBIoEigSKBIoEigSKBCZUAoVYTKi4y8uKBIoEigSKBIoEigSKBIoEigRmpgQKsZiZ41p6VSRQJFAkUCRQJFAkUCRQJFAkMKESKMRiQsVdXlYkUCRQJFAkUCRQJFAkUCRQJDAzJTDticWf//znuPjii2O99dbr6xnwTuEdGhqqbsX1U0qRQJFAewmYLwMDA2W+FEUpEuhCAuaLMnfu3C6+Xb5SJDC7JVDmy+SP/8orr1zdt9SpTHti4dKp008/vbrhd5VVVunU364/RyzuuOOO6hz+cr5812IrX5zFEjBfgKRyydssVoLS9a4l4OZ6RHz55Zfv+pnyxSKB2SqBO++8M+CyFVZYYbaKYFL7ffPNN8fg4GB1qWunMiOIxcKFC2PBggXVJVP9KhSYIIGkosj9kmqpZyZLwHxBxFdcccWZ3M3StyKBvkjglltuqYj4Siut1Jf6SiVFAjNZArfeemvVvX4akGeyvPrdtz/84Q9x2223FWIxVsGyKGFos/324LHKsTw/OyTAoiRssHj4Zsd4l16OTQJlvoxNfuXp2SWBu+66q+pw8YhPzrgXYtEnuS9durRyVfsppUigSKC9BMp8KRpSJNC9BMwXpeTwdS+z8s3ZK4EyXyZ37Aux6JP8lyxZUpK3+yTLmVTN3XggSk7/f4+q+YKEl2TUmaTtpS/jJYEyX8ZLsqXemSiBkrw9uaNaiEUf5F+St/sgxBlYxfBwxK9/HbFoUYQcpkIu7hnk22+/vQodLK7qGaj4U6RLSD0H8kQ4kc11P+M1xx12wFtRkreniHKVZkxpCQhNV0rO6+QMUyEWfZA7YnHjjTdWi35JRu2DQGdIFYsXR7z2tRHXXx/x5S+L95wYkDMdxGe+yK8oyajTYbSmVxsRCnMPoXcozODg+Lffu7zTWQTjQWQcdoBYlGTU8R/L8obpLwGHHSirrrrq9O/MNOxBIRZ9GLTiseiDEGdYFayX8sde//qIP/wh4kMfinjwgyOWW26GdXSU3SnHzY5ScOWxjhI46aSIo4+uPQiHHRaxww414B+vglCccELE974X8ZGPRKy8cv89F+W42fEavVLvTJRAOW52cke1EIs+yb/kWPRJkDOkGqDGiXcHHxxx5ZX1v9ttV4OOUiJKzPh/awEruzvQWNfHw+I9m3Tu2GMjDjkk4hGPiHjGMyKe8pSIzTYbPwnceGNNKE47LeKrX4144ANrT0k/S4kZ76c0S10zXQJlvkzuCBdi0Sf5583b5VSoPgl0mleDWPzzn7XF9LLLIvbdN2KffSLufeXGJ/QAACAASURBVO9p3rE+Nb+c2nGPIFm8//jHiGuuqcnneIXT9Gnopnw1iMV73xvxmtdE/OY3EY97XMR++/Xfi5CC+P3vI44/PuKssyLe/e6Ixz42Yo01+iumMl/6K89S28yWQJkvkzu+hVj0Qf5Cobjeyj0WfRDmDKmCBfq66yKOPDLiBz+IeMITaq/F2mvPkA6OsRtCO5wIVe6xiPjHPyI+97l7wOm6605MXsAYh3DKPo5YfPKTEaeeGvHKV0Y87GER73lPxHhdWv2Tn9Seih/9KGLvvSP22CPiAQ/or3jsL4xW5bCD/sq11DYzJVDusZjccS3Eog/yRyxuuummKnm7nELQB4FO0Sp4IboNU0EsWKHlVpx7bsRDHhLx8Y/3H3BMUVF1bFa5efseEQmh+eIXI666qg6nedCDSi5ORwVq8wXE4tOfrnMeXvrSiPXXjzj88P6HJ2UTFi6MOOOMiF/9KmKjjWoy499+FsmokrdXLrGU/RRrqWuGSqDcvD25A1uIRR/kj1i4vpz1tViU+iDQKVgFUnHccRHrrFPHbXcq4uWFYQA43/lOxLx5tQUVyOmWnHR6x3T+3Hzh4ZvNx2cKgfrTnyIAYeB0lVVqfdlgg/FNNp7OetNN28nzM5+JOP/8iFe8IoIHSM7Faqt183Tv3/nKVyIuuSTihhsi5FsccUTEox7Vez3tnnA8M2JRDFf9lWupbWZKwOEgSjmlc3LGtxCLPsl90aJFVWhHufCrTwKdYtUgFk97Wm2JPOqozqEqS5ZE/OIXdYjEt78dcdttdQy25+fOnWKdm4TmLF68uArtQC5mY6FPQOhnPxshlAbBkPALlArdKackjl4rGomF454ZA/x7n/v0h9QzGig5jyVu804yGPA4GVO5Mv00IMz2+TJ6bShPzkYJOByEwXcei14pEy6BQiz6JHJKXBK3+yTMKVoNYiGkSbx2p+MrEYtLL4345jcjzjkn4uab6/9vvHHtvZjtxXxROs2ZO++McCS5ZNiZRMjoh6TfXXaJeM5zIlZfPeKCCyJe/eqIRz6y/8m/s0nfGonFm99cH5jw8pfXnouxgn1q614aF+GtuWb976GH1kdL3/e+Ed7HM/nkJ/c3Wbzb+TKbxrn0tUhgJAmU+TK5ulGIRZ/kn8nbs9UC2ycxTulqEAtHSb7znTUQbFcAxx/+sA5xEZLBOv2lL9XW6PFKIp3SwmtqnOQ6oR2dLEpnnhnxrnfdc4xnTUamU09bt5V+uN9EWN0b3lCDVCE1z31uxNZb11b2UnqXAOD/0Y/WoVDnnVeDfmkJL3xhxEMfOjbdkTflQl+HMJjD/kVaXvWqiJVWqsOfXvCCiG99K+JJT+q97e2e4BFHwjvNl/6+tdRWJDA9JWC+KOVwkMkZv0Is+iB37FhynfyK2Rwz3gdRTukqEIv73a8+QpZ1sh3AFT8P2Hz/+xEXXxzx739HfOxj9dn6Jf+SF+KWCiR1ihk/8cTaii8R16k7yoIF/bUGT4bSsXD/7ncRz39+beVea636dvYnPjHi8Y+P+L//m4xWTf93mnef+ERN4h2a4PhXl1IibEj9WEgpMiiPQlgVDyQvCM/Ei14Ucf/7Rzz1qRG77jo+xEJOEiJeYsanv46WHoy/BOQkwWXlsIPxl3WrNxRi0Qe5F2LRByFOgyoQC3Hab3tbfXJPJ2IhpwIYvuKK+k4LuRlbbNE+ifTuCKExAaBpIMqeiQXPzzHHRLAaS6JnMR4LSJxsGckt/PWvIw44IOKgg2oPxde+VhNP1m63tJfSuwTI1Z0S3/hGfVLT+95X18EztOWWY9MZZNCBDB/4QMQvf1nXh7i85CW1N0RIG1Ioz6LfHgun3MjfK8Sid50oT8w+CRRiMbljXohFn+RfQqH6JMgpVE3z8bKIhdAHJ8xIwm4HbHliJW3/7Gf1MZTuKnjLWyIe/ej2YVR/+1udgyE0ZiaXbkOheCyEmgBrQlxYjT/84Vo+0znnwq3swClvBQs4T9jJJ9cE4+lPr4FqKb1L4KabIj7/+XruyWmiK+YioL/ttmPzdN1+ex3eKNSJJ1Jx4pvxsx48+9kR228/PsSihEL1rgvlidkrgRIKNbljX4hFn+RfkoX6JMgpVA0QizwkgEUsHAn6pjdFbLZZZ2Jx0km1VVosvYRPANmtvLwerQprPOsnb8jb3z426+oUEmPLpnQ7XxCLAw+M8O+nPhUhmdutyiz60zkJXs4N0ukgAHoh9Al5omsA6iabTPURnJrtQ+CFlDkw4etfr8OikLgddqgvqZRsPdriEAFkhaHgxz+uwxv9/sY31sTimc8cP2LR7XwZbd/Kc0UCM0kCZb5M7mgWYtEn+ZfjZvskyClQDU8FUuF0mXvdq07IBPgQC0eCvv71NUFo57EQNnHCCTWpcAM3YqGebbap8zOai3eKDwcqEQthU51OnhqLqLzPz1iA1lje3+3xmQiFC8fEzDvGU6iLhFzEbv78sbRgcp8VGufUMF4Y4VBuaj799Bqs7rNP3b9SepfAX/9ah5S57d6lg3TmX/+KeMxj6hyIsei7sRGGZ27//Of1qV5IDH1EBHmaxstj0e186V1i5YkigZkngXLc7OSOaSEWPco/ARlQmcCyXxfk9XKzc4/NLl/vQQJIBeC3//61JfmDH6xBLODg2gWhD86pbwdSgA8hGUKbABL/smh6Dog01s7DR1joUZ44I8nUO52E5IjV8cgj8G55H7/9bcTuu0/OLc/dXpCHWADeLo5zahLrs5yExz1u/G5S7kFVRv1VRJPVG/ilZ04bO/vsOoZfcrr4/VJ6l4D7JHgK3Q3iZCiE1NyTu7LHHmMjFhK3eZgYBq65JuKnP4345CdrI4DEcInb40UsygV5vetCeWL2SqBckDe5Y1+IRQ/yB8hcdGZTcYOyIwYBP8Tipptuqk6E6nTKTbvXsW6ziJejJnsYlHH4qiMlhakIRxLrLgHUSU6IBR0QurLzzu1BipAd1k0hL2KzeS122qlO7gQifQ7YIxGOrkUshFrsu2/E2mvXVtB+nLvfSjwIjdjzPDmHV2ai8xXMF6dCrWQStSmIhduTnaglIVcMvd/d/9Dh0XHQjP5VSR8uvLAOf3rxi+vQLnH7/ua0oa226t+7ZlNN5pR7JOSvfPzjdQidywfdZr733mMjFgjKa15Tk1rj56QyRgdzCXF5ylPGj1g4Rc2pUOWUm9mkzaWvo5WAww6UVcQulzLhEijEogeRA2Rc7M96Vh1bK6wFIEMs/vOf/1RAaaRzxgHHRi9H82t9zlr98IfXpw6Nh6W6h67O6q86SlLyp/hs4+HeCsnCTpYRrgTYCotq57FAHNzI6196c+21dfiU40R5LFimjbexpk/q4iWRY8FTIalX3PZYQjdGGkTJrNrGWg54IToI7UQW88UpN53OGdc+QBuAkzQrrGW//eoTeKbznvGXv9REwo8QORcvIhXWFcfr8myV0rsEEAohZY7y/dCHapJx9dU1SXeXxVjmkzF73vNq8m+NOOWU2rMo92fzzesE8fHyWDgcxD0WjjQvpUigSKC9BBwOopT5MjmaUohFD3LnrZAUuNdetbtdMmDeSTA0NFRZlFrdJCzs4a1vrc+sl0SoNBIHVnBgD3AVW+0Iw+mcmNqDSKfkVwF8FnInvghJcREWMmB8xPi/7GV1LkQ7kMLr4VjKTP4GSsRh83QAxN/9bk0eDj+8tqQ6a//vf6/DflZdNeJ1r6sv3BoLEBpJuPogfwSxYNUFilZbbWKHwnwxV8yZdiWJhRAUoULCUYS0IGDkNBUK4mgOC5PrtgjZsZbIs7Au8IwJjQJSJQNbW0rpXQLCk+gJIi/J3wWLV15Z3xNi3o7WM2d886Z0a7n1+gtfqNcG88ht6YwG40UslrI8hfVgDNnnvYuzPFEkMC0lUObL5A5bIRY9yB9BsFFJJhVb63KkPOFHspBFv3nhBzpsSMJg3vGO2mUu8Q85yeRc3xEuA6z6nIV8PBN3e+jyrPyqMAegn/XTEZVCn4S+yUcQriR0BRjsRCyAYfctCNkRjiHsSQgPYwpQwjrthKk996yBPfIhf8PYIxis1qMFQu0GjrUVoQCIEBvhWSOdVDVeCjDSfGl+H2IBELqU8IIL6nh55EwCvRCuyS7mLkIgjNGcRi668TayqLubQ+gOYrnhhjXJYFTgxer3PQiTLaeJer/LKIUoMQ7wJiDwl19ek3meoNHOJ+PM82H+mtd5n4q1QYgeA4R5NF7EAhFXePlKKRIoEmgvgTJfJldDCrHoQf5yK4TIHHFEHQMvWW+99epQKMlCwqCaQztYtoS97LZbbd1iLZaIavMTSw+cShYWiw9gIBZA1ERbkLsVw0xPMNc/IFG4gzBNlmMhFEKiEAuJ2H532267Pd44I5LuvZBDoU5j6mQaJ9cAJ8b8pS+tPSGZEOqODN4q9YvZHg8cAXSxjMuxQJL1i0dmIov5AiR1clUjFpKbyeWSS2rytfXW9Rwh18kuDAJycHgfXMgGwHYzZtYEawljBWK58cZ1Xg9iybBgbRnPYi3TTieQNZfpPMeFk/nhXUbQhK4ifuYUEt/N2LSSuxBIR0cbK+QPzjeHzFOeTadOMQSMF7EQOsjDJ4+vlCKBIoH2EhA6qJT5MjmaUohFD3J3ko6YXYBMXDRABnAiFjfeeGOlxM03owqJYfnmjWDdAji/+tUahABzNjzkA8AAWJ0uYkPkup+KpVOuSD/aPBHvGKmdxkLoBGAn1wG4d5oT4OBf44R08Ch0IhbAMMLgB7FgyWbx5LECIIU6IRUs8DwigAsyojhyVO7FaIFQu3EQcoXcOiqTXtI775/IYr4g4d0kbyNfPBS/+EXt+RFSdvTRU+MSQSRRLD8wywNkTneTr3LVVbXHyvogtGvTTWu9E8MvhIdejFdBHKxfPD5ybRpLhmVal6Zj1A2DjdvuGWsQCR4MY+N3xqDRzifjZMwQCXNUfQ5n4KXiFZFsL+duvIjFzTffXHnDSzLqeM2KUu9MkoDDDpRVp0q87EwSbhd9KcSiCyHlV9y6Cgz4l3UK0GQ9RSwcBwgoNXssWL1ZIoFRAM4G5SQY7nNx1QxQyDVLrJAPAINn4/7376FhE/BVgEPi7BveUG+mAHc3IR+9No0VX4w52bLut3rHSBZVVkSkZCz5KS7YAkaOPLIGiEIchKUYbwAQkAT+5EC0Ayks2Y5FddqPHxZiVlSExQVbQq2ETriJW94NTxXg7OhK1lGkgs5k3L4+6xuwN1a50zVH4bpnQ6gXkkPvJrKYL4ODg115LORTIOVCUcgReAfqJjp8q5V8kERtEdvPk7nFFt2FaBlrawBdddoYAwViKbxSojrCN9ZxbtXevKOFbiEW7tFIHQOWUzeBZ7o/XchF9ovn6LLLark6wcnaC/ibe8ZntMTC82kQcFocA8Txx9eHOPCKIBXIxXgRCx4+xKJYYCdylSrvmq4S4OFTxnJK53Tt+1RodyEWPYzCWWfVxEICqTAm4E/uhOICI6EdzTkWjsd0pjrS4PtIhM1PjD0wJ/nbHACYxAA7CYilrVWIQg9N7ftXgQ7WYsAH8Ge5GwuAb9VAwJk1XRIkIMlS3fwOwJ9lGOkQPpJF+2zwYtedGuS50QAzMe/GWVw28ODEJKABuUAsJN0iVrwRnYiFG6O10Q9QLHcDsZBzA7j5v7wLgARBBU5Zv3m19M/pU4Ad0PTnP9/jKRvrMbTe704INxML5dMWyacTWeRYCO3oFDMuFAqxMH8k5AohElKGFE2FY5lZyF3IRm/k3tATa0OnQr88J5xLmKS1gI4gm4iFkJvxAPV5X4q7MtJj4aAhc0WYn1Oq5A+dfHI910cLxDv1v1+f6w+DAqMHTzJS7oeh0vzhZZAXZ91wmEIvCfaNbbSOGzMheAw/8qQQC+u/3BhH0PJqWruMKz3op+y6nS/9kmupp0hgOktAjgWDL+NVKRMvgUIsepC5zVYolLh5lnVEQTiU4hQCQKn5VCjx7FzzLNw2ahvSRRfVpw6ldROQZCkE2J23LuzDv5NdGj0DyA9Qp8/6glj0O8EcOQAeeUOAAsSi8R1AhHAReQ+S57UlC88n4HDuuTX5A9hGQ3yMFTArfAmB5LVwUhersvchLv6V9NwKOOQFisZUbgDA7nnAk/yEQunj+99fW6Wte0KiABKeEtZvIGbHHWs9EEYH+AmlEh4D+PGejIY0pay0g7XcyWY8ZsccUxObiSwjzZfmNhgLgB0QBh4BRUVc+1Tw6mmfJGwhcsZFCFs3+SrAqFA0BgT6zqBAt+i2nI3UjX6PSd6XgqwhFohs3sdDR80dRJNuWK+m8r6sL/TB2sRgQ2bGQKgc0mmNYrDhIealcxLaaPuTN6ULweO19T7hhIwO1iTv5X1kEEA4hDf2Mx2inHLT75lQ6pvJEijzZXJHd8KIBfaotDqO1Wf5eYqjFUhvFFXjM52+m89deeWVsXDhwliwYEGsPorMT5ZymxTrlLhdFmzWPYXrDTtuvscC4HCaDfe5TQixkKshjtq9Bu5H4Gb3t7yQjediokNTmtXQps1i7xhUllMnCQkrsFmzxKfFs5/q6308AkA8MAn8IHFZfK4NLP3NxMJzgAOvhbhx4GI09xwArMC99wNaQtmQPBZkwAUBFKYlVKoVSDG+ZIckkRHAbpyBeZ4rlkyExR0ZwlEAFmEwGVLBk8C6yvrJMupdvCZ+R7bOP7/Wv7EUQAhBBiKB9c99rraUT2SRXMe71809FnTOPEPWhBDRRZ4CnpvxsOr3IgfjKKkc0DSXhd90k6+CRApHEwKXOTZAsP8L2UFezb1+l7Tu0yWeH0AZwUBUzR1tYnGXB4ZYTNVrE2wn5hrjAy8necmhMNfMJ5448wuhQP54gwD+0RKLvCkdmRA6SBfd7M04hBAKZxQ6Zr7yOPFkjGb9GWm8u50v/daXUl+RwHSUQLnHYnJHbUKIxQ033BA//elP4+yzz47Xvva1sS5E0FCOPvroOBXSaSi77rprHAxZtShu7T3zzDPjW9/6VgVODjjggNh88807JoKOlVg4ohPwBKyBWN4KIRpIjuQ6J9w0x/SJyQeCxM4DEdz0NjkJ2uJxhU2wtudRky7Kkhhsw5qsAnzIIwFw3CoLWAuTYM1nRbeZS/7sJuSjlz4AZzZpIB4oACgbw13kLQBxPnciS6PHAlBD/BAPF6p1G5KifRmf7V/AngUfsBKrzVPAMi68CrEAbBEf4KEZpGi/diE5vE+IEYBhnMXPi/UGHAFjbSVHHgnWauFVvBHAvpApxPItb6nzOVg+jYGxGC2xuJvXVwCSjBwe4FQiceNpne4WpNOPseZ6mC9IePNhB836AhTSAxfiIWzANz1AhhzR2k+rcC+6mt91pKnxZvXnifJ7N0YBXktAl6WbdyrzcHgveMMYLcajbwgvkGwNQyh4RxxSYDzpHt2nEwiGOdRvr+RoZNzqGfoM3CNI1k53sZg/1tj8oTNIPM+QeWvOjZZYmJPWROGDZEcHP/vZeu4ad8Sc59Gllzw+5v0obFcjikcyqrDBTocd9Eu+pZ4igeksgdtuu63CZeWwg8kZxQkhFt///vfja1/7Wpx33nlx4oknxmZiQxoKYvDDH/4w9oKu7i6+g1y0Kt/73vfixz/+cWi84rvbb799PMKO0qaMhVgANcJXxN8Dj0Ch2GhAe2BgOFwhj1g0H58pNt4zQp9Y0WzsyAavhM0HaAU0bVqshwCGEBpu9bGEu4xFnWyaPDMIDvCpLUK/ECRHYSIVgH83IR+9tMN7AXB1UwUW/0YOKkSInIB8XoVGYuFWZjfuAnms+zwO3eSpGFd9Y4nkXWDNRyCFtyArAAmviTFBaORBCHfgFQH8cowAHSDGUZTIAuLIgoqMAGhCNnBnIU9CN4TV0QGhW/QCeHaxF12Ry0K2LuICnHhNJCrr32iIhbbxlrBI6wevizyfvKAN8BKi1clCrh5jBBDLC/EzWh218PPwdUpGRSzoAUswUMg6zVINfFtGJvtYZh4KhJJXyfjypHWTr8KLydthiUM86bncF78bd8SiF1Df7SV9DBvWH1Z3smPwSAOBOUV/eQb1Iy9z7GUOT9R3c946IMG84S1CNIwDjwXvMMOPNQMBYJCgS6MlFtZxYyZfg+EiiYVx43UUwmbcrY3aZJ730/DisANGtJKMOlEaVt4znSXgsAOlk+FqOvdxKrd9QojFOeecE5/5zGfitNNOiwsuuCAeDak2EQteiK8yE3dR3vOe91TJ0o/l9w7W1zPicY97XDyHiWqciAXwIHzJ5gJsA2I2XpZp4QKLFi2qLErNyajAAuswaxlwCAAAsjYgANVG6PfzzqvJh98BKaf1jBa0pQhGex49iybrJXDMcin0BvnRD54aVjmW9G5CProYzmVfAaABeaQNkCQjJ2dlESJETrxEwG0jsciTedQBPLEaCodS2snBuApTAkJYIoFraohY8ViwhvobEkC9ABTEBREwRo4FNk7ekaREWJuxBdi1VZiTzySXsk7TAQRCkr7vIBHezRuGOPFosSbzWBgDREMbyGa0xIK8eFkQGeSLHgvn01ZjKuwlb5FvNWbp1XFBHd1HeACs0QI18wVQ6pRcBww6TQ2J5xXg3QMq/Q2x6Sd460VXfVc7kGCWadZqhEB7EdRO3h/WdXJ3dw09ut/9amLBG+bEMPrfbjwa20qfEFe6TzfTm9Rq/chwQ+sWYoFIMG4YX94K4ZjWqQznmQqXELYaFwYaJFMyNaOCYu2UtJ033NNT+AJJou/m9GjyrtTNhsWwYn7yRApftTYiNOSOSCAW5rvP/e5ghn4V+52Q307zpV/vK/UUCUxnCZgvSnNo+nTu03Rq+4QQCwIRBvWMZzxjRGLxz3/+Mz7/+c8vAxsAeqt8DHXJkXjwgx8cb37zm6vv7LvvvrHVVlvF62QVtylj8VgIBXI0ozASoSMSaQEbF4y123yBRBsaoA4oAAEsroADMMtarW5Ak2UNyARWuNbHQiyAHlY1VvVeN1PhSKecUid2CpWwSaqLlV2okLbzKCRw75fCIy9OXnEiFGDOM9EYViIcgZWdtZWHoZFYAEgIEZLnO+KfxTwDIMAFUNoq2Zq1UVgKYA8QSNY2Fk4d4nnwmTUKCdBnVn/jqC08CGRDvuQt1MlYCyVhbdYXYDNDqHhCxGJrh3qAIl4q4BjQY/EG9JEQ4BCxEP6EUOgDj8hoiIXxQbx43IAwoRsIEuIEkGoHr4qQmJEKAgZAiVlHKhELIGo8wnUa25DEQvsA+AzDIkdW4qaoyn6pYsd6AHF6gQjQUUYG/zf+2tUpNwHBA+TJ0DoCzAOv5hYvqPWlW2+McUFA5QTJj6CL9LjV2CAW1iSnmqmf5wcg1x+6hpDTRxc40u12OtFRSOP4BbInLzI0D62p5js9FSJHfsbB/Odd5NXghex1LcwukJl1xVxEasncyX7WAu82/sJXebAYILyb97mUIoEigSKB2SaBKUMshEhtuummsckmm8Szn/3seMxjHjNifJzPN9xww3g3NBnCB3aNbbfdNt4KxY0TsWA1tpGI8Rbf7fQXgIJ3gcVPshAy1GxREtoDkAOdig0csEQcbEBi54Fb1mpgVl0ALCA1WmIBWGivsBcnmNj0OllQG8UGECMWQAoQz/Jnw0YqgG5x4MCyuOZ+FgSLhwDIBmok7DaGlegTrwlQhDg0EgvEAJgCkqTm8CrxtADQQhNY7F0+2CxT4ATYRqKEOgEKALf6ATUeEuEU3glQsDSz0rNYGlfhbN5L5jwSwoyMue8hAyy/dAUgkfAtph7pyHsPWKeFWgmbInNWa/JXv3c6vUlIFk+BMRgLseBxM4YImPYgFUgkcpxhec3jqV90lnyQHrqJvwPEPEujvX9opPnS/H7EgucKUJNvA7D7QSiEDU3W6WkIjjHSNvOAR4U8yEZbO1mrzSNEj64DxAA8XaSDdBkwbjy4YKR5ZmwQYHomXAdJdDqacfLT7FECgpNMIhYICc8joM5wglCQM88FnZiqF3UiSOYaA4NijUIcyIKXk1fV/LfukYc5zuPA8zeaddWcFgZlDbIu8ZzxQqvfj/WaHjBAaAvPZD91k4ePEa1YYPu545S6ZqoEeCzkWHQ6HGSm9n+y+zUliIWciSuuuKIKJ7r22mur2xK32WabEXMsdt9999h4443jSEgtLOpPiu222y7ebjdsKs7/vuSSS6ocCLf9Xn/99bHHHnvEmmuuWS3SyAAlTFezvw0OzgvPLVmyOJyHTDl/85vBOOWUgfjb34bife/7T7zmNSvEeuvNjWc/ezjWW29xSK6TXyGmT52eV+eVVw7EqafOj8svnxO33lqffrXyysPVJrTvvnNio42WVnWefPJwnH76YNzvfnOq+OonP3lx9f7cTLQr26kOv/vRPn/3b/4NeLj88qF45jMH473vHYq99hqI5Zef8z/9HKnO888fjlNOGa6s6rVnZU5lKT7++KUV6N5hh+Hqb1ttNdCyzux7yg7Z0uZsf7ZT3/JvZHbzzfPi0ksHK1Cz1VaLY6+9hmLbbWuypq7f/naosuofddRy8bnPDcTuu9d9v+uuxbFgwQrx0IcOVmBiv/2G49OfvjO23XYgLrpoMPbZZzDOOGNJPPKRwtUsNvfI7mc/W1rVddZZg7HLLgPV2FxzzdL4wAf+E3/5y7w477zBuO66OXHwwUviwAPnxrnnDjjbrAIon/jEknj84xfHKqssjTlz5sWHPjQv7rhjOFZddUnccceimD9/Xuy882Csv/5A/PznQ/H2t8+NbbYZjnvdayCuvnogDjtscRx66Nyqrle+cjjOPXcwrr9+aUWGrrtuIA46aCguvXRu/PCHAxVZASLPOWdxbL/9nJgz579ll7qcss9FNWWHWL33vXPjYx8biptumlsRZP0ElPbaazh23nlOPOQh/z2e11wzL77whXnxspctjo9+dE6cf/6ceMELllb/c3DGdQAAIABJREFUf8lLlsZ++y2O1VZbskzvTL0cT+/102p+aavkbUcCmlupD37P5/3d8yefPLfyXG2zzVBFJtZccyBWWYUuD8XTnrYoHv7we+bCaPWuk+ya5xfZ3nHHkrjuuiVxyCHz47GPnVvNk3e/e2lsuOHSeMIThmPLLWsQONKcPf304Xjf++bGO94xFFtuObfyHvzud0sqXd1//6XxjGcMxFprdZ6zc+fOizPOmFclFV9/vXEcik98YjAOOmggDjhgqRFZdsdO3Z651dpwxBFzKlJ48MFLY9NNB+P225dUOR8f/ODcili86U0DseOOQ7HGGvXzI83Z0ciueR0QEte8Bnda7+64YyiuvHK5OPnkwcrDss8+SyqSccEFg7HBBnPihS8cji22WBxLl9LdwTjzzPmV13jFFeu/Na6hrfSueW36zW/mxcKFg/Gvf1lXF1XE4sQT58fixXNi/nxjvjge9rAlccQRy8d2282J7bdfGhts0D/ZIeK5FzSOx2hk17hX9DJnW+lyN7IbaZ/194mcs/3QuyK7eu+c6noHS2rnyiuv3FeMMlnr3XTTOzjeSakPl/DWoQwMN58J2+mJhs/bhUL5GsEB5xdeeGGccMIJsf7668e7HLHSosil2GCDDeLwww+vNgghVnIsDmHibSoWE8njcjgkwCEYvo9YJHihhH4AxjvvXC4uumhebLbZklhrrUXLiMVPf7pcnH32QNx225J461vviEMPXTHWWgsYHY6HP3zRslOhnNoBEHmvOq+4YiBOOWX5+MMf5sbf/+7I3eG4z32WxgMeMBAPe9ic2HLLpbHmmovjhBOG47zzlot11pkTu+02HLvt5vnaC5Lg6667FsXixYuWMXF/Jzfv8b68+dumd9FFQ/GiF82Ld71rSUUsVl11bvW9tHzld/NvNgjEyMQRWnDqqcMV6Nxii+HYc885lWw+8pGlVSjMox+tfQOxww5z/qfO3CzUm8SibufSigAsWXLXsvdYnPL9+nnjjcvFJZfMiyOPHI5HPGJx7LnnUOy4Y30k6eLFS+KXvxyqwtCOPXZ+fP7zNbEgk9tuWxTPf/5KseWWg5Wl9/nPH45DD/1PlXR+2WXz4qUvnRff+MbimD//rhgaGo4ddlgu5s+vZXf++UPxgQ8MxLXXzoutt66JxY03DsU73nFH3HDDvDjnnOXi6qvnxFvfujje+Ma5cfbZc2J4eKACX4jBdtstijXXHIq5c+fHIYfMizXWGI4nPnFxbLzxolhhhXnVe+64YyCuumpJvPa1c2OrrYZj7bW9T52L4uij51ahFLvvPlz1/Y47llaJ1tdcMxALFiyJf/xjMP7xD9+vvTnf+MaiSu6IYqPsUpdT75JY+LvN+/jjB+LII+fGUUchPXMr4nLddUNVIu9OOw3HTjvNiU02qYlFfVTf3Eofn/OceXHmmYvilFPmxhlnzIknP3koFi6cGy960VDsvbe+36N3pl62iR54t3FtpXdIPp3zvdQ7vzfrchKLRz96SWVZX3fdOXHve8+Jq68eij33vDO23LK2So2kdwl89Cnf06x3nWTXrMtke8sti+Lqq5fE0UcvH1tvPVh5Gr7+9aXx+98PVcTiaU+bU7Up+944HvTutNOWxnveMxgf/OCS2GKLwVhpJYR2Sey332AsWLA0nv50OtZ5zg4Ozo8Pf3heXHbZQNx+O6I2FD/4wbx429sG4mUvG4o5c+p1KNeRxYsH45JLllSkZtVVh+Ogg5bGFlsgeovjuOMG4thj51Z3uLz61QPx+Mcvifvcp9aHfsquWUcSpKQXK8dzJNn5+623LonLL58fZ501L9Zddzie+9wlVb7Sd74zGA996Nx4+cuHqzm4dOmiOP74wTj77OXjk58ciDXWYPn393vWu1Z6l4A7ZferXy0Xp5+OmA3FbrvdGTfdNBAnn7x83HrrnMqw8IQn3BWbbLIkjjlmxdhsM8RiKDbZpH+yayQWCZrIaTSya9wrepmzrfaKbmTXvM8iSJMxZ/uhd0V29bo+1fXOWGunU6Ga95+J3itmo94hFo7InnRiYYGz+RoEp0RRik8wobUo+++/fzzgAQ+IV73qVRU4PPDAAysPh9/blU45FizCQkS4sVm/gNQMIRJfK95ejC7+IuRAmIGQJqe6EGJaaRvbwMLPZS7cSZiLMCixt8Jh1CmxN5N4JQYKiRBqJEyq0WWvbUKFxE17rxhfuR2tkmeFugjhEcaknerq5ehDIVBCs7yfJdXz3iOkAOYUUiS8Km8d78QvtR3ZEcojlnykUAThPtotJhrRBda8R5HgLswB6ZHkLnFSKFTeJCwJVtgUb4/TrITqOCRM+I7fhSnpk3FgucxxzdAqORiOjhTWZIzFmksW5yExhjiusBGhSD4X6uIdGecvlETIk5OKWNgbT8wiMx4CYTPi6T0rX0LkHj3TJsm/QlTohxAPYVj0kNzlEwjREool5IW+CZPpJTlfjoV4ev0yDnTJmOij2HDvagxtE5suXIeMhZWRGVkI25EDkndLCD0bTTFfgN1OoR1CoeQcIYlyesTPmyNyFCSPj/VOj9G03TNyjsjBEuUMCfon/8qJceYF+bQr5GluCqvUL3qHyOsrPRHm1c3N4vSOHITdWRuQUmSRrtLP5gRw3xNaJ7eHx0I+h3G3ntBFeUCIhTDKycxh6TQuTlYT2mQ+ylMjfzK11lpfhaS58d58krtGh+k+cmqu91qsAUIVPSs3znqUYYXWNDlh1ixrpDlhX2g6o6TXV/7X95NYlNCOMYmxPDxLJFAbioU+jsNlQLNEhmPp5pQIhcoOcJ384Ac/qJK4H/SgB8URLlJoUY455phwXKVwKNbYiy++uDpu9pniEdqUTsQCmAJExcsCDDaQ1EubiqRcp4CISRcfbXPL21ZHugDQJg48CCUCjAFFQMKzNkUbEgIAzAMmyIINXRxv41GmwCDAKn5XSIhwITkfreJ4nfojHli8tBhuAB1Y6LYImwEqbcJOXrFpAy1SWrTPxm3j1PbmkjH52p7g3RwHuH784/qEFqSqVSI1QAQwAjjipJEEAIvs5E0IB3KELHkCEECvMZP0Lb4dsfA3gN0YIgpOfELc5AhI4CT3vDJFvUJIjGcm0IrfF1fuGfUCtk6UIUs/5KjvZCSu3v/JSb0IofExno1HhQJ/4t/1x6k9gBAgZ2zkaYitF86h/8Ce7yIWCAiQksd+4s2Spr3TKUKNFxg2joN+KY0yRiwAWTk3ZEaftEv+CH1CVsggi7rFrIvTd0oWUmYOuGcFIaGLPms8tatb/fK9dhdmNtaTxAL49WOMAW7yJ+duyW0vbevmu2QAxMuBQSzMV/qChMsPkjvT7sQsnjd6avyRJWTVuCOP5hu9p2vt8gHygAbGCeuAcUQ8FXU7crn5YAnEIi+RpGtOO0NqtR0AJ1d5XgjOVE5ARuwYBQB+8pPzZe2QJ2KNsF6YZ+YA/WWQQJYQ/9GcdJXH1RoT8x4xlxNnrWVEMF4MLk76YsRhDHDkbb9Kt/OlX+8r9RQJTGcJlPkyuaM3IcTCMbPHHXdcnHXWWfH0pz89XvGKV8TOaYoOlp+vVCFQXKQsMy7QkzPxRDt0daLNcVXexXNl5IWQkEvju9/9bvzkJz+pXNq77LJL7LDDDlV4VLvSiViwLEuqtXEAoo1HPgLxAGDe6AogIwY2YVYqLtK5c+f8z3GzeREWyz+SgajYyIUU+QwYBRaBNmDSZs8KKyk8QQWgCAA6EhTZAewBAu+3mTUXAFx9ALpEUGA7LcsJOttdcAacSD5GHCSdA00sdcgMKyDg7UjNZh4nXIclVFKroUNobOxAmNOlAFWnuPAktDrxJu/8AASAJKfseIf6gAIWYs+yTPJA6BdwzPrPQkmu/ob0UQVjRc7u3nDCEnmRaRIL441YSO4GULUVwAOWgXggjFUSgJH4TCe8X7twXtZ7ln4EBtCgnrxcxqYxWR4AdIoUlddGMgTunSDD6oqwaov3AS7qogvAPzkjK+QvwVsd9MP3JZizONOHJBHGV0gIYsBqnn9HLFixWbfJwA/gC6AhQYgFMJQFUTJexp6uI2VAMOssK7i+IkqjPRnMfOFObz6euVmXEQttoOdImb4jXICzMR3hqptxX1XziGjzBCFDBsjeWGkjYjHSiVk8TQwJPJ8IPH2jL/QfGQCSeTCAVkRwpPssjDHd5+HIuzCML28W0or8NSdfG29tNqbWGt48xIjOWeN4Aq1pebIRItnvov/mxFguWbQmWOMYDuigddH6Ku3OWmetAvi9g96aw+ato30Zh3ot1mpjhsRYs3kZ/W6tZVhIYpG3ezfPp17f1/x9BjSlHDc7VkmW52eDBMp8mdxRnhBicdJJJ1V3WGR53vOeV5GBLDwUEriFRayzzjqx0047VTdp3/vuY1GOPfbYWG211WI/O2UVhnB7nH/++XEuE1Wwru0bG220UcfLgzoRC8COFRI4BSQBpzyZhSXMZsLiZRMD3lj4WLhPO2047rzz9lh++TpeXLFxAm+sg7wfwJ8NHdiwuTn5R8x8hk2wCAOfACQviDCGJBYAhPAm70UugG7AqhWx8F6eD9Y0VjqgAzgHtH0GyHoPgAikAp6NtzL7v5AZt2yzCAtLInZEAFhhxQR0WI4b7jOsQBWAbFiBUgBJG1nvhDix5GmPZ5pDhcjL84AUMOsEF4AIsQGyyB0IV79NHNkB+I0TgM3S712Avu/b3BE4AOlf/6rJDGKiT9qexILXADgRIkGuyBtioR7t1ybeEeAWeQTOWc3z5u0MeQB6gUIExWfe1VzIAHCnP0ImvAeoowdC4JzO5G95Qo/jLYVEGXNy0G7EhjVWG/SZTIAmllGf5wV2SBG5pH7SI20HxIVrkY3vr7JKTczovdOqeHmyAG68XsgJokZvyM0zdFm7EPAMn6JbSrenj5nDSEU3F+QhFsgbwE5nyUA/hKeZo6M55Wesyy69os88nOSfJ5QZf/PX2LQ6McsYGRueDoQXsSBPfUCC6Y466IPljdzNoVZ9NEa+R3/NO2OB2APb5phjgZs9lZ7J45i1z/usR0LikHWkHnHjBTHX+3myEZnrP9npt/lgjo40frmGtvrc3NVW8wDZp4sMD9YBhIJuWOPoo3Giy4gpHe8mxKxZP9z5Ys03ttZs8mKUsNYiNuTllECeVOsfYnG3XWysqlY978IvhrdyQV5fxFkqmeESEP3Ca1EuyJucgZ4QYjE5XfvftyaxeOELF8Qaa6z+P1+wWQGfgIzNGljIC7hYwliQATAbBp4kvAcw+9rXhuPaa2+OddaZH2uvvcKy27W53HlAxC0D4qxmLJHAkY0VKLF5AwAswsIRfG6jBA5zQ7VRAd3AlM0YWEB8momFTRsBEXKEqHg3oA2o2PwQFEDMJsvKCrTgQQC1vqmXYQwAtWE61x7BQU6AG0RDG2z4NtG7eV4lx/okqrovQCsLnv8D0azvrKHaZFN32y+AnaEi2mWjBpK0GXEAYAE2gF/dxkObjIfwB6DC+JANT5IwJZs+L4kwE4ADOQGaAWzkCpETn5/EAmExjgA7KyQZa5PxZYFGuowxcoGcAF7GkTz93VgJSzJewtzIC8ARRtJchJQBG+QGAAJ1yIu4euOBdGovuQD+cjLoiLs11ImQAi30DRjTdu9D2HhN8vJF7xHGhRSlVTqJBQu5tiOv5OAZBaAFLn2WhZeJvhp/npK8GFI7EC/jQo+BtdQ7zxqfboC+wxQYEhx2MFJRb4ZCIVz6qY08UeYNPTJO3ZKZfq5DPAXIAS8dwpOeE5Z+Ok9uKd98rzFB6v1r7hkfupp3cRgHZJNsEUs6QTfItFVYFS8DQwiS7f3mOtDOa2E8taUx10c7rHHqpMvmGM+oNYXeMEbok2eMLXnzrvWzWF+0Wf3ehzS3Cou0xpAt7xjjTrNOZegWPTUnycxcskaZH9ZH66/nEDRziZytA52OAm7VX2PBm4NUmPM5/tbaNDZYY+lrznVrSr+KU9QQcafclFIkUCTQXgIO6UEsRLqUMvESmHXE4swzF8bznrcg1l579f/Z0ABs1mOAT4yujTo3IZYwm7uNEJgCfnkxPIM4HHjg0th774HYb7+BamMRK23TBhZ9zlIGRADdwJGNXbKuTd17AA1AitWadVaIS26mwAArJOsi8AywqLORWHgWQBeCwboGHACfNlxtBlgBEaCXxRI4BPTNO8nQNnttAuJZ+4B83gqgVtiL8AEeC1Z5m7nwDaA3C0so4I4QebeEYCEWAA8iIHmSN0I/WMC1yfcy94RXx2cAI3JB1plcjOgAIYgL+fmeMATEAnEQjqCtxgWxACSMIy8LGZI78IfIyU3I5G3kgwyQHhZfxMV7WeOFNCFdZGOckQ/eDaFMxg+YMy5ABmCBkPo8vSwjEQuhFICQMTQW+gjMIQ1kKvQJ6JfoL9TJeLKGIhvaSmfyHg1gXxidtgtZAdp4OVjC9RsZAkrJGLlCEBBjgAv44nFRr357L9llEeoDgAqlA9b0z/iRCcBJnvqO/NE78wXAFyLmO53IBYtSnlQ00rKnP2RjHmmrOZShgAAePaWbrYDpeC+lPFCAZhobMtcD2UaCtLkR1OfhC9pLbsAyUmuMc40BpM0fawJikbdF+7xVDqJ1BmA2d4FYc5VOWifoKSMCEtZY6BuPIGs7YkEHEWlzwd+QbgCcnhjnVqGWrWTb6PVsJ3vrhLlpzSM/60NzyBhSoZ0MPOaTdaax/z63tvFMeq95b92wvsm1MseFc6o3vcbWPHpifHi8ei3WFGOFmNB9xMK64rAJ77deIGFky5DiO41Evdf3NX/fYQc8FukRH2t95fkigZksAfNF6eQRn8kymMy+zSpi8ctfXhUnnHBWXHvtC2LffVevNvhGS6DNCiiXzAtM2Zgzfl3cPEDGQgr4S+YD3ITAsJyyUGbiJWAs5l98NFAAnLL0A7A2c+FAwDXwYAMHHL2X8RY4s/EhFjZTvyMvwB9QzxrKe6CuZmIBZAOawIY6gBeAESAD9FnkARebISs7IAF824TFEAMViAVLOKCRYVAAr3YDtkJj1KN+1tVGICrGWYJwAmekivXU+4SEscgC1IgA2bKomv/As42bMQ44Qmr0RRuArcyPcGIXi72/aTtiAagIkbLpIzLqRGSELfAokCkSCNAD3UATUuDvGXql3wAOedIHgIZFFTBBeMgBYCZXxM54eYZXgQ4BcT5DrAC8VnH/acUE9AAR7aAHgI+20yFkD0AFollfAWd/JxOE0xgBo0CN54Aq79YeMgfwfQcJ8n1eHONExvRF8j2PiXEBfgEhciB7Hiz9zsLyzWNBD4058EoPjTl5IyxIBRCMhJgfiABLcaNFeKQEZqe5AUryLEYqGTJkbvGmkStgp090GlEy3t0e/KF9eaHZWBddumlekx9PlHmsAPX6b51oTGwHho2b8eVNsK6Y09aQDFei19YdcxY4BcBZwH2XLjcXaw+CTf4MCHTRfLB+IQTAbnMoE0s/Aoxcm9OIDlmqK3MRECI6qj4GiU5FneYH3Qfw2xWeSUYUcmIcodvN+Va8kNrD42nMrWm+Yz7SeXLhMWRkIBt6WJ/MVxMlXtGcA9pCd+m49R3xMC69klGkWjhhnuzGM0xe5j0SgfDaF3Id8LfGCzw7ybDT593Ml051lM+LBGaLBOTeKu32l9kii8no56wiFldc8Zs47rjvxYknPjcOO+zelVWxMRIjj2lFCGx8NmXAiOXJhgQMJnAQz4scCIEBfFls0/rHagXYAswAmA0U8LIZ2iBtTkgC8JjAHyi3IWZcMWBg07XJA3QsxnIfAEKAwKbWSCw853sIjfcjH5kozPIIELPYA/5Avr6xYvNYAM9AcYbaCP1h/UM4gDf/txEDrsAD4M7i7V1ZgCagFggHVBAAVkeAAyEBRjNEitXQRkzOLKzAAyBlwwdUAQD1ARfebZx4KRADgNfnAJeN2/PaDmAhFsA6wA9EA5GIDKKXXgBARH/9HbEDOBAf4JzcjHceY8ryDkwIN+KtAWi8gz6wgBoT7QXCWNa1Wd9bgaskFsbTmJM/cgUwSrKXbIrsAYPGHsgE4oXLkE8eQysJnu4ZXwDVmJAlEAhYIQPGlazpJfIAbCJT9JWOAbIIKjKCLCNp+sVT1ziePBb0mpx5i8iQXvPg0Dc6xdpNjyWGSwzO40vJClEEsFqBuG6AEnLJKowM6y/ZpBeRTMwvXpZWoLtxMQVGeW2ML/nrd6/AshWoJ09rQuPRouY57x4C1Hg3kPfzcvAuIbnG33wV4pchU8gK+Wkf3aRfvIi+3ypizHfoh/chLMbDGMjvUD8ZIZKNBQkwV+iJ8TG2viecEFmm1/qk7fqlLe0K2dJtOq/v1sd2sjWn6RXdpMvmnf43Hp2MKJmzdA9psH6RHfJAB3k/rT3pMfO5OUUneTGsj4hdes2sPXlIArnkethLCJ08jVzHGQC0B0GjU+aAOWicrCXeRe/vPmukL/t6N/OlLy8qlRQJzAAJmC9Kp8NBZkBXp2QXZhWxuPTS38dHPvKjOPfc3eINb1itIgKNp6aIf7YpAHmsZSzsQk/kPwBdAB8wjBzYIFmEgWkWQsm9QK2NHHAFbgEfwB2hAHRZJ3kRWGB5MLwPoLNhAm6ANaDonXkKCaDKgmfDAvRZQwFeFjTWMWE4rNOssTZ4xIE10t8AMUDSJuc9vCKAv7hg4MH/gQuEAungqVCAOaCO9RmQBDZYjwFZ/fM7wqR/WYCiTHAkO94AfQKMABm/C9fJPBGkB2AEmICb9IIANNYEQAdgJQ/eAySFJ4NckQlA3nOAElkBG6zuNvO0TJIJIqFuwAVYJ2PjxnOAqIid1ncAB/hGLJA+gIolHEhHLACmTNT1XaCP9wCx8F7f831tBXKaSxKLHA+WYMRR/8kYaQSwACLP+z7Q7DNgFAgCZoSF0A+6SJa8MZlHY8zkngA89I58yRmANQbGO+/goA/G0zsRBaCUHicY8y4AEPAFnBBoZFL9wCyywgtGP8mFhRgJRJa8U+ggGdN5cmkOjZKM6oSbdueMG3N9AbbpqX6SiVAX5M7f6UCnsHP6RDZIMVCJJAKfvQDL5vGkn2RDT9SJFCryW4ypEJ7GewzS02ZOWnPE4xsjADlPKbIeaJvPkAaWdkDZWLcKFUYw6TJ9Aew9h1iog9eLF6PxbhLtowt0Kj2E5KhN5haQbJ3hEWLA6OYuhswboQeIMAPMSKdheT9vpXljrUFGzSWkMec/2ZGD9c2csmYgjtYkxAXR4OnhzSNzc0M/6ZzSKuGb0cK6jkyZt9Zfa/zd52x0tTGTjXWAXpOpNiJiDBzakMQi1wVzq1WuVVcva/EloYOsryUUarQSLM/NJgmUUKjJHe1ZRSx+9KO/xPvf/7P41a92jBe8YOUqjCJjkG1IiASgJvyEFc/vgCxQBlgC6IA7oAREAYNIhw3ks591A/hABRrEuLOs2eABORZpZEO9iIUNEgAVFsD6BZgCbkAKYACQs7JJtgWmWJWBPBttJjLyVticbaqAFuAC7AByNm6/C82yybOo25B5DlinM4lZQjrLNu+HDd9mC1gjDuRh80ZChHwA8CyhAJnNFBjgzcgCGAAl2glg2GBtvOSj3ayQ+gdIAOP6SAYIGdJmo2ZJB4gQN2FPiAVQIcRAO71D25GNJBbNx34iFiz3NntgEhEiU94EhIPF0rgA9t4hjt17gCsEkLWVHLzT2AkhYqmkK3QBYNAH4wP06Yu/AdbAGBDeDObIKIkFvUgyAoyl/OkGkEYWqRus0MbGGOclg96NLCFvxgBhRWjpILCvrUAUsMozlif+AOi8L4Al4AgMq5/u0FfyML4JChFFY0lH6CgSZq7IdTGeZIkEkrNx93d9RNz0iYUX2HVKUSsA103ytrHVbvPAnDO2xgGpRSjkf/AMtDq6uHFZNReS9GRitbEa6T6VbpZkukCPjJV+slYr5gQd8rcMj/J3xJyeIUb0CihGBIx3njxnDugrMI1YsNojfb7rO81EyBzKY2XplLlD53kDEFjja+waC5nKbaD3CBmyRZbAtzHjaaIfPDupp+3kQbbWSUTKGon4tsuX1GZz1zqE+FpnrYPGka7TV/lF5p25nrpD//JIZ+PGq2q9oPvmRLtcEEYg67q5Qa5Cq8jIWtZtoYfmOGOLd5kf6Zn2/jx5L40p+sHT2q9Skrf7JclSz2yQwC2sUmEtKsnbkzHes4pYXHjh3+Koo66KO+/cKnbeeaVqUwUCgWibhg2OJZ9lColgtQVmuewBYBs19zzrK6AAgNoMWWkdObvTTgMV2EMiEAwbWCbOIidAHaBsYwTIbE7ABcuvUB8WQhu//A1A2OaEHNjIeCG8y7MAgc3X/1nfbGzqZckG+gBMwEZd+sCaB+SwyCMkNj9AjcUQkEQk1Ke9fheWBfzon+eBJUCSjAAXFmtgDqjNYsMGaBERxAJAASCQHOAIMAZm83x34Vj+jxwBR0KcgG4kCkAiAx4GIFLfgC3yQlTIlAUTIAI2eG58H3lCxIwnAArwqMvaApjwdCAMcgaML+DG22FcWBx5hBAr4Vv0wP/JzHgiAULltJm8AUvW4oyvB87JXB8TZDZO6CQW5OY7foy3vrGkIoT6B2wBacaYN4UO5PGYQJy/5SWFPFk8HH6EbyFU2s6i6xl9AnwBVh4dnimWabKkf6zVQCRPBdCrj4gzMEf25EomCXSFVdFjMgHW/M7rYoyNJ5mQpTWdviA/SV6bPRaOm+WxaGeBzZvPtUmRA5IHCRhfQFYfO90qT055lCv5azcSTr/zcrpeF19AnNy0B5nMhGD9RXzpYsNVPZXX0RxDAP3fnLee0OEkRuaaOckraT7xBAGy5hXip/+NYUbmKSBP98xvZNL8JC9EJEOEGvtmviDEnqVj2g6cmx+IJC+JvvBkdXPJm3XQGmkcAGlEyTwe6VZ46wKvF2JDFmRgraNH5oE5YS4UQp/XAAAgAElEQVRb16yN+sPzaB0wX63X1ijeUcU7ybDdRY3mLRkiE3mZJk9RL8fOMqYgydYCa4j5YU5Zb70/iQUvEgJj/FsdO92rnuX3efh4LEoy6mglWJ6bTRLg4VPK8cyTM+qzilhccskf42Mf+0lEPDk23HDVClwCbnTQBgWIs6CxlgHzNjxgD1C26QFjrKSKjRNgZsli/QSseB7ytBfgDxBgLVef51l8WR2BOxsxoAH42kD9n9XcO1n7gTtA0QYL8NnYbI7ALiAFPAODgIHnWDXlPgDzLHk2+DzSFiixYdsUARbWdnVLFEUwgAzgE0ECNngvWPMAfiBJH5EVQBFgRLSELPg3C1KDoABcNlSAXVtYR23IwJM+AwYAC4AhhAMQBp7U752Ak+8gQMA44AcY2Kj9y9JJ9kkstIcnxobuO3lpn3ECsL1XHYAIYE7WADVSCcDoO9DrOR4a7wemEUNj5X2IlNAhbQTWgCBtYeHXFz9C5HgIyDmPD21FLJBBYAp50F4gCVkwNoATmaufJ6HVJWLGQxu1GYDPey6EoGiv54HotDjrq9/pX17CCKR5P1DM+wHgCY3RR1ZrbfQM/QGS6BsgblwQC7JHhIE57zfGGTbIC0Tf6DhPi3ChPJ2rUR7dXJBn/Hm9EDj9zsMRtIcukneGFrVbPhFKwBngRmDJDwkCsBHD9Bj0sgSrz5qBXCAQCVIRROOCzBvHLNYBcwMBNMd9LpTNsxnKhVhoI1BPfmTpd/PH2NDPJCF0AzhHCoQbWsuMGyCO8DcnlWc79J+umRv5He3gMUXa/Z1OmIv0utHr0ko+1k5tRfSsd9Yd7TD36XfzcbX0jL5Y76y11l9toov00/v0C1i37tI78878tCbk2ks3rM3k7915AWirNiK7vGoMCuYbfbdG9XL5H4+cdQqptt7Td3OFLPOySPOBXiGX+q5t/Sou/HLYQYkZ75dESz0zWQLlgrzJHd1ZRSyuvPKqOPHEhXHXXS+MFVZYvdpAhUYAFixYrHk2ehZYG5UNDwgAwmxwLNXATBZgzebCsgqMA60IBRc+SzRQh6wAY0iBOgFCmw9gBIyzMqY13sYM4APdNingnEUS4EcseBpY71gEAUeAxLsBZ5um74tN9h4gzDtt0vqIgORdHEAEYJxgFODRd5uh9muDdgnZUoA6BakhEx4MxIcHIwvrqnf7nIx4LnhYgCLvZQX1dzIHWJAnYRo8QE5FAvpt+gAY8A7kIA3ImroASAX4AGRs9OSi/7w5rKYZpuR7CIvwB/L2Pt/RfqFRCA3PBnkZK+OvLmEovpOn/XhWH4RIsVIqvBJ5rCdAlsdy8vawNgOqzbcdey49FjxBALf36b8+6x/LJ/mRLT3L+1PaLQ9AnXEH5IXYANz0hRUZiTXugCcQh/QBdcArUIqseta4ICOs1fRQf+kV0moMhJyQAw8FcItoCQEx3p4lR2Dad+iUdyGsxpSOshLT1+YwHqd2AEp+RioZbsJbiBzoG5nzQNElnhqy6mR5NgeNs1AwwJncjBH9Nxa9AMxsqzkE0AL+ZJnkBNC0ZtAzssmS5MY6Yy0hS8RUmGLerE1HeB+QBzqOvBgbHlRjLL4/L9NjGKDfZGAdotMIk74BwPQsE58b5Wt+8hJar7zX/CJL46k+axLdMQ/MG0S5XTFfkUzjAmAjW/TLc/piTcux93dW/pzvZCA80vggXDxq2sNIY/1C8P2fPM0L+ssooo90wBoiH8pa2k4HfN86iJzQSaTFvOjmxKvsO/KG4Go/vU+ybg10GIX5jHDkKXbGwzzoV3Emv9JuvvTrXaWeIoHpLoEyXyZ3BGcZsbgyTj55YaywwoK4+ebVK5ABVAOSNluWLZspyxiADQzY9BAGGzXrbOMRgjZiG42N9frrl8a66w7EaqsNVMDcJg90qhNgtPnnufaABZDEQm1zlxQLILOyA3TeD1z71zv9H+AD7ljvtM2GjPAAdYCkPrDKA8uZJO3/rPnaD/wAzqyhrPuAhQ0ZrrNxs3TaMAEu7xCOAXg1lrzZ2WYOLABRWQA3JMt3gB2eAGAeGMokV0QF+CVjn7OAs+ohcurTFuCKjIFX7WDVlGzKyohcADIIG7BFNuSGbCAhiEVe6A68+DvrMcADVALMPFHkA7iQbwJkoEMysHHz3TwmVdtYtnkQ8nJDckeEeD1Yio0PcOZ7fm+VTJzEQh+1ERDjZULQtEF4CGCFEOlfJvS2Wx6AS2SGPiCO8keAWcDU/7U5T9TiFeGhIT/6qc30jSzIiGcCiJMP5N2IGiCHSCCdnqHPfkc4gDr6DFwbX/8HFFmogWygSt94coC+ZmIhuY711SV5IxVzwfwy1nnxG6BKvkgRPUHyzJ92hY7ov3bro/liDLXT88BsrwXApZsS4pGTJAeImHlKro3H95IxMmI+pHcN8GYUyDwCOmL86Rnjg3WF1d6YMDQgJWRMh/LYVl4M8yPzeoBogNw89N3GcCx9NE+QfWuQNltHeBmsRaz61kNtF27Yze3R1kj9yfEwr5KU6CsZ5Kld9JWBBImmm96j/2Qj6Z1nUIgYw4q1wtjzWNEDzxg/nhDhTNrNuGBN84524XCMMcbdGpiX6Zm7rQ5ZaKUHCFEe18xYwyNpjK3rSBz50UWEjHzNK3Ujhf0qd911V0Uq2h120K93lXqKBKa7BBYtWlRdkFcOO5ickZx1xOKb31wY973vgrj66tUrKxYLK0uUTQqQBTKABkDIRugHkLEBskrZnLIA8gAwi/PixUOx+uoDseqqcyqLnc2MxThzJoSP5Pn+NkobJ+APAADzCAjrrvbYvAC2DK9SD2KifTYuYQos5zYzbfcOBAPRET6hrZ7n4QBYgUxACkhhIc1TpwB1bQUitAOoFkblX5t+q1NN9It10TOAdAJwQDOPuc1brhELIQesjYAsECz0CrEQyw1Yewc5Ah/qFqoFpCAFfmcNBMC8E0HIkBj/B+7JiHUY2AJ886IyJEfIBUAsb4GcyBmxMJ42f/IC9lgu88hN9ZB3GtIRMyTFMwC38UcMyE3bgAyyAqSREMRipMvMADV9AmaBESDcdwFIsgIIHWFqXJCFToUshOMBd/pAdwEmllMyUbcxB/zIGJk0tkAkWQH9SXaRxMypoKdJLIA71nPeCLIFDAFQQB1hQE5Z3p2ABGDnEc3k4V1kT0bNxEIyKlKxYiLyFp01F8ifnhlHY0Zf6ArSi4DR61ahZ43V8dJkuBxAy+oPgJtn5rSxNy55epW5YHxHOjaV7gs7pAsApzYkP7KOGE8ybLxA0vxGIHnvyJ+eMGaYbzmHyNjn3qvv5E7H1Ik86TPdIcvMaTI+dCZDjqxXiJh/hVtpW2NBSBwUQCZET0+EWzIMWA/z9DaGBjrSuN610kekQL947NKja3wyPBDoR3S1Bwg3L+mtz80p/fIubdIPRhBzEslAqPTV54iT9cwcsd4h0EgV8oLUt7nAvVrbEFNrFL3l7WCEaCZd+tfqoj9tFwJlvHj2tJnHAtHRBmsTPbIOmIvGDKnTtn4VNwnLsWh3U32/3lXqKRKY7hK4DVgKa0y5qX4yxnLWEYvTTlsYm2yyIC68cPVqY7DBAkZ5pCqLmU3jbr2sNhqgwAZr8xBekIXVHZhkuVpttaFYbrmBWGmlORWAtRkCFzZBIIArHahR1C8ECLBhQbbReZ+cBRZDbQGiEA9eBAAOmGPZFB4ifIa1HoAHRHgcxFUDhkgLsOBzQNIGDWgCJKzkgKAQC+1m5QaogFhgAHgDNFlym0M5GpUTKLbJs4YDDp4HXIFj78rbtlkbgWShLMC4ZFklT4UCInyO6KgD2FOv8WCV1E/9R054SbTfJm8zJzPPAWkZBtQYuuHvABZ5Or3L8/qGNNj8WTqNG1kA/IhagpPG6BzfYYWmBwAs8JyAxBgCSdop3M2P8KZWx5imx0If9Tm9BQBpWj8RS6EVxq6b9RCx4snKo2CRCjqBZCCXdEu/eeaAKuNrbPUJCfE+wFZ/gXd/F9oDIAGY9JPXgqcDoTWuLMb0EtFFtIAoP0C+dqf3AGlRgFaW3GaQDihJ3m6XXGfsvIcnR3vohXeZB4CssaBXiEK7Yr7lRXJCdIwRi73jkpFzIBDJpF88evSKNyt1uzlai/7ksaW8W+ZRjrk5AGgL/0Lq0hPH65M6ad0A2pOw5rPmsTXAnNQ+P9YchN24mbs8AmRpTIBkumOM83Q788OcB/jpQYYQJmBGLKwV2mPOI4K8htrEU2B9MYcBcW1MD+BI8rU26hciaTwQP0TCXKTjxossrG8MBoC5dZBsrZH6xYNEjtYH7+atoYtkQRe1PY9bNm+tqd5n/fEMYtHpksS8y4Rn1/OICn1tLGTH+0NudCTH3fcRaiTSfDNnyMnc8H79pEf00HpgDtIpc6nTDfTdbvqAEmLRjoh3W1f5XpHATJeAw0GUQsQnZ6RnHbH49rcXxjbbLIiTTlq92rABWKE7wBdLJMsf4AAI2qxsvoAbMG2Tb7z0yjPAs/CQ9ddfGnfeORDLLz9QWcVtRoCqjRdgsHHmDbs2JVZ4mywrJaBkQ/KdPJ1FeInN3mcsmv4PGNt083ZoQBJo11Zud8QCYLBpAoF5opUND/DzYyP0DtZA4NJ7hVnY/AFG3wXmWTtb3SBNTVligSLfY4H0PGsnr4wNlWUbQQCyeEh4TVgztZElGOhIzwpAD/DKKSB39bD0+Q7QAeSRO28HMAKYA4k8FQAcwALoyytRF0uwQsasoMYIwQDIgG1jA7DmxX1IH9IldKGV8TyTTQE1eRaAFu+LAqTQDe3WDu/LXIzm6ZzEQj36QxbG01gCe4gjEC4kCeBpdxdA1k1fgUHgWz28OOoQ7qFuciU31mgeIm0kJyTXdxAu4w788O7QH4AN4aM7gCvCaEwQC2OIWLDWA1jIILlqL2u69+fpfkiX9gF/5N5MLLiqASXkYqSSc8H8MmYANKDqER4/4wDMAqPtCoBO38mUjtB7wFm76SqCRB+AR3OH3gKFCD5g29x2RIVMhOd4rjGUCiBGOpA+bWOo4GH0PmPgGbqg7Xk0bYJP6wSLPMCtzUgDg4ZxsBaRv5Ah7WEUoTNJOJLQAcfeQafoOsALVJOff33f2mLftbbRB2Nu7mu3NpjP5g9i0ZiA3krG1hv90i6yNUfJ0/ylE8bLuqf/3kMXzRGy4ZXRL2uCtQNxZIyRp2D9IJ8kTNpurUK8M98GeaV/vDLd3kli3TJ++ohIZtEGfUESrfU8bY13utAJaz2Sy9DDIIWM0nMhauRsPVSPddP6SYfaRPr1tOObL0Kh2oUO9lRh+XKRwAyWgMNBlDJfJmeQZx2xOP30hbHrrgviox9dvSIE9M/mbTOzkdgcABeAE0gCvGziADkg0mgdBXptNghB3heQ8cU2aHUjFjZEbvRMyLUpcft7FoDIAjALWxG2AQzbsD2PpADiQAqwZZPlfmd9tdECAX5HVIA51rY8HhWwBDpYLgFwoEMdeR47IMILA2Bw99twhSj5m022VUEs1AO8+tf7gBLyBLwBEqBf/wAFwIjsbNZJLLSBtdQGjCyoy8auD3mMK4ACfGhHJoJqL3Ci/QAucAsMarPvJcgjcxu7MDWWRu8nP2BLe4yH9qkfwMhjLJv7C7wBE2TseSCGh0NBTP3wEpEXuWRo0UjEAsDiYQGa6AOQCPBrp/A1uqStbfD2f4Eh7WehJjtATp+FJYk557EiS2ODYAFfiDHwSg4+o+P0nqWWnrMqk0XeS0LGQCeSAgwCgMiU7yNivg9Q0jOW4wThfjcGZMv71U1/mmXmPcgOUOf/wCsgqb3AXIazjETmsj5zWaggkArQp84hPcZAfUgF75866QevmPdqezPJEyqZJ6DR6cY7FPLmZXWw5COmyLVxN8bmiPEHPptBJzJAL813OkXnEGXeCWuJ+c6KT5YZkqY/5G+dUlj/EUfrGbkge9YEXj//IkJ0z/8RC6AfYUPUvce7zUsGDPOYMaJdQTj1y/ro+7wq6kQs0pvF06c/9M06A7hb28iKd4Ze+j/94e3N0/jkPDWf9kSnrH8IiWfpW7ekQj+sseYFr0njcbDkS97Im78b1ywZkqeN5hqvFoOQMaaH6fnKuY9kIOeMB+UY/ckBNuWtRQJFApMngVlHLM48c2HsvfeCOOyw1SvAaXO38QO3NmybBkDLYgZ02Nh8Tyy7TacxqVYMNHAKqO2665L405/mVFZYFmlWMWAi764AWtSt2JRYu21yAHEeYQvUs6zyAgD5QB0ADhjZ6AE7ABF4YK0DorWLZdOGDpTzkvg7EMI6zrLLSikmGSgTS49oIC9it3lstBeAYsm0IQIIwMr/s3cnUJZV5b3Ad1U30ICiILwkTmAWCCJDUFvFEU0iWU8JRIXl8AJBUIMGRUU0Cj7jgIpPlkE0ggsiGiGiSByIiAOTEJKAGgbhQXwNLSqCMnU3TdNdVW/9zukPD4dz51u3hrvPWrWq6t5z9tn7298+5///pl0N+6qqKADNwo00sHojLPrKM8FiSZ6IUOwA7kUuPADwDWJBdsbLIgy4sToDFCziQCCABywhEgAaUMhSCEw4TygMggG0OA8AQ6ZiB17gAyAgf/dgmXaOcQLFLJdAMsAjZAWYatrIjcXeXBiDPgGEIRc6IoQI0ATOgKlWSaThsTD3KtIAWc4FioxJW+bX/COBreL7q/NgjKy34YViEUZMzD/QCCjSTWMAzJEwY+AR4kVAbgM0R1I2UOR6hBeQQnTJH9GgUzwVrN/Gbe0AYWQirEoYTFh56QXLOzImlAzhqepTmVw3mZYsWdqSdNB1AJ4XgH4Jg6HDxg20IbR0j5cRUI4yvPXHqTUk/IdXChCnA9aQPtElMrTGrG96QnfJRr/pU30DPmFS1qhnAZJQNTYA2cimtRibzhk3wA68C0EE9hHwepgMYmE965M1TVeE5dC92GDSuvOcMs/mB7A1/gidMy7rmu7Tac8FxNUzxtqzxv3QdXNv7QDG2iJjngy6QV88B6J8c6tXlL7ymiEs1r2xeWZaC7FxI2MNWQDlUcHO2o3iCvpCz+i9cViz5tPzIvLS4v7VHIhWe2W0e53KSTFOc0p2cdBVz1h99mypEgsGJAQ3cjzMO0JFD4SRWU/WTYS6Wgd0wnOlU8Wybl/92WPRraTyeVkCDDHZYzGXejB2xOL8889PhxxyaLr88m0K0CLRErgG8AAlYQtALjDmhQiEsBICzcBANVwGeUAsgKj/9b/uT9dfL6xjaWG59pIKYIugADFBSgAIAAFwAYYBaQeAKZwmytiy+rOKsTjzcrAAAi5e/qyOLMte1lEKF9jxcgOggRkgg1WWdQ8AYvUXOgXsGZ8XX2wYBUB5kbO+Aoos88Jlmg7jYxVFUPyNtAAiPkNk9Mv/ZMOSDiggL164QSwAJ3ImEy9f4IhVnFyMCQiMEp6uAcL8ABPGBeB6sSMGrJjCwvxUS0g61w+58jqwpAJoZMDia05ZKYUzIChNYQtkD6ixsAKdwCcA5UAWAFAAHiHQRlTAqcstiAVLMrBnzpFAeqePQCT5aR/46jY229iAev0DugEg809mrOVkKVcCOI0cEfMO+KgOFGMG8qwHYwQK6S750mOgDzA0J1GdTDUk9wDCeFrI0LjiADTpFpAJuCLRwmpiPlatWp0uvXRpuv32ZYWOIgUxX0GqAEBjAHrNAy+IMRozkmGuEWBWbkSRjtGjuuwQBf3zPXKEaNIZ1wD71iNAC2jro7niMUTKyLJePphekxfPku/DYGDsyBB9Qapj3xXnIC/kySsEvCKR9QNYFQpHR/z4H/n2Nz1FrgFjxIIOC8eid8hB5BgYF8Jsvs0xom98DBHkSw7mBsk0XuMGjIX16Lvr6R/ZI6EAczuPAFnpB12y7j1n9JkO8XYwVDCwmCtky/PFc8Cc6DtiEmWzeQLNPZDvuSBEdVjAPGQdJbjJkD7FwQijTzxwSFiVWJCF6zxnebLI23PNMxSx9mynW9FX3mPPfB6YCOUa9CWfcywGlWC+fpwkkHMs5na2x5JYHHrooWnrrbcpXr5AApAJ5LGcebGxhHoJCiUAtLyoWfl8XrUkAw4skMccM5Pe8pa16aqrNklTU5sUL3FWLi9q4MqLloU1whW8lGLjNcAoiAWgBoABiQAr0hP7BgDnQkG8xIHjiOf3cmb99yJHLIACVll9ZfUDxHhjACZtG0cQCyAAOfAiBPKBEP/7HFmob24VqopMAE9c//qPkACeSEF4PYBRsiFjctVe7Iqtr2TnpQvY6jNQgiQAVjxEwHu7MAIAiCXbuYCXPiN+iGDTQbbkQEaArt+AvH7wHgAyTeE65sIcALdkCPiHVwTB4f0wVhZxQK0VCAtiAZyQs3E6vwqCqwSql8eCe9MLoWAs78AzKzUSQD7ALB0B8vQ/NgA079FfVnYgXhgVshseC4AWuQC6jBugNhahaOQJhAFj9BtgjsP3gDHQTSfIEGGl+8jxYx6zKp1xxtJ0ww2bF94zOqLv+u0++iWvKIg1UI5Mu1fsem99xgZuSCUSBezXCSJLs34iVHSPFRw5QaToANIO1Bq/632HdACe1n/Vag6c03NryRitm2qiPX23Jo03kpLdE1gGToFRutpUkQgAt2cKUkfu1jwy7BlibQCujA5kRWeRImuA3oYeWRfWHS8asiHniW57XjjkMTCUMEBoxznkShaucViPvBDuR6e04Wgiu4wB5og+e15EGJAxk53+IC1kyXPEWEFP6JR+8ITx3Gkb6XA++fvbemvaE6aXtVE/l055tpsbXuQ4IleEnjCOVIkFOftxDkLL4GFuyMjz1fpDLKKvPCGeXZ531Zy8Qfqdq0INIr187bhJIFeFmtsZH1tisc022xQvNS8Zcc1etF6yQm+89IEyLzdxvl6YwEodNAL4LIjc4R/5yAPpoouWpDVrlhQvGqQC2AMKACUvrLDAeym5Tn4EQBPEgioAl1H6FYATMiWPwQsXIAR+gGF95lEBchAQL2wADTDUfyFWASR4VvSBFwHp8bLzUhR6wALnt5cm0AEMCmfgHWlVbQexQLZYo/0gPGQJjLH0sggDEcAqIIWAeMkCCuGxIAPEgvcFidBvgIolNqortSshaWwADa8PWbN687IARa0ORAvoQoTIEeg27tgQr4kUAKXCsxARoImsql4RHlftdkq2DmIBUCEWACRg161not1jgh4Zj9AX/TH/dIRuIM36bn6QTTIK63Z1vNYBcEl+LM3hsaAPQC6rrfWAWNB7XjpzCIABYkLBqhvCsWADnUJOgFr9YKUH0JCAd75zfbrxxsl0xx1Liu94elh+6Y176CMCgtCZX+TIPRELhMbf9CcINLBODkJ36hW1wptD1603QJseIgHIAWIR4zC/+qxPvDF0tmp1BnqtPeFxZM7yX5UjOVsDCKcxGTuib/2SrxAgckLI6wdiYQ3pF9khWQCqOTWfrOB0lleVfHnKEI3q4T7m0ndyY5AF3iXPM/2k+543CBH9IwdycQ1jiOccwqavZA8sIwbGjcDUSRtSTw7WITlFVTLEQkidvnvWKCrgWegz55A574mxGhcCFSRKP8kHQW63P0U/r05GAOuZrM1THDwnnteeZYhBlVgYH10kK7puznk4eNHkEJljxCL6ygNEVgwd/eyR0jSuvI9FP7OdrxlXCVgvjryPxdxowFgTC2CPJdQLLjbBA6SAFKEzrOheEsBzU8w7KyQwD+x8/vMzBVm4886JAnh4sQN0XqZeshIWY28CLyUvay8kIUtVYkENInYYyUFSeC+8qITuaBNREGPNogjAAAS8Iu7PwuiFFgAbAGNtRpy8GFmpWfX1y+dIAE+K0CQABKAQggAItApDQCzcC5AAHIRWAEL6BHCx8CI7ZMMjxEMCTLFgBrEwNtcDJj4XdsFiLVwrrukE1gEn4/fiR9zqILC+pMgVgEIEWaoRNv0A1qpW3+p1gBmgywKNEACe1WTdCN/plEAaxAL4YOUUQtHpmm4fCSzkxgVQmzPzjxjQQ3OATJFzbHjWlEtChn6APPrPY2FtALdCaQArOibEBxm0ZoBdIIwVV35HNR7fvVnZ3cvfgDqwqE3tv/CFMwVBecQjJgpi5zwgnw4A1/rBugzsIRv0HgC21vSJztGzyJegr76vWo5DfgAg/ZMXQgbkFGWkhZEhTqzn5pd13vdCyhBJfa0mh8dmlMCptVInMeYZOUbqrDmGCWuS3vMuALYRklWfXyTXdXROO3QzQjQ9M4B8MqJDcqWQXnNbPaxH80hufjwb5ODwllhP1rY8DuQYSXAP3g/ytWaFejKkRKghGdMh88s7FMnzcU/PDbJiDHAujxgjgWeeuSFnxNC6if0teK6QUmvAGBgeokyzcTtPP3027ORnxI+MPUPpWRyeX+bH8wdJrhIL40esETrPvigPTj95XzyfzWnk4tDN2KujiUB2u66r53WzU30/7eZrsgQWowTyzttzO6tjTSy83FgzgRWAArgEhlj1hFXwAnjReKE3WZa9xFkMxZafe+769NWvTqbbb19ShEIAvcAUAO9F6px4SQJGwI4QEdb6OrEIlfBSZu0D6vSNpZeVzcsR8fEiA3gRBECMh8ALzsudN8PhfG0AKFFikWU7Yt0DULMgAhf6wmPhxdpqkzYvVxZkgIsXBHCMMA7y0BcAUXvAOws6YIc8BLHQN6TG9QAp6yTrJZCqjQA+7ZaHl73xkzNLI2LRabM0oIvXBuHidQGuAIxWnoOo/MUia9wAUqcqRPU+GzPwCmQA/VF5aVhLH6hD9IBPYMw9yBVgpBO+Mz+xU3hTzX/WaWQXqAOwERPAlI7TV98j2Eifz3i3Yg8F96Ev1QpCyILwK3rGOwOIy9HQjnOjGhWdFCaDuEe+kPaAUuSc7iI5yJ3QFbKzpgBsYVbIqf4xAAjHov+AKpkHceM146khC+A4rO6s8AC3/Ad5ENao9iLx3bnyAcL7Z76QKuNbPqYAACAASURBVOPRFj2tlyjWN8CSvjBWsGgjFsYHoJKrcKuoLFbVAcQC4ZUsH1XleGfCa8EAAOCz7APHYX2vEwvkDekxNnPAmKBNwNizQf/NBT3wDESqjMeateYRC+3zLnmOMGiExxGhrHotPIvIAcHxnIziBohFGGT0geXe/4wIfsgRefAcJHMyoaM8epFYzTjSyWvZ6xpCkukmsurZEYfnPtLME0SPquVm6a5nLOKkz+TIqOTdQd88M/0OkslzRq7WYqdyvd32f8OGDUW5WbvV5yNLIEugvQSsF0e7cuZZhrMngbEmFqx1XhiAbWxy5WUNeHpZsLwBM61ebkCT0IYTTphJf/u3a9Lll2+aVq3atABdAJGXM6DBquXF6YXk8FJixQOikZdWxMK5YREHfPXXix048VJkgWRZ1BaA4+XOomg8sWcGUMCa7aUtFEU/6iUc3Qe4AqKBECCCpbVVIjKADSACXIA6T4dxAlPAKQCkjyyAXtTGz/orHCuIhbaFYwCvwq7E9ZM9CzhQxCvQ6R0KPBs/oCSMBmiKyiytlgzQxYILMPpbXxGjVofcEcCclZeng/w67ZvQRCwAGvcFvBCtNhtO97zaETjAW18RHwQCSWNJpS+ANZAN/AA7TcSCvpo7pIF3DcAzN4inzxBJ3hDEgk7yEDlHaEjIprrvCe8VfQKMeYSAMx5A4WtC+n7xi5m01VYTBaGX5CqczDk8K+SMaFpbUbggchoQi0jAB4IjfMYcMgiYHwDXmIFZYU/GQSbGh0BUSaT3j/EgJ8ZtjUa+BwJO1z0H4nBvZJmVnizq82i8jBHWAm8kMunZok3rk2VcH5Hs+qFt65UhgedUv/xG1M2DUEpryfolK/PNQ1g9jIdRA1EwXoDZ2kPOeDoAfn3TJwSBh8DzB6i2ZskLqbOmWeSRf0BcqCGQjMBXn4fadS9rFgHyjDF37mMePKMYLRAUcjQPCBvPDzJobcWGh3TWmnYdb658rybvWs8LpHKBebYu6Jb5owvWhnGQPb317JOPE/dG9hARzyTry/g9ixk2kGnGG8/ceF5qwzwjaNW8o0H6fd999xUVB5d1cuMOcpN8bZbAIpHAWg/iZE1uvkhGtLCGMdbEAsgNCyKw52XHSsqa7WUBhAEJ7UJWvPynpmbSvffelTbddFnafPMtCpDkpSxcgRXZS9kLNV7IXkpAARANnLcjFlV10l+Jll6CXnb6Bax4wbmnFyWgFztRuxa4Z5kX5oA0ADpBOqptux65cC3rLuDYanMnYAsQ5W3RvnAt1yI7/geAAMTY/8N9o6RtEAuAnrVanwFJ9/TyZi0EVLqpjMTC6lwghAUb8KlW6GlaioAqizjQCfgCPCzMrQ76Abiz6gIc9b1Mul3uscEXgNiJMHXbZpyHIJljYFpIEF1jMQXGzQvPApDMEyG/pmle6T0LNwIEVCEN2ovKVWRgXs2Xg+4CgoC+cDRzXN2lGfhW1Sx2LebFA1hZeensddfNpG22mSh0SA6C9wCyKrwKQaSH9No6MgbEKfYyAbYRCe8M5AJppnsIsWutL1Zx4Ju+01Xf0U2W+qaDvHgDhLCQjzaRDOugSgJ4EngLydw968QiKrsB/AB0FIBwXuSnAKNN/SArbZI77wU9QQzkOrGGx47WiK358BxgvKgTCzI2dusPcQDyjQ9gF5ZovvQhPLSRg8H7gJjHHifkbx16HpIN4sjoErli7ktP6IQ+A+Keo37ck+4gG+bPXNHLalU1hgThaebHc8tvfecZoD/G28/+J+3Wj/m13vVXAQqkWR+RH4SL3IRoecZFsQ1EgdcFKaVX5lI7wrg8t2JTyyAi5Ot8BNCzfxjHPffcUxCLR4Z1ahiN5jayBBapBO4FvJK1vdUiHeH8HtZYEwtTg0x4CXixCFUAGFjZvTxZCjtZwLUhno9FyS6Pm266afGy4kVAGOg3cCL0IUCIlxLgDcCxanVLLNxLP/U5Qp14I7yMgRJWR2El3PVBYnyHhLBiIgCAXFNCJOAL9JOFF3psHNakvl66AB/gJAxCTLLQBn0BGoE9AJHFlPeG1RwgZ/0EvFg0hasAXV7mqjQBHfoVZTW7McwBzO7J8yP8AGipgp6mvrNUkrfxAYlAlP61OgA7gBS4ch/jaLUJXqelXq3B3+ncXr4HAAEiwNX8s0YDcube3NBH8kU6EIwmsCZUhjcBUBQGArQCeuaE3tItv4NYmDPePgTAutE2r1McwLd+Abau4e0w74g0Qvenf7ou3XHHZLr55k2KPgG+iArPlxAiawNwQ3iAaroG5JpfoJ+XxDh4X5AhXgkkAvBDks0TT5bPEByePCQ0KhzV5WudsqDTW+1aC6zb5Bc7rbvGugVIeRUiQbvaljk2Zus/vHeAtvebOUJ6quC6ei0ZIIgInr8BVWTbOuFJ0AYZeGaYK2sF6K0e1pd76xuviudCVEozB8KOkBfPCMSFwYEBRDihvANg2JqgR7wH7uX5xSMIRCOUsdGn+yL3vIb0x9jJ23mIBcMHos8rYDxCzVj8I7SMLMibTH1Pf82jPtIZ4x9WHlLISB+jUhrvCgLI6OG5iMyQK9nwaIZnF/HyHIgwL88mOkbOdJ6eVTe1NA55MuYs9ijqZT03ncsCi1jkZNRBJZmvHwcJ3O/hknhMl43DcOfdGMeeWJgRIALYiRcOC5SXa8SBdzNrYvo8+P14uQMEwqhY64QPCAMIHfc9iyRrL6t9L8RCX6obQwEvrPBefF7GrPdCi8IqDShGcq/fQEo94bQ6vm5KniIWQCMQx3JqHMK7AFGgBbEAaLykWez0CaCIBHLkw7rXZ54hlmgvaIC9l02vgryo3iM+n8eik0HPC9+57m8eAE0ek1YH8CbMA5CWEwNANHl8utGR2ToHOAQMzS9dQmLNAXJgboRlIBY8Lghpk8cESQBozStAR3/ov3AUYJ73A+AOYgF4AlbAGdJKJ6o7tQsrQT7NMfKH3MTGjeR+yCFTaeXKiXTZZZMP7mNgrZgX9wFkEQJEEDiVqxCk0doBhukqgEcP6DlvDO8CIEvv6BWgSw4AHtBeDWuqzod701PkhnzIj1yRSbKLQ1sIgrHLzWnytNNh/QFa/Y1YIGhkCCgDqLyh9QOpQuAQWf2xTpFDXgZWfP+bEyFliA1iLEypepAHT4C+ace6RCiMx/rTD8DXfBknGfMaIEt03fPIsw+h4lVFApBHxhYeKZ6FKrEO3SM3c0f3rBHEwpzLeXJvz1frFIhHJOMwh54V+mxO9cf4wiM7jKppdTnzjCHDjCh0lLctQsT0AzGyN0sUaYjKYYiR3C/rw7x6ltAvsvZ8i76Su3bphmf/MI4pE5vMWc6xGIY8cxuLWwJ5vczt/GZisVH+XjIsVmJ+hRcA4L0kDlJkyXVBLAAQQJTVU/iSl1S4ymNPBWAJ0GNt7/fg0vcTwFBFGu2GpS9AGMAPFAIoTTH2vdwfKPFS9QL2ghZfLJTFZwABj4dxR81+RA3oD0+Jl7KwiyBCQAtQ2itgZ+Vk4TReng+ArVNIJcLHms7SioSwRiNirQ6gFeAGqIRQARCdwq16keUwzgUIEQvhWsaGLAGgLPms8AifMA56CGQ2YRNyMU4kyvn0EtgSdmNetBvhdvoMtAO9gDzQiMDwSMShP9oBTpFfc876LQRIaM9BB02lW2+dSBdfPFl4u1itEQfgDFgFmOk0ogPIilWPYgJ0GomicwA4DwEQy5IO+NFvgI4XxfwKnSIHQD/2IKnLnb7yggCE5KOP8od4qqpJ6cYD/APS9LqVviFULPlRJtZ5PAKeKQB60+aTiEUk2yMExsuDipAB/DxGUZ7ZWqP/daOEOdJHIB2hcyCJrO+xH4dnHE+iZwSPkOceD4e+miMEjceIRxCpoQfkjfz7vJpjRG+QEmPTH/pQVv0qnwGeo+5NljyDQtLqHj/nkaeqXkgwsA7Mz9YhvNFzy3pGYj3P6A0ZWDPWCq9ShKvRAZ8zQBh7GG2M2TpATKp65ZnEqELnvQOGcVTfL8NoL7eRJbCYJaCKmgMey8foJZCJxUaZAzZenqyaXpReIJ1AakyXUCiuaqFQfrzcvSCVsvTS5Lng3q8C+ij9yorIOtuvZQ4QAgiBOFZS4U6SNeMAuGI/g/AmDLrWvIiBSYQCEALehGchU6yfiIX/eVKAKC/s+q7l5I1YGDsQpI1eN8MCLN0D8AMaEZ1WeSEhDzJicfbiB3Bcw4ra6hB2A9xJDJb0yRo77Nr6gy57gNSY6BtvEDDIO+ZzXjjyZ3kG1oD3pvlHrBEAVnWgFRgUfgRwIRnWBrAbHgvWfbLXLl0XahSb1RkPqzi5IQeqCwHrgC1rNID64hevTbfdtqQoeGBO/ADb1pw1qB+xySRQTdeCWLBwR2EEVnvEyjqyhqwzln1kFdHUpjUoZAj5JJumw9i0KS9BX13nfAQFUA5vifAgoJSuy/No5WnXJ948FnDrk64ipXTcWqjuhRL9EY4Vifis/84VaoiwAfvuRU5RmlmbdQDu2YO0M1YgFuRp/OaJ9wkpEyJFV+iBOWIMoCfAs34xgljjZMuzghwgisg/siZULQ7jo1/aRfCQWkQfsfAZwK1NxFRuA2JRX+dkj4jJ4/C8MtfVcq+Dro/69e7HaxE7ZDN6IM+IIKJh/MYaoX0IEdJEZvSsU94HL5t5Q3o9M4ZxCO1guMqhUMOQZm5jsUsgh0LN7QxnYrFR/l6qwIcQDhZJwKrbiiSIxd13313E86lCgDSwkkpE9lL3kvfiqr+QvIh9Jha330McOte7H6DCyw9Y/h3pKV+KQCYgPQxPOtDBKqosLSALLAIExszqy1quL6yhrI9e2EJmqvcGWspdy0tAJ4SiU35EXUbu4SUOBACixt2JNLHCAm/mGYBAKpp2QY57kR1gzKthHmMDtX7nazauI3tgnBW6WlEL8BSmhNiGZ4zFvInEAnasyjwF/qYrqhqx2gKWrPMAfRALlZ3IJcq9ko/z42AV1575iTwJhFNbSNoee9yd7rxzk/Qf/7FlQYqDqEqYZQVGlIB3emaeWJBjnwBzAgzTNyAaiHdv6wlo1QZ9kI/gM2DcGrO+6WrTQVb0lBytSTrFUu1QLSoS/OUSAObIFNDYili4L0KBwALv5oh+mgehVU07MiMW1jEQ73wglhysHSFcDBNkjoQbs5AyhoTq4dmDtPNeaQsJZlm3vrRHnkA17wA9oC/WAhKFCCASiID5RKr9T6bacl+yqMoQcUQaECFzgUggmciQNe6+SIb+8K4gFvVQTEYI3hLXmWP3ru6JMuw1Q0ZAP70wds8rzw1jFLqFgPEShYEG4eEpQuLqGyI29Y0u0yM6YN0M45C8LQzqEe3iWIdxo9xGlsAikICd6h252MHcTGYmFhvl7qUOeAC5gKeXSyfL1O/A+0yyhbzEbRYlLnB6Deh5IbHIeyENA9TX1YTFkAXViwxw8BKsJps6v9tN3LpVQcRCOIEyoUgYcAnkABcsxUKMolQmyx2QRp5VQAuk8nawTAJSrLCd8iPq/XMP4IdlFpBhEe/k+fGyj5KnwKhEzXa748aeI2KugQTekV5C5LqV6SDnkQMQzoMAvNI1cjAnkv3pIYCJ9LXa7RtQ5bEAOJFQRFhCq7kDJIFC4C+IhdwYgFn4EVAWm8vFOMwJoIbYyEcATnmTAGuAfaedVqe77lqaLr98WQGeeQKE2bintumwuTUG4BYYjwIfUfFKn4XiANLAaOSHWAfuC5zTU+0Yg3abAL0+012Ewfq3TgFoxgb5OK6RoOvgISQnIByob0UskB594CX02xwZE88BolC1+ofMPDMAfd434BfI5Umgc7xBnkdkRRbub455/OrEAqDXN+NBEHmgomIaHal6Uz07tMkjRa7WBGLB6+B/RMF9XacP5FxNgJcTgZSY2yjJKr8FEURo9BUZl/Rt/QDr9VBM8+T56zshaEKOmnJQBlkj9WvJF6HxzPcs8kOfjZ+Xx5wjOvSPXuqj/pvvTs+Y2GeI7tWrdvU7hjVr1hRhHbl8Zr8SzNeNkwQU03FsMcy67uMkwAHHmonFRgF6yQARQnlYWsXRd7J+V2W/fv364sHPqhSlRb0gEQsxzN28kPqZS7kCwB6rIVCmVGKryjf9tN90DWIBNLA8C3kBWr2oATJxy1WC0CoZnFxYtbWBYAD33YaeNYEEL/tOL3zXAZnAoaRcAA9Qqu6kXW8b4GRxB+7oht/derKGJe9u2omqXlXy6jOAiP7ps0o3rcgyMAowS9olR0QAGDdeesXiHAUCfA9os8gjWYiMxNYqGASQEXQkDvkFGHmYEBFg7glPWJ/uuWcyXXTRkmJfCwROcrkwIeFI2tZ3pDXKfIZeIT6INOCGTCAYvBw8YNaxaxAqoBfhclgT8o+iQlJdpiq18ZwgH8YnJwP4p+vGGMSCxwAA9d6Ss9JKFwBV3kTtyUUxL7w/gKsQm6ZNFhEL40Ym9EEVNWBWKJZCCdrQHtBunAA6Pa4e5ty99c0cuI9cJwSaRzDKBiP0CJn1yzjg2WcOeBfMOyLt/wglM4dBxBEMc6mPZCx5mQ4gkGQv34bRwbMJqZA4zzMAqCMq9eeqtqwzc0+HlLTtphpfN+ui3TmeTcJeeXH01XOBnHivEGDPb4SH55nsefO6MQ5ZS+SJLDOaDOPwfhEKlTf8GoY0cxuLXQJ5g7y5neFMLDbKP14yXoBihsPq2+301LeQjz0nvKSETPRCUrq9p/MiSZRXRPgAy3F1p+Be2ur2XGALcItKO8AqQCOMgQW1G289IAfkALuApFCPTvkR3fav3XmAghc+AsZCCgQ1bRgYbQC3wCsSwvoI1HXryRpGf4fRxsZNSNv2GxhFLBA9AJT+A9aSpnmTAGu6BujSZVZq8wZcsqw7r5q/gFAgjKzmUV0IGKcjwOy229r7BYmfKMgMACvmXVga0mputA3o+oxHKvTKdcAwCzlLOi+UudE3ngL9RlLdJ8onA8PCmSQnNx3mmWVf7ghddm6AZEavIBaAPw8EMCz2vhWxQBL0T/IuguE8gN1YEAt5I/UDkEdaeA8AWKQA6ULaeBzIQ6gj2fge4QHC68RCOwiH+QTUoxww+QkhIwvPCWFMPCrmiew8qxAYcyFE0P/+RkCEECJn7i28iVWentAJhJtnhFwQAt4eHgpkJHamdm/ru4n8BwFGRpAv1aNG8SzwjKbDyKIqW54LQjONF7FGKCKXhreM0aSbg/x5+LRJ9t0YPDq1W3+/dDo/f58lMM4SyOtlbmc/E4uK/FnRgR8x2r0SAclCrElVi5JQAi/NXnMHelEJoIpVDADhskcumkBLL212OhexECYCEAqj2Oh1LMJnAJpuQoW81IXLAKRivlkyh/EC7tR3llGWblZgpAbBaJeMbf5YYoWx8AohFqPoZ6dxDPv7sJYDsbwLgDmLubAW3qQIrQpZId9CnawZHh/yrBJaVnLgH7Aib2CWp4GVHZhbunRdWrduMl144SaF10gIHyu5EBTnsXTrEys5rwlyE3qFWPB2mRfeAH3QH1Z6gBeY1icWZvHuQKKQLqRX3kXTgXzppzaQBuuJPht/lVgYs3sil/S/VYU1a0JfeAYAeH13b8n/yAXZ1g/jAHIZCQBrYxDXTz4APV1EhFnVAXXPFdXK6sSCNwPhsK4QMsYSfeU1FQ5n/SIq7kWn5ZbwIhm3UDoEkfy1Y10C3ObE57wKCBK5yJfwN48FMod06VPkpCBUiIXxkFe7dWNsnikIWC9lvgdZB+4pB8j46SnyJI/KfJEVUshbSY/pAqLRzUGXEFPkmteo09i7aXPdunWFx0K4bT6yBLIE2kvgAZbL5Pmc18tc6EomFjWp97KPQlyKHdvpUX7FqDdkiV2HJegKBwFaoqzkbCkUYMLCLVyExRO5ARpU0mKp7IZY6BuvBysn0D6qSkuRdyD2G6lQ5abTHjoIIpDAYjuKEI3Zmrd27SJcgChAyUqNWANTrMe8SfUNTJFZHidgkJU5dlaPe/AkCFcC2iLWHjkA+iX0r1lzb5qeXpouu2yLAqDyONBdpB6hEOMvFwCwVSSA7CNc1vdIKd0BXukeSzwQj0ggIwigd4v7C3ECiJHeVqQb0RXCApDzXkReFCt6lVggNO4J2PPYtXpvubd1YRzK55KfdclTgOQgb03EgtyAf8QCaAdQAXUEnO4aF0IsVFPpYF6e+oEgIGrmEwlxX4SNHHhPPSvMBTAt3wVx1F/EAfEx/ypqaYf8Xee5qA2hZP5WEcu9Y2M5v+mDteT5QH70BlEx592Q8W720Bnm2ojcJPLmWYvNAIXUkQc5+ZzHkt6Yh24O7ZJzVFkTwjeol1MyqlDbLbt9uHbT0XxOlsAilYCcV0cudjA3E5yJxRDkjlh48GPHoyYW1o84YQDByxyoAs5m8wAc3Nc93Vt8stAMFmtek17ypfohcoOMLRLrxaqL3WdJ78Y7Nep+DjLGfq4FptXfF/IF+Ppf2BKiCHDWE+vl9QDZQmGEzwCk1bwBlmCVmegHgM2TxsqN0NEV62ViYmm65ZbNi3h0egO8AWB0C6EByIFZ1wG7oVe+ZwkG3FiX9U1oljAhIJhnQMI6ECzEhzVe/+RwNHkKyMv88hggJ0hW5EXxFgSxiH08WPBVmRI21opYOJeVH0DnCWHJ94PMkq/wsfrBy8FCLoyG5R6oZyxAfnmSzIkwKt4hZEEIXyuwaxy8Rq7lCTIG3glgmQfDXEkSJxPzQfb6qm2Expwgje6JxJGxje0QKpWxhJiRsflB6pBFRJIc9YknE/HweTekoh+dHcY1DDMKFiB0yJjnJ28VrxTCi8SaSzJpV5a63hfPF/ppLhH0TsaLTmMBlBCLnIzaSVL5+ywBz/I1hRgyEZ8bbcjEYkhy56qWuD3q5DrhCsAb74EXu4ouvQD7foYPOLiveHTghAUXIGT59F2/Sdj99KXfa4AFhGI+g55+x9bPdazFACOwTIeQgUgOBh7reTOqfSEXNjYD2oXWVHdUFqYEwAKzACpgLHk3due2XspiB5sU4JWFPgied4KdzpETJEESrT6EXvle2I0+8FCw3IvzRwaErWgHoAOWgUTEArjmYWjyFIS86IRqSogFT4F2eAfcV4iMXBEAVHUsVc6Mp10uAKIgR0RyObJPnlG1rcmr6HzWcUnmgCgvgNAs4yc/HgthYTx8kouRA/1rOoRIIVnGDBAzdMulkL8hXwQx4s2RN8D7qH+8CzwsZG3s1jXrPe+U8SIygLZ55zFBXhA1BJInxW9jQMZ4Sxgg5GPM58Ncn3RSSZTMOTKtsh1dEnKHaCAVxlGvttduXOSMWAifCkI2iByEdgiFsk9SPrIEsgTaSyCHQs2thmRiMST581p48I/6AIYQC4mkqll5iXVjge+3nyySAEPEoYttZxn1vvPiFeqxEIhFv+NfzNdF2U+AEwg117xSQkTqERh0TXI0MmLOgVLJznGw6rNki79XGpXOAJy8RPWj7g0CTgF8icwAtPAkfQirL2JB59yfV8N9heawMvNc8KABwwiHalZAnj7yFrRL1NevjYauB8cLuAchQKCQLp4QYX/AfjucB7QCpIiVUD/yFY4l76Fpl3nJ0ggZr4I1BNTywvAMCGlCLPwG/AF6niQhUq2IheRhHieeCKFY+hMJ9J4R7qeqE68FLxBvFe8QWfveHCIW5tBnKnrxwsjNkLdiPsnYPAhXQ+okz9MF84J4Vfc2mY9rh+4hEGQbm0eSEe8dMsmjgTQLEWy1B0rTuDwXkd+olhabO85HGeQ+ZQlkCWQJDFMCmVgMSZoYMo+Fn1EeXowsohFb3moDtGH2CRhxX9ZfAE5CK/AFUAilGNTtP8y+5rZ6k4B5lTfD+hxeKZb/uhcM6I/8BeE2QnSQgDh4MpRNFZ4HuLGcA/es8A7rhceiycMH8CIKQnDktPDG6UNUYAL85FfwiAgzAvR5z9xHvDzSw5sBKALprM88afIlVC9qdxi/I2wEiAU5AMmuB+ZjN2kJ1d0QC1Z+xAJIFdrEiwGk1w/AX2iX0BzyFqbHcyPxGnBHTBA0pIQBIXJbWhELRMv1vESxt4zxheHB3zaJ40WRuCykDfkK44Q5FO5mLuS4aMMYWPfJlveEhV+uBkLiOaAdc2wuyE6exXw/eOvIovroNnZrgI5JmEcQmsLXWo3N2kAszLVwsEE2QXWPXG52vmtR7t98koD14sgevrmZlUwshiB33orqBnlDaLKnJrwUq4Chp4sHOBl4YNFksQT6hFyw7M3HfR4GGObYXWpOhTCx3gsliipRVUEICRJHD8SzeAOvVdAOfPIeCDPRjpAQ4IzF32G9IBVNOUmR4wCcAWSANtIS+QyIh1wE7au8BLyq+OR7lmbgHNlh9ef5kBguZwM476WwAa8cqzyPA73WltAswBmo71R6ODwW3nFCjfSVBwVhYPFvIhYMBKoRMRDwbCAVyAHPhHaEOCHuPAhAqz60Iha8ScceW3qJWpWARgQQN6V2eRwRQN4QxArpQBqECCEdwh0luFv3PCC8Mda6BGc5MsimcDH6ov/kXS1BvJAWEs8Ljw95+0Ese/E6ILaIBUJMd3h2BjnyBnmDSC9fO24SyBvkze2Mj4RYiKdWNek3v/lN2mGHHR62e2h853sW/+222y5tu+22LRPVbrjhhnSXwODK8YQnPCE9vgNquO6669L555+fDj300LTNEMsQIRb33HNPURVqnHZGZYkFKrj9gR3AUThFJhZzu6gHvbtysMJfLDElWxGGuhdKLgBvFYs28AoEVytmSeoWgoSYIALHHVcCzdhHwnphTWpKRuUhkC+E3ACwwLilHd4B7Yn/5w2x5wCAa/8BYBYZ8BMkBCnRjnAixLfbql6IuvAY+Q32owCQhYfJNWC9R6ToertqP0EsnINYIDyIFqDf9PgxbiFE5IkIuId++1+OiEpFQqmixK/7+2lFLHgWhbIhQq3yrng7eXuEN5Gj0CnhsSq+5gAAIABJREFUYogF0qHSk3AssnN/lboQNu0hiWStYhpyyUOCoJgz5IL3st3mk4Pq6Wxej8SRDWLnuYbA9VK5EqnwYx556nohtE3j8o70bszJqLM567ntxSIBxUEcj6xXHVksA5zn4xgJsfj5z3+eLr744nTWWWelE088Me1cKyZ/6aWXFt+dffbZxYPzkEMOSa985SvTHi12ejvggAPSvzIJVY7jjjsu+Wl3zBaxcM+1a9cWFthxcr0BOECF5FhgQ8gAq2cvL+B5vj7GsnvyF1QFYp2WFC30pk4WAUghU7wWLNpAaTW8ByhzLdAKkEoSBo4jD8O+L0KhmuqMA3UIqz4gNEJxtB0gHmB3f59LMheGJZG4CTwjFogv4iHXoFfLMWKkAhVdFwbkPQWoA5p0vRti4RoeCuCdTHg7mqzfiAVPD/LgGqQBseGNAdLJ0XjI1qMxErlbEQs5EkLBnNsq74n8kEcEAuHhWQKCEQvloD1mhUqx2Cu7rHQwokkfEJDYewP5QZx4khAy5KteKWyhLSYhUsZnrL0+03j0EAs5Kbw51TDBfuTQbr30016+JktgMUuAMdvB2JuP0UtgJMTi61//ejrttNPSf/7nfxaEYK+af/yb3/xmuuWWW9JOO+2Ubr311nT55ZenpzzlKeno+rayG+Wz3377FQTkDQKCNx5PetKTkp+5IhbT09NF8vZcJHCPXm3KO0asOyssUCfcRLJmLlwyVzMynPvyRiiXyRsgTwIorgOriEEH2oFv816NI1c5SfUhpAMgVqVJ+c6w3LZbL4CpHArXyAdQMpbFP0C8d4Zdne0VIGQFwJZ70ASeAXFx8mwO9sloCkFqJzV5HjwzwnwA6tg0LnYkb5dSFR4LZAYxElbEe6J6WpMhjZzs9Azcs/rbe0PJVmMgO9+TNcIgkdhnTcnwxiN8h6dFhSN2nFbvV2FekuzJyBwiNfqLWJCtvAmVwrSHaOif0r/GjRy5Hpkwz0gRUuozJM6PJPeFfPS7t4YkfEYXeiqUbdBNS60XBzKejyyBLIH2EsjrpVk+9RzC2dKjkRCLn/zkJ+nMM89Mn/rUp9Ill1ySltcy+q6//voiOW2XXXZJv/71r9Oxxx6bHvWoR6WTINaGA7HYdddd08dsS9vDMZseiw0bNhQP/XF68AtJAUKFqghDYdkEOjKx6EEp5+Gp4vyBeiAS4JSoW59TSdOqkQl3EpLEQl3dYZ51n8dBvgCi4G/hNAHsrRckvKnYgfAa1m/Wdte6B6AdIB64F8sv54EVnSdEmdSmogFAuSRcv+Uk9LrHC7AsR4NXLkrHCgFiwbfxWzfEArgG1pVm1hcej6Z9zhAHHgFhRcarGhHCxOsjUdy4kTXgXQK5EJ1Ihq+rESKgOhavTjU/pX6eil3Kx2rXHCJgCCJiQbbC4iRhC4eSI6JN+RZxCJcCohEenhQlVs2N3BQel05VuOah+g+lS4i3kru8XfQ0QgD7bbzdeum3zXxdlsBilcCUl0jyfB5tMZ35LE/vUu8yZcptkDqbRUxHQiwI+4ILLkj7779/I7GIyaAMt912Wzr++OPTVlttlT4CWbQgFnZUfPOb31ycJ7/C7yYlwlx/9atfFVVoeEV+/OMfp9e85jXpMY95zINEwH2D4WoDOfC/H/kTVcIQ55Y1+EuldZ7kOmFQXG9xfSh3nNuqTZ87t95mWHXj/nGee9bb9F30PfoUbToXiIv7x4JrNU5jjnPbtfnAA5Pp6qun0wc/OFHEEh9zzEzafXfkqjfZ9TLO+nwsVNmR/TD0bjZk97OfTadPfGIy/exnS4pQpy235I0rH9ShdytXzqSzzppJn/nMZDrttOm0fPlEAYZjPlatmkynnLIkHXccvUMONhShMVtvXZLv2MBIKFTod+jdhg3T6ec/X5L+7u8mi/yM00+fTsuWTaUlS8prN2yYTP/939Ppf//viWJfhYMPnkkvf7mwqofr3Yc+NJG+/OWZdPzxU2nvvSfSttuWFt/Qb+ui3fpas2Y6feMb0+m002bSU5+6JF155USRN/D610+n3XabSkuXPvQ5UJ0P60OI2O67TxW5HZ/97JKi0MHTnz79YF+r62vduul0663T6W1vW1LIibxUovroRzcUpGz9+sn0pS9NFnkt2jj44On0tKeV/a+v2be+dbIghiefPJ1+7/fc73fjrOrd6tVL0kUXTRben732mk7ve99UesxjjMkcAcYz6aCDJtNHPjKdrr9+oti35MADZ9L69TPp3nsn0o03TqSf/nQiffrT0+m3v50pKnldddWSwsiwzz7T6Q/+4OHP0OqarT+bOs1HzNtcPO96WbNyjH7wg5n0xS9OpE9+cibtsUdZirxbvau/K4R2kI31shDfFb3IjpxavWdn43nXS5vzXe+y7EosFVUH5fDN1/fsKPVuenomTU1Nphe/eLII43372+HN9phxkDW7cuXK4h2/u9roHY6JGW+vPo9uiIWEm6997WsJ2+GROIgpsuGQY/Gd73ynSMB++tOfnt7ylrcUXhBejvrhgXzGGWcUnhAvcefsu+++RYK4ijQe1PIj/HhwS772OcX0GUtRfEYR4lyf+SF8n0kmdx2Co00xsT53+Ny50SbvTLRJpHGu8/zoZ3zm7zhXez7Xj/hM/3yu7fjMPaOfkVAu/yPa1Oc4Nz6z+GJM2vS5nziPbOLc2GF8ZmbTdN1169L73re0IBbvec9UeupTvfi6l120qX/uNS6yG4bezZbsVq5cl048cVm6+eYtCiA8OUm/Hqp3v/zlVDrzzJl00kmbpM9//v60fPkmhRU+9G5qalk6/fRl6bjjNi0qlp166j3pZS+bTltvXa6vO+64o1hbyHhd79autXa2SO9//7IiRv200+5PGzasTZttVu5sPzGxabrppnXp2GOXprVrJ9Lhh0+l/fbbNKW07mFr9vjjl6R//ucN6YMfXJWe+cxN0u/9Xkn8e5Hd6tX3p1/8YkP67ncfkc49d5O0887T6Y1vvD898Yn3pS22aL1mU1pWEIunPe2+9LjHTae///tHFOFQu+32QJqaeviaXbXq/kT27373I4p+2q2ct+ADH7g37b+/UJhl6bzzlqV/+qeZtOee96eDD16X/uiPSvnV1+wxxyxLK1dOppNPtobvS5tvXsqurndTU5una69dlt73vom0885r03vfe1/abjvPq2Xp/vun03e/O51e8YrN0ic/+UC6+urJdO21k+mww9an1avXp1/8Ymm67rpN0s9+Npk++9l1adWq9em885akH/5w8/SBD0yk5cvXpa22Wlt4oxfD866XNXvbbTPp+9/fkE44YWk69dT1afnypT3rnbUUsgtiYb34fKG9K3qRXav3bC9rtio7slrI79ksuxJL9YJRYt08+tGPHhjfLRa9W7p087TvvpunP/mTqfSmN61Nm2zSHt8NonccBJ5Zc04s7rzzznTNNdekr3zlK+lFL3pResELXlCA/6ZDVSgVpG6//fYiH0OFmZe+9KUJ4agfHlJADEVDWK688sr02te+tvBYhHWsasmvWi+bPBZxbtWLER4LExEW2EHbDAtkWImr3oXqZ1VLZb1P+hCf1T0W9XFGm5Ej0mqc0abzMOBbbplO73jHRBEz/oEPzKTtt+cdafZYtGuzbql0btPYmzwW1XG6ZiHIrtV81i2VEVo3Stndddd0+vCHJ9M110wWQHjJEnPx0Pm4886ZIkzqox+dSOecM11YZIUiRT9Z1//hHybT0UeXltpzzplKL3vZTGHhD49FWGDDal216Lr+He+YLOL8v/QlHpNSl0PvWPbf+c6JIjxI4vY++0ykpUubPRZf+xoCNJV2330iPepR7T0WzXo3k4z3ggsm0+c+N5l23nkmHX30dNp+++kHx9Okd/ffP1mET/3Zn02nHXaYScccs6QILdpjD17Qsq/VNfvAA9Pp9ttn0pvfPFl4DSRsq6T1938/lQ44YCbxgHznO5NFzskznzmdjjhiOu28cymTquy0edRRk0WOzD/+IyNE2c+m5926dZPpxhsni/Cr3XYz79Npyy1L74YleOGFM2nffSfT5z8/k/7934U9TaR3vWs6rV49k665ZiJdccVEWrlyIp15ZumZ+NrXJtK//MtkEUqnj49+dLPHYrGvWSV5ye7IIyfSV74yk571rPYei07Pu6rHYrHLrq7L9WdgfleUz9D58K5opbdzjVHCY8GgMSgWa+Vl7LRm27275wKjwGvek95Jb3nLdNpuu4e+f6oYY1BsLDpozj0WatpfffXV6cILLyzK0T7nOc8pfrdLhMZeEQr5GR/60IfSS17ykvQ2tRLbHDnHok93U4vLxOypHKT2PWIhFlySas4vHK6cR92aWH+ViOTPCG0xn/V4TGEyiIVypioHiSGv5mEIcbVZnTQoe0rIgagm8naKGQdqxesrECA5u3p/eic5XLiN+6jxIG2rKbdHboV9OVRjkkDbaj+HTjIWciW3QciQDeRUQVJ6tp2uq24lD8W+EPIPFDdQKYh3uOk6Y5HzIFlaLogEcQna8lPshYBE2afDOpNoLY+iVbUhZEvFJ3kd5NIqnjZKyKqDYaNB81VN1JdDIS9EXoXEc3tjxGaE+mI8krrJWPK8RHdxvMoVy7cSHjeOh7mSY2H+rKFBY5pzzPg4alEec78SyOvl4ZLzPkIqFA9p9+7oV+bV6+ZFjoW8BxWjeBZe//rXF3tYBNtrN0islNfi8MMPL7wc71J+Y46IRcS9jlNVKEBIqApQAkCoQKNSzGwmBQ1D6XMb7SUgdEm1J0m4dopumk+AFGi2uZoEVWC7mienDd4OFnqpUnVAX7X2tOpNqwoW9E5yuFKsDn2Q99BU+hWxkJMAFNs5udV+Dp10AsGRpPy+95UECaH6H/+js67H5pWs/WpRkEUrMmBcLN3KNiMWkrOFQ0leR65UmUL2JFIDqrwMyE39KEPPyn1IPBLbrce4J+InqQ9xquc7InmIFV1ALCTrI0D29VCm1lyoHqXPdEL5WgTDnLQqc9tJ3gv9ezJDvP7n/yz1T0jcIEUtulkvC11muf9ZAsOSQKv14tnomecZN244JYiFioKMwbvuOixpP7ydkRALMWrf/va3i6Rp5WZ5JKr1hT/xiU8kyR6ve93r0uMe97gi7lpIUWw2ZxfFyH8wBC4WFs/f/va36Qc/+EE677zzilAoBGMuiEXEb47bPhZensCOCjnSWwAn4TDjtmBnb3nObcsewq3m0nc2sePZQCBUem46t1UbYmblDzXtY9Fp1EFoeUKQCSCfV6CpAAhiAfyyuqvM1FQ5qtP9fM8CbSdqTlH3si8Hne9F19vJ0z1i3wQJ2yoz2VhNoh0PgN/WmhKyrE3KzNqbo1X53F7KCpKne9ljRAWjpjFJFzPXiAUvCE8lsmZ3dP1CLBA3XixeD14sZHMQMN3NvMznc+xlIZxNxS3EYpC9urxDvQNzXf75POO5b/NFAk37WHi+MogxiqhgOG5GjyAWDEj21lHVcLaOkRCLT3/608XGeG624447FuVkD1aPcOMhhEk5WnkPSIUHqHyJ96tVmbxAjykStd/97ncX/0vW/uEPf1g8ZLfeeuv0F3/xF4XHQttzRSzuvvvuB5O0Z2uy5lu7wIvFytrJU8FSmau7zbdZmr3+sIjbidmy63UfonY7b3fqcQDwww4r78sLoOpSU3gRYiEURRK63a573egs+hJhf7xzPA5KLHsx9UIsuhkXkC5hG2HwiLSXBe+R0rO+U23IuJ///JQ8Dps22ut0n/r31rEKUmTZ5AFxvvEL/+K1OffcklgoMWsjQy+qT36ynAOkw2NbiBSyOc5hkVdeWc4VuQlB4OHq97DzNi++aoj5yBLIEmgvgaadt4XPXnZZ6ZVlJHnxi/t/HyxE+Qex8FxmlGPsmK1jJMTiiiuuKEKd4nje8573kE3yLrrooiJxu3qoCvXHf/zHxUfnn39+UjZMQrfjW9/6Vrr55psLa6esf5WhHvvYxz7o4WglrNnKseCxkCeiP+NmUQJKgAugZJ99hgu0Zkvpc7vDk0AnK3yrO1kvPHwqtvR6uGd4yoQ2RQheK2JBP3ktVK1qt1N2u34gM8KhWPZZ5t1z2NZ44/LwFwIFhArx4g2yHwQyw9ommV3uhUeh7/sN7WoiF+1Ikr6xwNv8DslBKnmKbGQowZzXkpdFOdorrij3LGko0tfrVC/o8//rv0oPFw8UYjHI7tu89FHNb0ELJXc+S2AEEhDl4oAbHZ6r3gP29UH4GUHlrTXtJzSC7s3JLYJYMADJ4/Nema1jJMRitjrfa7uzRSz0Q76H0I5x3JCll7CLXucsn784JaBKG88kctHrQd9Y0Fn2hZe0C8HjsZAD4gcRGMSCjlyw0sslABYHaavVmIUlvfrVJbHwEqx6AH3nfSksisdCvsewyU27uagWG5e8z2NiZ2ky3muv8iUdOSWzIZte9WSuz7fRJMvoi15U/gyySd4g62Wu5ZDvnyUwagkIlWfwFQHjWL26zFWzASiDh1DWl7609w1TRz2OYd4viAXPPU+q/L3ZOjKxGJJkKfE4JW4PSWy5mTGVQGyJ0++aAfIlbysaoEpRq/wOITqSaIVCDSNsaWMV5FkhFaEK3NSs/x/+8EOVI0IPeSzskv2Od8wOuelGJeWcSOZW7epTnypJRSYTD5WcXcoRUTk5wi5Uyer3GHS99HvffF2WwEKUQH29KDAhpFQZbM/Wv/7rlF7yktL7PC5HEAvPah5mIeyzdWRiMSTJSq5jfe3HAjukLuRmsgQWjAQk1wntCItSPx1X5pRFv13suhcKK/8TntDPHebmGuPiyGmVmP3zn5f5HV6Qc3UgOcrZelk9/vGZVDTNw803l1ZSeT2ielXy6vfgEUfCB1kv/d47X5clsNAkYL04ojiItci4pLgEj7Mqe9ak/LBxOYJYeGfK3xMONVtHJhZDkCx2LLlOfkU/MeND6EJuIktgQUnAeoldtxdUx+dBZ+dL6OEovDfzQNx9d8EGhV/5SknA7APyp3/ad1NFDh8iHjHj/beUr8wSWPwSkJMEl0Wxg2uvLYt4fO97JbF4+tPLPDUhnH1E4y5IAQaxEEYsV1B1wdk6MrEYgmQzsRiCEHMTYyWBTCzGarrHcrCq0EgYVbcEqWAl7fdQ5Ub+XiYW/UowX9erBPotzNHrfVqdz3ABDPeTG1cnFgpP2LASwUAsbOQr14K31V4z7TYRHdZ45rqdIBbyTRQI2VhkdVa6lYnFkMSaQ6GGJMjczFhIYBihUGMhqDzIBSsB9fIBGiEY4rnlxvR7tAuFmmsA2O+Y8nXzVwIiiXjchJnORYVjpAIA/n//r6ymJpeulxyueiiU4h02KVXVTvKyqoLWpnv4TvuLvVR+EAvhwZ5FcuRm68jEYkiSzcl1QxJkbmYsJJDXy1hM81gP0i7qKkMpIPBnf1YmTPZ7tFovUaJYOEcvwKvffrS7br6E6M3G2MYp7M883nhjSgcdVBaQYNEftW6tWpXSD39YbuQm4VrxA+C/26O+XpD7q68uC3jwWPhbWWxl8pX09tliD4kKYqHgho07/8//6VaavZ+XiUXvMmu8YpzLzQ5JhLmZMZJALp85RpM9pkO1Wzmrr+pdiAWQ1O/RtF6UH7YTu93S7Wny1KeOHgDGeJQfZgFVVld9/GFUYOtXVsO+jpz/8R/LIhB/8zdzJ+N+xxUhRYoIdDMvACjgbXNOoP6Vr+x9A9R++xrX/fjHpafvn/+5vP8rXtHbTtH1crNnnJHSf/93Sr//+yWJ+Jd/Sek//qMMiVItysagfWyp1PMw55J8B7HgBbLP0Gc+03P3215gbNYKz8/NN68o8sJ23333jjeZmAka2PHU+XnCbO1jMc4b5M3Pmc69mu8SGGSDvPk+tty/LAES8CK/++5yA0Fg22Z5/e470rRBHjBvh3P16L/xjdKqywLbDXgc5gwBEzZulEei0gzwPeo+DHM81baAJQTxrW9N6Z57Uvr858uKbAtlfOZGRSS5PnJ8/uAPOof8GO9VV5XgUylvwL4Xb8GgcyG5+MtfLknFTjuV4VA2s7NvUTuvQpTjBmzrG+TZEA/J33PPklgokW2MT35y6VHcccfZ3yxP/370o5RWrizHM0ovkHsrEY74M0Z4ZiDLwzq0T76eQwcc4F4r0po1mVgMJF/E4u677y4qQm3uqZOPLIEsgbYSsF5UhdpynLY+zToxVhJgKRbL7UX77GendOSRrUsI18Gs/6vgVbEDVaGiyo3vlSVW6cZ+Ll/4Qkp77FESlyc9abTAVzy+EsjAkj1YjNOxUHM/qlZlcygm/ZhjUhLa9g//UALThRKPz8ty3nnlhmiAuo01bSra7jBO+QiAvE04hc20Kn09GwtaGJTNQYFU5ZqPOKIExO95j520m+9ozu66qwS3yML69auKEx+5cbAIkpwnuU5yLLT1X/9VVof6279NabfdymTu2Tzo0kc+ktK//mtK3/9+WYZ6FOSCbMzp//2/JfEX5oZk0odhEWTPAEYOmw6qvPWEJ6xIa9dmYjGQPiEWa9euLYBSrjM+kCjzxWMiAetFlZuoMz4mw87DHDMJABPvfGcZZsFrocRluwMI8JJmma2CV8VB7GOhpHkcN9yQ0rnnpnTCCSl94hMpCR/55S9TOvvsErD0Chr6zSPgOQFWVJoR7sVjEdZj/RgFeBqmWrHskr05YPG/5ZYSYEv0FZe+/fb9e56G2c9u2rr99pS++c2U3vWu8gf5A7zbHYiU/AaA/uijy1wL1ZNGddBh1nQeBftO8PjxNBx1VEpbb93cC95Ba8FO90Kott9+XTGHsV6ECvLEIEsIhA3yrJ/nPS+lQw5JafnyknDM5sETg1ggekKxENR+PZi99JM+q4j16lenhLTRaXkz+kBGvT4nmu6NtPGK2R9Dzsr2269I69dnYtHLPDWeOzU1VViU+t1JeOAO5AayBBaQBKwXa8WayUeWwGKVAIANHF1zTQnOgO5Whxe+xMr3v78EAay0EfoxvRH1V9eL5FMkApAClK68srRMesH3ChY0DzQ/+tGlVbOb6wElyxdYYf11He+JvQG0ZeyIhhCpQSz8+qatQdrQV/Kt8LKHTYPvkQfg1GaGrLruDYB+/OO/+04uyyji8YexJlipAdnTT0/pj/4opUMPLTeGi4NcHNUQI/uuAKK8NOaPx8Lu8aM6lIQFeq0FIUu8Lbxwfrfa6FRpZxb4j32sXA+77z5dhKzFehHK5kCU6Lj1ZV5f+MISZPPktNtodZCx011ylgtlvXLSf/GLJUFt5YEZ5H71a4WS8SIga4jVZZeVntTPfa4McRtG0jryzcOEvJ51lvlakSYmMrEYeB4lC1HiDJQGFmVuYAwkkNfLGExyHmIhAaRCqBLQL9a7lZXQ92KwWVCFarz2tb8r9YmIO3j54kAgvvSlsvLUPvuk9JOflG37vJcDeBYyw9ooifWkk9oTi6hEJQ5eDDzQaa8OG26xcovjf8MbylwEJIknIyzN4RVBXLohL4Z96aVlmVAx//14YniAkC6x5cpsAnNN9gyeFyCMHJAKCfdAmHKnNjpk8ZYn86xnjQYQ9jKHrc5FPi+4oPRmie1H/GyOxlLOyq/UaliwQy+FE/kcECYLORbI1KgOpOaSS0owjuQhBdtuW3pOdt31d72oesWuu64EtNbDKadI9J4qCIT1YnyHH17OGY8BYC+ZW/K2MT/zmWWIFL2djcP9kW/9B/DlMwupE37VLnfFdf3oe30MF15Y3ld71qd1yxCg2AJjRzuy3a08kMGvfrUMYfvsZz0TVqTNNsvEolv5NZ4nFEqykDCoHNoxkCjzxWMiAevFQ78a2jEmQ8/DHDMJAO1/93clwfBSB3CarO/yFM4/vyzxKeyDVVXVGofQQR4+eXxxsH4CSAC6ajfAgnCOXokFgMlKDXT84R+WxCKOpio2AJ9EZpZeABxQBWDFqr/udaVV+dhjU3rsY0swJe56l11K67/KPEJtyABI7BQKwoOAnJCdUBfD79XJaWyAnJAgMgOmmjwOwoaAWuCTJRuYFWbGyk22xs3ibxf12djboSkUjfz99DrmmD85OBdfXOrbOeeUcwVky5kgF14J92W9DsK1YkWZu8NbgMypyPS0p5Uttqtq1G1OTafKSPppvsmfTtIl4BfRk6sU/RB+Q5+QHjkh5hYRPO44ZOH+gihsuumyghDypiG3gC8LvT7IczBOOk+PW3lD+n1cxXxaX8K7yJ1+WQs8K895TuvwK9cA69a1n25IeFM/9UHBAeTC82TvvUvyhWjycpLdIF6TmPPLLy89RsiFfJYdd1yRHvnITCz61Z2NSj6T7rrrruKhn3dGHUiU+eIxkYD1goTn5O0xmfAxHyawKmYcsAHqJG7WD+AVUUAuAECW4oiHv+eeewpveCSjuvbUU38X4iJx1fXCK3olFmKwhcwoi/uUpzyUWADTwEm1v4iS0AfWbGBPP1l/Wbj1W9jKySeXxMGPOHZAXfgFcCVUxX2ME3hsB5qBXCCMJ0FZUMSp19ANHg+hJ7xBH/xgaaEW314/gFThIkAYLwwwLiTHOFn7gTvVdCTjz0aVJPMAqFVJDyszkNlvtS9WfKCPl4VemBuhaWTAio08yDlQGUkokDHLl0HCyBuJQi4AUoe+kEN9DnopadsUflWdCx4HYVAs+nQLUOV1Ep6m7w5tCPHiGfNbWJO5Qg7p4LOffW/hadpii62KIgc8H4gG0h4HrwgdNH6eLOcP80BoyMoc8qgIEzIX7iPnQ85VGA6q9yVL40DcESI/vep8tGfNGbM1gLAZq7XAw4hMITf96nLkgzEO8HBp13MAWd1ppxVpm20ysRhIn3gslAMElLLHYiBR5ovHRALWy9KlS7PHYkzme9yHCcB56bJ+Cx9qshKyZorlFzYhDh6wkFTq4OFDLMJjAeBJ2EZEAF+7CLNC7rxzSl//ehnM4+HTAAAgAElEQVSG1K2VG1HQN+BLmAaQ6QBwWCC1DdxFm0gMksRDIQ8EuRCGZc8D8fjGJ3QIoQCqtAmQIy+8ISzorLAAJKCr3aaN/nzm3rwNgBYZPvGJvec3uJ51msz0SR+bQCT5SdImN0DeOAFCYzQnPpN8zwuDEA2S81FdD8apLLEKSMKxAFDkQn+FMfFyIVdk1u2cRvtC71jzWaftTYGgkTnwLQ8GIESi3v3uct4QSGPmHQLYERJECjHUHzrifJs9Btg1x0LNhNbQXxb5ej+DeOgXPSE7YWVNgPn448vQPGvAfY2Bl0sCNz1ymA8x/XKWyIf+Wj+IJ9L04hevLcjr5OTmxXh4ZMw52cYRYWLGJfyQXPSzkxet07Msdg3n7aInyBHSf+KJ5VwgE8IGfS70sH4gUYg7uZsn5KmJCHfqh++VgbbmeD/Iibx5dvTjGc8o124/Seuxhw55WjMMAIiFdcqrh1j8/u9nYtHNHLU9xwZGQjtyjsXAoswNjIEE5FgI7ajGjI/BsPMQx1QCLOGADFAL+NSthFHVRhiH5GcEAYiNRNv6egF6AW3gCkhhXWaRZ5GUoImUaKObEppAGus1IAQUudbBWi1UBPgERgBb4FoIDeu3WGrAkyVZngeiwAoK0LGQ+hvhEGLCUwGospQiGIClcC8Jxfoorl8eCvnYU4Cll8VXaU6AyN8ADCt2txWqg6y8/e2lx0G/9J1nRmneOIzZjz5LELYxHEB2001l7X9eIIDMfSOMSnUlfe/XklxdBuael4SXB7AHpAFJeSVkwjOFLAL+3cTDx0Zl7sFTRi/oigRnRJS1H5FQGhgQBDDl8yCxxgMEs/4jdUjCfvuVuiD0yHmIIoIRuiVczTwhG+aYFbzaTyBUaBOSJCwOAUGc9ElIWZCQyJlAvn3vXkJ19NleHIA/AuAwFn20XhBspAJ4dm8VZvfbbyrttddMWrduaTGv9NV6QLDiIBd6z6NBP7XpPrFfRjX8KAoVdEPs6Kr1oG+8k4A2QidUCCE3Zp/5oVv1gzzlAwkrJHu/rYlWR+hvky7ywgH8QrGQK4eqTeaWB4enppdSwu5FLggwuSJHdFT7ilSQvTkWCvXEJ2ZiMfDrTtUOQClXhRpYlLmBMZBAXi9jMMl5iA9KADAXBiFGPGrpx5dAC6sqAHDaaSW4BazEtQtDcdSrQkVoEKAuIZdHAWhh6QWAASVlOtXpD8t6q9h21nJhEgAtcBagULiGvA/34MWIZFPWVHHagCKQCfwFgGEB9ZlwGgDNeYAUcK99wBmRYYEmBxZpIBrQAUiASInozgFGgT/WcFZkQFhITjdkiczIFRhW2ciYhJ6w3gKiqgABSGTCU8GDpM8sxAgM675xsygD/qy6yCD5AqPCvozP2Mi331K97h95JDw5yANAyrpOJ8hPuA69IFfArV28vfaM2ZiE0ZE5kMqDhCQJBaM7vAVAunEhsAA53RSCRAf8IFbAp3kxV+ZYIjWCJtwlQDGyhuAibMCl9qpzRGZCsugT8oooSqzmhYkN+2KvBf3m9aBrdEZ4ltA5ukBP5FyQtzHwqphPZMHaolPyQZDUAw+cLsazatVk0TdAH5lUSCAOY7T3graBd4BbeJT76R+di3Afa0tfjbmJXFQTrcnfekeyrEWEjRdC/geirU39ZBBoIgyup3s8gjxsdJIxoumgd0gzYG+tIxfVhG+eHP0QEkXeDmNWTlg/eEO6SVoPPTU/5t/aYnAwBp4YumE+kG39kLy9446ZWAz8CpRcJ7Qj72MxsChzA2MgAXX5efdy6OAYTHYeYvFy9zKX0MxKGKUtAQPVoFjvWf4APdZk1movcFZgR329AB68CQAFIgIo+gzoYoFkiZb0qt2w8AM3AELd6g2Q8ahI8nRdhGcANUAjiy7ywnui3wA4QCf8AdAyJgQEIBM2xOMiuVtsOOs0UMMKLQmb5RtAB/RY/1n+AUagGoBiYZeXIVQDwGZNB/p5S172spIcdBu6wXIMVPMESNAFeIFk4/QbydE3bQJ/wBswh3iwwOsTGQCggD5QD2wD/ORqHoEy5MK9ALpe98cF0MiYl0hf7J4OdJIBkMtT4Z6S/xFNoTVN+TmxxJAg1nteF2REX3lflJt1CLcCkpEUsfd0jNeJfM2lHCD94YVC7owHWSQjHrcyMbcka0AkkkMfzD25IaDmuJo7EKVg3Q+Z0Q4vlspnvAgAPN2kv+bFfZEKugAoA7BAOfkKfaI/4VVBiJEi9zAPxs37c9BB6wryePfdmxWkhoXdHGs/Dv12Lnkh4X7rG93mqTHfUdiA5wApAaTrif8R+uRzc2Mt0CPjQjjcm64jPEiMtSFXxPp0n/oR5X71gx7wCpiDpgNp+8xnSqOE5wHZuG+scaQUabeGIgzMGOkWEmmPnW72KDEOMje3dEi/ETHzba6sEYRb+JbnB2LxlKdkYjHQq0+OheQ6FW7yztsDiTJfPCYSsF6Q8FzsYEwmfMyHCaR6mQMqLO8B3gEqL35WPyCApZRVl2VQ6EfU37fz9tSUDSW3LECCc4FvwE6OBeu5dlS+AUB5FYA4xAT4dR8ggHWc5RjwYMF2AHLi3lm1tcfiGNVeeDMcQDlgBXwCR6z6QCeQD8gAdvoEkCIDABYrMODH4+B/CcGuA0yBG4RK3/UZKEGohL8I8+KpAJKQGmSGV4TFGcjyf4T7tNvgC4DjVWHNFp+vX0gYKy0LPFkD3kAy0MkbIrzLXACj5kw+hTwTFnRAU/gHEoEIGP/PflaGBfG0IHQIGIDb5FVAIvz4jswAZcCQ9d11PjNGoJxV2XfGab5V9EFMeSGQGp837dwcYyZH4Nb/5jqs1cYEQAtHA0QRQSQDoUQ4gFpzCtAaB7IDBAPsLP/kIs9FfxBNYzG/dMcP8IqEVHNY3Mt3QnGQTmSYZ8Y88CronxAk/RLSR+YAbOwgT1cQPfNIj+mJeaBPdIR3hIyAXXOIRL385as3EqlHFP2N0EJkIw73JA99M5/0GkDmATFmeobguxeSi3z4rho6RD50wJqKimH+p6d0DWnXvjVJXxBq99VPc0Kv6gcPhPHyACCSyFDV01I9H+kkV2vPuP2EbBkUeJDIH3lA9B3GiVAhcjxY3SSt86KQg2cKYu6ZYlzuSzeQFOM2b+bzD/9wRdpzz0wsBnrtIRarVq0qiEUunzmQKPPFYyKB1atXFx6+avnMMRl6HuYYSgD4ApyASOArwkhYMyUXe0ELv2GhZWkG4ry8AXHHTTfdl664YiLdfvvmBYDxPY8BwCAvgUWVVdf/gIjvgCifAy/AHPCDWCAYSAXwCiwDOazDgKZ7AgqAqz4jQgAWkMkrArQIr2C5ZeEHjpEPZADwQEq0x4sBfEaIhPGxjgNmgKtzESHtAIe+B5AAZ0TH/QE5hAe4AvwBSGQF8AaOgNwI+0AcANLqARSyrgJ/+gR4CmFyD4Ca1ZwcxfoD9PoOcAF8wB1vh74KzYnYfSQDUEUMtQ8kHnlkCQJ9LhwIUK+GnyETzkUMgDlzDKCZK2NV0Yh1GtExNiCffBAv/+sboM8rYC4RC3/zbCBcZABwIxrmGSgVqiTRGZgF0FmmHcYFkCNPLOhkHPuQAKDaMB+8SOSEaAhN4y2il8gsYM2CTu/CQ6I/9AKhAdJd6zvtGTM589pFYjR90ndeLDKmC87xPc8VIhDkLELCkFR6xsvlMzqOmBiHudVPv5GAl798bVqzZiYdf/wWxX1co8/0JA5gmJyRSYTYHJEPvUYKrFEykTsk+V8bxocYh96Zf/KUCG/u5RXpjzUKtCN2PGaIDw+P9pBmfeJpQLTqhzUDtAub4ikwrurGmvSJbHlIrGHkgTfBHNMja5SH0D2EwhkXHaIvjtgwjzfM+YiF9dYuX4iszQ8vC/KHtPhtrNaR55h+kZ1nDGKxfHkmFgO/6h544IEiETUnow4sytzAGEjAehEKhVzkI0tgsUsASAUCAFdVbyK2WpgKYOrl72/ACYBCNljyhZ4AJ9/61oZ07rkT6Y47lhRABXgELgBugIl1WjIrQAsUAQJIAkAsjwFJANIRCmAeaAGSATj/6xPgC3hFgizrJ2AIoLDkCmsCkABEgIQFFPg1Ntb1CLFBaAAZYNTBawB4yvsApAB99wJUgUAgU5tAHpKCSOh3hFIJNRGqQw48OORirAA2oA0oGaNQMuAmvBhCUoQQITRCjAB58f28OTwwxiV8ByETKhXJ3cATksBKC7C5J7KFhAGh/ga4eWncC4limUY2gGpgndUWKDSnwDjASj7aIzPA35iAMLILUghM+zs2EAS6tWveWNcRDPdA8pADxNGYAXMeGfqDIOqnuQWOhTohZA7tIEHmguclNjQEjpEJ8w0k0i3ki4VdOJj70EXtk4N51AYvgh+kmLycSz4IEQANTNM1c0tPkCjjNmZy0BeyF1ZjbgByc1/1CpAdEA5sGw/ihUAhYmTB26WvvATa5gXbf/8NxdycdBLjVTmH8pWq1nlrjqyRTXpkzhA980ZvrFF6yhBALnSG58HfXlv0hZXePNA1RJruG5ewL22YR8n3UbgBabAe6BcvFyBe926Rue/0BflxT16xCKtCZtyPnMhMYrp1QwZRwlmYnv4Ym+t8h6A4oviCZ4w1aK7pBf1p2pDP9Tyo8n2MBZGhczwX5olOkAfZ0znrSyjUc5+bicVif6/l8WUJZAlkCWQJzJEEAAEAyMsY6ADMAB/AJMqf+h9AAzyicg6rJ4svUgBgAf48EgAPoAy8A45e/sAHgC52HfgAEFmOWSoBOMAsSq7yWgA02mJlBbCBXmEsYd0GfoF8YBGwAywBRBZ+YI2VFqhHmAANoNH9ABhhMn4jDUALEMsSDrggCFF6VxvICmAJHCFegBTQigwAoEgNAOg3IuIAoBAjXg79MlZ5IkAn4Os++iP+mwUc2AVEgV2EiaWbFwOwRlwAYQTBOcKSWGHdW6gJouBH/3l2kDj3RyYQKOPXZ/f1w+PBogsYGo9rHbHvh7GYS14X80FO5InoaJMnRqiWeQN26UNU0wLC6Q7iACyTM/CJDAKKgJ450ic64Vw6Yo7jkLPAAg4AIiiIGoCJPNAJ4BdwZ53XFlmx1pMl8EhPebHISR/dHwj3HRkCsUiX/tFJcvS/sSLNyIa/5TjQO//TT1WTEFB6Xc0jAVzJlI4InwJuzQtdM25WeTJGnng86LCQHERJe2RNR5xTzc+hhwgZgmq+eTvc1zqlS0gMwkVW2nQ/YwWw6Yy+012hWcZsHBG+hfiSl/VjbO5vPem36+mZvAhkpZ6Xg4iZS/I072Rr/VsfZEZOyKM++hsRMS7tmDdr0bzH5o7uhVSFtwbxcT0CjYx7JiHAftPbanJ6lELmbXWu69wTieQNtK48i9zDGrCOkBDE4kUvysRi4NfNunXrCm9FtsAOLMrcwBhIIK+XMZjkPMQHJeBFDIyyfnoJA3QAJYDN6sjS5xwgEhgFmIFMIRpIwN57bygsu8pnAn6sv4A374TPAQBhU0AjbwJigSgAw4A1EBK7RwMhQkkASmCWRdz9otQmEAQ4uY5lVbw50MYzou8AG0AJRAPpAD4wgjzF+SymkXgdZUQjPIjHAsDXhjAj4AxwAeQAXPcD1nhc9COSbI3N+eQSYWWIBhIGGAFKrhPSBLjy6ugTkAZYsaqyUvOEAJGsy4Apyy9QFpW4AGr3BszleACVxi8Ui4yFgGhT+JHYdwCa1wmRAMSBSjIyt/rAewTYA16IlvmjB8ZNniz+yCVw6lxky1iQC31gkQbmtA9sAm+8IHQDYQPkfIdAIBYAqHEiYoCreTTGOBAbIUixW7o+AclkCZT6HukR2mMs+i2sSh6PPiGKwKn8HdfQRWSAnPQDgEUmJILTC+BaeBhvC93iXTIucyg0ju4j2PSPrtPZqhXfGFneA9wij3QcqUWCzTnCgSQ613i33XY6bbfdTLrlliVFGJO+sOBXQbxzycC43JP3w1owx+RN94BmZEK4GLmaHz/mkKcvcg3oLLkgJtYVnUWOkQI6gdSYV7LSd2vButVGVIGL+aE35oOchB/ROwRNeBMZIZsIHO+LtvXdWmOU8LnwJPJDXMwhXafzkaStn0ia/iEf1hVZ+FufIvHb+BF2nhcydI5xW/PWFc8FI4Y1S27m3xpCzOxj8ZKXZGIx0CswcixUuMkx4wOJMl88JhKQk4SE52IHYzLhYz5ML+io9Q98AEQ8DSyAwA7QBewDrUJekBCWSwCEVfWNb1yTli+fTCtWbF6AOeAKMACqAA8gACACClm8WaMBfmQCAPE/sKs9Vk2ACEgFBLQFQAIGQBlrr/YC1DiXx0IoD9AOoAGqLL4AH2ChHQAHeBfegTgBNE37LgCVgCTQwxqOaABDgB3LtrAU7YfVmsUZWAGUhJ0AV5HcjQQYt3GqSEOe+s6ijuRE5Slx58AWMgDM+U0uZIRgICv6gYDw4gDDCBGA7BzAWH8ATfdhSXYfIJqcgHzzBmyJXUfe/O07pMQ4gX7XAN/kTV7GDZzJfwFOgcOIYxfmJlQq8j7Ms5/YDd1vsjEPSJ+/za8xG5uSpyzKAHfs4G4Z0iteJ5Z0usRKbR7pHvCo/woNAJoIJ2u+vgHv5oysyJ8ckCKyMP++R3LIHmlCThC+iPGnrz6jE0GoeVyAYbJwH7KqV12iL9YAWSE45oW86CMigmTRS1Z6hMea+PWvp4o5eOITlxR9oVtCl6qbGpKffsS+LeSErPMoAvXWjHVL3yMUjd7SDXqGyNFx64mRgG4wDgDn7iPnh9x9z1NjrLyG5MzbY41bK3TY+o9DOzyVwsIYGozJveg5ksODSA7WAzLOa2aMPAqIH912T2NgUHAtUoLEOuggkmYu6S0vimeQ+UNEEU0Hco5A0Ef98D/dtX6shagWx4OHQCHZ5ETvEIuXvSwTi4Fee4iFqh0StzOxGEiU+eIxkYD1gljkqlBjMuFjPkwvZXkOLLZeyMJOAB9AwwsdmBa+Aih56QvjAV5YnQGVv/zLVWnPPSfTL36xZWGhBx6AYKAn9lFgZQUSAFVx1CykwCMLKsADZAIgLJQ8CoA38AS4AvTyLICsIANRwQjg1q64b/10X236HPAUEgHsIySAiP4aJytmq12M9Qe4BkxZdwFLAAUQ03Y19QowBWwAPuFEiJDxA2pIgfuzwOo3y63v9UHMvd9AUNUCLkcFGNV/IJHF3Ny4D+8DUhMgjNrGHgVVzwvZOPRTO+4RJXsBPNfzaphD4TjGVJUFwuJa42LNBvSBWtZwcmahBrSRBlZ1YI53KcqZIh2s1eYFMDd3rnM96zqwCpzylERITixB1yA9yKL7GD+vGeDqfgArQiQ0RrtCkcgVaRA2hFQI50GqXOsaB/IR4Xb6Tr/IPTw4wnnoPKu3ts0DYAzIRtUx3pGmtDtzELtimwceH2TSOBCjqJAF+BrXJZdMFSTsgAOWFP3yd33/Ce3oi3HHeqJP+stCj0AhLPTSOqHrvCT0nNeBriKMQLX5MIc8EkKiyAbZi8Ro6986tR7JkRfA+XTOudZjHJ4DyJF17lwGBzJCrhEGpML31rexIc2eF8iWCmfuKUlcO8ZozfDMVUk+gkf+xkOXrVWGDoYFuupAMhEvc01O5hLpQbKFEfoxb9r2XHBv46H3T37yiqIy1+5O7nBMzEDQC/i47rrr0vnnn58OPfTQtE23xbC7HK8644BSDoXqUmD5tLGWgPUidDDv+zLWajBWgweOhFOIF/eSB1yAGGDDi1k8NysxIAp4eqH7jrVxn33Wpcc/fiJtssmmheUQSKqXWgW8vKF9ztLNCg24i48HegFt9wW8gBIABIDicQBAWYvb4QCgDeirhpMAsizcPAmssEgRoCcOHpirWoirkw3MAKzi1+WFAGdAKUBat1gbC4DO4g3cIEessUFEgD5WUlZhFXyMCxBrtUM3azzwLzY99vlwrXYQItbjbnf3ro4JsAfKkcOIoedBMoetCBZCxNLLy8QSrd/m1zwZKz0AoFm/kT5zjBQBccgA0GnO/G0MZEi2QGMknYeFOfrK6i28CqD2IxTHNcA5UkYfjEPfhEyxrCNoSAYQTHZAL0IhxEi/EEO6QH48J/qkXfMpfA35AsbpDvIpnAZRJCfjBE6BbXPbbgNAYyAD90dI9bOacAwMW0df+MJM0dbf/M1E4RWo61R13oIwhkcL8ReiBbDzGvCEkJl7GiN5+nE+L5++018eAF4r50eIXXU/D/esblKJ0JsH/SeL2LCRrps7a1ZfrFE6YNyIgPVMz61HYzRmgF9IlxwgBBWxMAYEjMdC7kf1QFAYGIRcmaPY7R5h4cl0mF/PIm3x9vDOME5Yj0Kr6I41jowgU3TS3BsLYvGqV2ViMfALLjhX3nl7YFHmBsZAAnm9jMEk5yE+RAJAhVABoFtIBdCAXAAJgJsXOWAHBACUAB4rKdD55CeXQMn7pdUO2tWbASFCR4RysMq7l5wIIAWAi/KovBzAofuwFlct9fXp02ZUK4rvWHqRJcDLtazkQjeAT2CuaZfiAFjaA9RKC3NpkY0E7+q9gVLkA4Bh3Y/KQuTIuhrVl8iOFRkod99W9+adYT0HApE3cwKIszYDvEK/mkK4Oqkz67mfCC9i9Qf8EY4mK7zxA+8AIc9F7Jwce4i4XyTFs37b/wSIREb1l6wQN2SEzgB1xgWEml9hSOZEaFIVrMc+IcAhIBn7GzgnvFDmxHyQrfl0HwAUMdAu8Is4IJWArANYjXAolmwJ5uaHvI01dq2m8+RsDDwXQvT8kHurOavLXpvmjVeketBH4WRIEJkDuYhmK2JXb5fsI7EcuYxwQesISEeKkH/gnpcOWQ8iyXOG2EVJ59ipvpXe0NfIu0AMrRf3Mb9ICoJE7v5HoHhThJ2Fly48aeENFcJnTfhNDogXmZuD+qaK9MgzCKnloaM/1mCUn3Vf82xs1kJU6pIHRKeQCOTHeeRAL+gjHeAh23nnFenggzOx6PTM6Pj9+vXri/KZudxsR1HlE7IEUl4vWQnGUQKAIqDAUgmMA8IAC1BVJQzOA0qBJGFTm29els/sxSMOeER9euDDAUwIrdEmogGkC7dgda6HIHU7P+6hv37zkMSmak2lK+tt6qMxIhj61AQAnQO8kBnLqDAvQEk8ufMBYueE1b2TxRvoEp8PJLqn/gKi5MBCC1C28rS0kwlQiajx3Ogf4uIeAHQTsSCvCEVjVUYA632PqjxIilAo4S2ISrV6kr6StXHEJnfGFZ/X2wTII6EcgEUsAtAjAAiWUBleLIBRH8ITZvzOASCRIiBWGA35C4MRJkd+CKAQHHJwvqOavI9YIEE8XMC7PnZLKoKY+l0fW/TDuBzCjQDhXtp2HR1xTcwb3aLbSAOPBDIlnA8wRzSch9AhSggZDxpjQTtPiTXpWh4oJMz8y3+J/CrfIY1IB0+EMSFyPAdITBzkiwQgIoB/lNTVfmzEWNfb2C3cHAqlE+rEO8VrxnNZ7lpekgQeH2uNDvAURtW2CN8Kg4P7kYmk/l12WZEOOywTi26foY3nsb6uWbOmCOvIG+QNJMp88ZhIwHpBwnNO0phMeB5mIQHAB5DgIYja95Gz0AS6AyyvXXtf4a0YtNgBEKDNKiCvht4MOk1VMtNtWwFc24E//Q6gFPH2/XgVok9koB1t+FsYUVQZarebd7sxAX0AJVAN8AKJrP1AYSui0s3YnRNyRSx4nYSBNeUNRGJ33UJd7XeQFWSKJwQYrZcYdU67+Yg+hawCqAqFkuMhDyE2gazLzLUIBVArMThi+rvVl3bnhcfhlFPWF4TmiCM2eVDXB20/5oDHB8BGupAnlntHJNoLceIJimIJ7e7LMyRcTrgjIsI76fkgGRw5RCyiIAPPltAl7Xp+1OdM/3rVXSSTrvKE8FzycvDCyONAEJEHnimJ+jwsxu17fUFCqof787Q6F7H467/OxGIgnUMs7r777gIkDfrgH6gj+eIsgQUiAesFEd+yn2DmBTLG3M0sgboEgB0hK0KG4ndUdWknrXvuuacg4o+Iki1ZtEOVQD+EqN4BVb8QFNZdJAhRAdok0/fjAWkaYDWJeVABRChNJw9PL/fhNTHWTqFH3RCqXu5bP/fOO1fJaEjbbLPVIM00XsszA2DzNsrBEBYVR3VX7G7kqkKU/AieH88BxIHeCINEIHhCtMmDQq4Ii98I3DD3ljUfQjF5w3g1hXGpGCWnQjgZsizXwrhVtEM46nkbZECneHG2335F+su/zMRiIOVDLNauXVsApZyMOpAo88VjIgHrBVBSojkfWQLjIgEvcJZNAEEOhATNdknOIRfFDngsskd89jSlmtvQz10kwwpzkyvBciuXgDVeaNSwiIV+DdrPfsbW7TXd5P9029Yg51kvjtnwiAexUJWMNV94Xhy9jl+Im+pevDxCjITT8XLxZEQoYbVkkns7lC7uhrj0IkMERmiekCiHnB4leOVkIU/C44xZeBmPRn1jv7iXUMXbbluRlizJxKIX+TeeOzU1VTz45VnkI0sgS6C9BPJ6yRoyrhJg1VPGUaKmkIduwhfyepn/2iK8B1BkyWX9BdSEA6kQlGHBaOdvemNix7DxGJAPOMuxMK/yU3gW+j3ojFwcHi4ERY4Mb48KVE1Hr8Sll35FcQl5FcKeeB54UJSx5bkQAibPSZlZSfPtdHrFihVp9epMLHqRf+O5GzZsKEjFsBV54I7lBrIE5qEEMlCah5OSuzQyCfQaepPXy8impu8biYVXvUp1HMm0qngpz8q6m4lF32Lt60LrxTHsYjoRMgRwIxeqXw3ijVIwQE6GAgbCjyRMK+TAyzVXByJhrxO5N4iypHE5M8aOr3VTlCETiyHMXoRCqdiRQzuGINDcxKKXwH333VdUuMnrZdFPdR7gECQgdJDRKodCDUGYs9SEBFhlWHkqYndzwFP4SCYWsyT0Fs3OZiiUW5rjdiWNux2tfAYJ+So/qfbE+3capxEAACAASURBVIFgHHhgty0M/zz5GwoPyCGRV2GvGVWiejkysehFWi3ORSzuuuuuIp4v7yQ8BIHmJha9BHLy9qKf4jzAIUogJ28PUZiz1BQwJlREGIlYdUBRGVk7Gg87Hn6WhrBomr1XokLiPRp+8vYwhSRnQm6OPAZAXmlXCdII6VwdUalKOWqOH+Fe+tXLMRJi8Zvf/CbZ9fqSSy5Jr3vd69LjFD6uHGKxrrzyynTOOecUeQr7779/etazntWyAgalufDCC9P3vve94vyDDz447brrrh1B/WztvJ3LzfaicvncLAHx5WsKj0W2wGZtyBLoLAEePh6L2UhG7Xz3fEY3ErDviD0M7ANhXwBgzKZ+di3OxKIbCQ7vHB4+x3yv0qmQg30rVJd6/OPLvAbeCqR0rg6b8dFjIpRLobSskL5ejpEQi8suuyx99atfTd/97nfTWWedlXa333vl+PGPf5wuvvji9J98iSmlZzzjGemFL3xhepo0+YZDe0jKtddeW3z7ghe8oPh5ihXc5pgtYuGWecOvXtQunzvuEsjrZdw1II+/FwnI4WNEG3bMeC99yOe2l4A9AMTLy7VAMpSdFSvPjpqJxWi1x3px9LKh5Gh7WN5N0rY8C3ud+H344eXu5XZLn6tDZTO5Qsph26vCvhUdoPXDujoSYvGNb3wjnXrqqen73/9+QQiWL1/+kI6cdtpp6cYbb0x77713Yv3nvdhhhx3S62Ov+Vq3TzjhhKK865577lmcr83nP//56eV2GpojYqEKgQe/n3xkCWQJtJdAXi9ZQ7IEupeA95wjv1+6l9moz7SbtQo/El7twvyKV5R7WNj5OR+jlcBCWi8qiCnlKiHcfhavfW3rUq6jkKINGM89N6Uf/ajcv0I4n837ejlGQiywx/PPPz8deOCBjcTiuOOOS3feeWf62Mc+Vrh7jz322CKp86OySBqOww8/PG2//fbp7W9/e/GgPeyww9Kzn/3s9NbYArGFBGbTY4HoYMd5H4te1C+fO64SkFzH+prXy7hqQB53LxJYt25d8a7LxQ56kdpoz7Uvic3OTj+9tD7bldnu1oPsED7aESyeuz3wwAOF0XkhhNrGLuYqQtkbYq73jFUK2w7ycj+UnlURqpa90FFRRkIs9OKCCy4ocieaPBZve9vbip2reS4cb3rTmwrQ8elPf7pxAAjKzjvvnD6wsdjvfvvtl5773Oem9zAPtDlmi1hQYMl1lHi+x/R11Ih8QpbACCRgvSAVudjBCISdb7HgJbBq1arC6JZ3qp+/UylpG7FAJuzPJm7+bW/LFaHmYsbk7ToW0k71EqUFvMx1BbEbbkjpS18qN+s74YSUnvOclLbdtrdZnBfE4sgjjyyIxRdljSRxZocXxOKUU05pHM0BBxxQ5FN8RGBaEsv4kiIUiuejfvCWXHPNNcVmHZLIV65cmV75ylembbfdtvAweFg7J2LygB2fx2dCNuIzf8fnzvHDiuQz/Y+qUHG9OHJHnOt6n/kdn/ne9T6Pz6JPEYcen8e9EZlqmz5Xt7neps/JsT5Ofa632WqcMfboZ6s2fd+v7GIPkGqfsuy607uFKjvrxcEC27S+st499NmU12z5vI1n8Lg976oei/yumHzI+2u+vCtWrVqa/v3fl6SjjrIx3oZ08MEz6TWvKd+1nfBEft4N93lnvZA7YjEMfLdQ37P96N1tty1JZ521NJ1++mT63Oem0m67rU8qRPWC72699dak4EQ9n7oJ0E/MROBab+SlOLudx+Koo44qgPnpp59euK+OOOKIQik+o+5Vw4EY7LLLLoXHwvk8Fs973vMaPRZcYt/+9reLUCvK5iHkfMSCh8F9fO48h8+AHef53AvMZ3681HzmJz5DAvxPiASPXPitPZ9X2zTJPvM77qP/ca77+lyb8VnULvdd3DtcfM6NNvU3+uSeca6++BzBqLYZ94rPjC36ZMw+9xOfATZxblTzGZbs4iEQIDPLrnu9W6iyUxWKztKrrHebFc+JvGbXPbhXQ37ePfRdEbkV8Q7K74rSIDGf3hXr12+Wrrpqk6KKzr77PpBe/erptM8+5fMtv2dL7DUqjBK4Cx4bBr5bqO/ZfvTuvvuWprPO2iydfPLSdPbZD6QddoB3e8PGv/zlL5Nw5zklFu9973uLfSBOPPHEAsgLaaKAkrSbDiVrn/SkJ6Wjjz66ON//Er+FVDUdwYd++tOfFrkef/VXf5UeM8g+7H0Qq3xJlkCWQJZAlkCWQJbA4pQAO6LCln/+5ykdfXRKBx2U0o47Ls6x5lEtXgkItDnjjDKkTyL39tv3Hp41L0KheCpuuummIk8CUbjiiiuKqlBvfOMbG2fv4x//eMGG9tprr8Ji/4Mf/KAoN8uT0e6YrRyLsDZG2NHiVbk8siyB4UiAJYU3br6XAxzOaHMrWQKDScB64bXIxQ4Gk+NsXq3C6VVX8VakdPLJKe2/fypCSPIxeglEeFxeL73LXjL5r36V0s03p7TXXiktW9Z7ueSREAtlZpGHs88+Ox1yyCHp0EMPLUhEHD/60Y/SpZdemvzm6lVG1j4WUZb2y1/+chEr91K1uJIEqR8WP9dff31xvopQzt/NXuhtjtkiFjwikuu4p/MGRr0rcr5i/CRgvSAVudjB+M19HnHvEpAjiIjnYge9y25UV0xPl4DsC19I6YADyp2UlywZ1d3zfaoSEGrryMUO+tML5MJPv4nkIyEWZ5xxRjrzzDMfHOEb3vCG9ApFnjcelODf/u3fHkzeftWrXlUkY0dG/4c//OG09dZbF9WiHEDJd77znfTNb36z+F9Oxh577NHxoTubxCJXhepPgfNV4ymBXBVqPOc9j7o/CeSqUP3JbdRXAWM8FwhFv6Bs1H1ejPdbiFWhFtM8jIRYzBeBzRaxML68j8V8meXcj4UggbyPxUKYpdzH+SKBvI/FfJmJ3I+FIIFq4ZyF0N/F1sdMLIY0o3kn4SEJMjczFhLI62UspjkPckgSyOtlSILMzYyFBBbSztuLcUIysRjSrMaeExK485ElkCXQXgJ5vWQNyRLoXgKKlEjezu+X7mWWzxxfCcS+ZLk4yNzoQCYWQ5A7dixPRPK2n3xkCWQJtJeA9RL7oWRZZQlkCbSXgH2SJG/n4iBZU/5/e2cCZFVx9fGjSb4szowRxbglKhESiVTFRC0RBHGhXEKJmigxbhjjrnGJihqMS4xlGUvjVmrK1Je4lQajSVxQ4l5GIUq5QDRgDAEVI+jMIJ+EkcGv/k21Dji8N8yc9/rdvr+umoK5c++5t3+n+977v93nNASqE9DUdBWSg1RnVYs9EBYOVCUstA5HXHnbwSQmIJA1AfUXiXCydmTtZirnREDJDiQsmslf6kQUMzkTWLRoUaheS0tLztVs2LohLBxcI2ERV95mxMIBKCayJ6D+omkdWgiTAgEIVCagL7ASFvQXWgoEqhNQchAVRviqs6rFHggLJ6qa06cbv34oEIBAZQJaCFNzxukvtBQIVCeg/qJCjEV1VuwBAfpL2jaAsHDiz4uSE0jMlIIA/aUUbqaSTgToL04gMVMKAsqipsKHqzTuRlg4cNdUKA1Va/l4lpB3AIqJ7Amov+jrK1MHs3c1FXQgoKkdGuFjKpQDTExkT4CpUGldjLBw4C9h0dbWFubzkYXAASgmsieg/iIRTvB29q6mgg4EFLwtId7U1ORgDRMQyJuAVqpXIdlBGj8jLBy4S1hoCXl9feWLkgNQTGRPQP1F6WYJrsve1VTQgYDSM2taBx+uHGBiInsCSg6i8oUvfCH7ujZiBREWTl5hwS8nkJgpBQH6SyncTCWdCKi/aCoUC345AcVM1gRYIC+texEWTvxZQt4JJGZKQYD+Ugo3U0knAvQXJ5CYKQUB+ktaNyMsnPgrWEhfk/ii5AQUM1kTWLp0aZjaQbKDrN1M5ZwIqL9oxIJkB05AMZM1gY6OjlA/+ksaNyMsHLhLHWulR8VXMGfcASgmsieg/iIRzhzY7F1NBR0IKBhVQpxkBw4wMZE9AcXwqZDsII2rERYO3CUsdOOXOkZYOADFRPYE1F8kLAhGzd7VVNCBgF6UJCwQ4g4wMZE9ASU7UEGIp3E1wsKJu4aqlQ6QqVBOQDGTNQGmQmXtXirnTEBTOzQViqmDzmAxlyUBpkKldSvCwom/Ri1046dAAAIQgAAEIAABCECgjAQQFk5el0LWiIV+KBCAQGUC6i+a2sEIHy0FAtUJkG62OiP2gEAkoP6iwghfmjaBsHDgzgJ5DhAxUSoCLJBXKndT2T4SYIG8PgLk8FIRYIG8tO5GWDjwl7Bob28PWaEIRnUAionsCai/6GsSwajZu5oKOhBQFjWNhhOM6gATE9kTUHIQlebm5uzr2ogVRFg4eWXJkiVhWgdDb05AMZM1Aa37oqlQ5BnP2s1UzokA/cUJJGZKQUDJQVT0sZdSfwIICyfmy5cvD8HbBHA7AcVM1gToL1m7l8o5E1B/UZEYp0AAApUJ0F/SthCEhRP/ZcuWhZs+N34noJjJmoD6i0Q4yQ6ydjOVcyJAf3ECiZlSEOjs7Az15PmSxt0ICwfuirFQsJCmQTG1wwEoJrInoGBUTR1kqDp7V1NBBwJ6vuijFQuwOsDERPYENDVdhZjXNK5GWDhwl7BobW0NN32CUR2AYiJ7AuovEuEEo2bvairoQEDJDiQsCEZ1gImJ7Ako2YFKS0tL9nVtxAoiLBy8woiFA0RMlIqAvsBqmJoRi1K5ncr2koC+wGrqICMWvQTIYaUioGQHei9jxCKN2xEWTtyJsXACiZlSEGDOeCncTCWdCDBn3AkkZkpBgP6S1s0ICyf+asgaqiYrlBNQzGRNgKwdWbuXyjkToL84A8Vc1gToL2ndi7Bw4K8hNw29sY6FA0xMlIKApnZoKhTJDkrhbirZRwJ6vuijFVMH+wiSw0tBgHUs0roZYeHAX8Kira0tzH9lTp8DUExkT4CVt7N3MRV0JKBgVI2INzU1OVrFFATyJMDK22n9irBw4C9hsXjx4vD1lS9KDkAxkT0B9ReN8BGMmr2rqaADAaVnlrDgw5UDTExkT0DJQVTI0pnG1QgLJ+4dHR1hagcLsjgBxUzWBD744IMwtUPiggIBCFQmQH+hhUCg5wSUHEQffLW2GKX+BBAWTszViAncdoKJmewJqL+o0GeydzUVdCBAf3GAiInSEKC/pHU1wsKJfwze5gusE1DMZE1AwXWa2sEXpazdTOWcCGhEXCKc/uIEFDNZE1B/USE5SBo3IywcuEsdK7hO8RXMGXcAionsCai/6CWJOePZu5oKOhBQTJKEOHPGHWBiInsCiknSexnJDtK4GmHhwB1h4QARE6UigLAolbupbB8JKMuN4vcQFn0EyeGlIICwSOvmhhAWCxcuDF/8u5bm5mbr379/t3Teeusti1H/cYf11lvP9FOpzJw50yZPnmzjx4+3fv36uZJnKpQrToxlToCpUJk7mOq5EmAqlCtOjGVOgKlQaR3cEMLinHPOsZtvvnklEuPGjbPLLrusWzpHHHGEPfzwwyv97dRTT7XTTjstmbAgWChtQ+bsxSJAfymWv7jatAToL2n5c/ZiEaC/pPVXQwiLZ5991mbPnm1ahr21tdWefPJJ22abbWzixInd0hkzZkwIZPv+97//0d+HDBkSjkk1YkG62bQNmbMXiwDpM4vlL642LQH6S1r+nL1YBEg3m9ZfDSEsIgJNb3rkkUfs3//+tw0YMMD22muv1QqLwYMH26WXXrpG9Go1FYoF8tbIDewMgbCgJAvk0RAg0DMCLJDXM07sBQERYIG8tO2gYYRFZ2envf3222H604gRI8LP6uIgNGKx4YYb2hlnnBGyymywwQbhX2XNWLXopV+jIFKw//znP+3pp5+2Qw45xNZff/2wv0Y+NFLSdehM2/V73B73k+24veu2ONKijFDrrLNOn22uek19uU7Z6u74uC3WR3WL+1ar++ps9vT47nh62YRd79tyPdmpT0pYKBi1p+3Gq4107cdeNuvJjj67oo2vyX2k6Pc7BW+rDnq+6HkT60O7K8b9jj5b3z4rIa6+0dLS0uf3u+g7nhUr3iWrvR9qnzlz5ph8oJlE1cpaH8a372p79uLv7e3t9sorr9ikSZNs7NixNnTo0G6FgkxLWDz66KO22Wab2bbbbmvHH3+8ffOb3zQFfK9aFCR6++23B9GiohcZjYREMaKUlwq8XrJkSWiIEgf60dQmbZcgidt0Q9c2/cRtgqjfdeNXzmTd+PVv3E/nXNWmhrUlhLRdSOP5437K/hG36f/aV6lso01dR9xX16ftut64TeeM++patF0vcXGbrjnuG7dJ2MVrijbFLm4Tm3hNMU2oF7sY+K5rgt2atbuislOyhhiQSrv7XLg30WeXhMxH3O8+fv7EZ0V8rul5oWcVzwqeFWv6jlLUZ0Vv3lHie5fexzze78rEzuP9bv78+ab3x+TCYvr06XbnnXfajjvuaNtvv71tuummq5UnU6dOtTfeeMPa2trsxRdfDC/nEgt77rnnJ47RS/jcuXNDJefNm2cvvfRSiM3QiIUeYnrJ1ku1flS0TT86TtvilzFti0pN2+OxeuHW73oZly29wFezKdvxPDpnPH9Xm/H8st/1OuMoStyma9Lxq7Opa9G+8cuW9o3KW9vjsbKzpjar1bOn7HTtq9Zzdf6A3Yo2G9tdUdmpv8S2SLv7VLj3xPsAffbj+zL3uxXPirhCfbzf86yo/uzmWZHHs6I37yjxHUfvYx7vd0V9zvaGXXz+9OX9TuEMmo6WVFjo66ViKyQsFLCt+Ap9oa9UBOzdd981CZIrr7wyCIuTTz654jG1irHQSaOw6G46Vi8GcDgEAlkToL9k7V4q50yg64cWZ9OYg0B2BOgvaV3aEDEWM2bMCJmgZs2aZUo9u7r1K1ZFpZcTTaE6+OCDbZdddrGzzz47ibDQVySpMw0hsYR82gbN2YtBQP1FX12rfUAoRm24SgjUlkDXqbq1PRPWIVB8Apq6pKIpg5T6E2gIYXHLLbfYyy+/HKZA7b777j1ehn3BggU2ZcoUu/XWW22fffYJsRaVSq1GLGKAuBoxK6PWvxFzxuIRUPB2jEkq3tVzxRCoLwF9QNNoeHdxhPW9Es4GgcYnEBdcVvA2pf4EGkJYXHTRRaZgjwkTJtjGG28cvvx3LRdffHFYVTsKh/POO8+mTZv2UbpKZZDaddddk61jIWGhCHi9KDFiUf9GzBmLR0D9RfNfGbEonu+44voT0AhfTLpR/7NzRggUi4BG+FSUCIJSfwINISweeuihMJVo7733DqIiBqpFHHfccUcYxdCohIpGOLSgnl5KFIQ9fPhw22KLLUJGpkqlViMWOqeyd8SAuvq7kTNCoFgENI0xBuwX68q5WgjUnwD9pf7MOWNxCcTEO/p4Rak/gYYQFvWqdi2FRcwasKooqlfdOA8EikSA/lIkb3GtqQmov6iQHCS1Jzh/EQjQX9J6CWHhxF9Db1LHq07jcjKPGQhkRUDBdXpJYupgVm6lMjUiQH+pEVjMZklAywuoMNU2jXsRFg7cFWOh4Do1Yub0OQDFRPYE1F8kwkl2kL2rqaADAQWjaqpttem+DqfCBAQKT2Dx4sVh3TGSHaRxJcLCgbsasFbelrBAITsAxUT2BHTj1wgf6QCzdzUVdCCgZAca4ePDlQNMTGRPQDG7Kny4SuNqhIUT946OjpVW03YyixkIZElA/SWuVJ9lBakUBBwJKDmI4vcIRnWEiqlsCai/qDA1PY2LERZpuHNWCEAAAhCAAAQgAAEIZEUAYeHkTgULaQ4sX5ScgGImawL0l6zdS+WcCWiETyMWfIF1Bou5LAloxEJT1EkOksa9CAsH7jHGQo2YOeMOQDGRPQHFJEmEM2c8e1dTQQcCiknS1EHmjDvAxET2BBSTpEKygzSuRlg4cJewUNYOBW4jLByAYiJ7AuovEha8KGXvairoQEBCXMKCFyUHmJjInoCEuIoWVqbUnwDCwom58ozrRYmpUE5AMZM1AfUXTR1kakfWbqZyTgQ0dVBToZja4QQUM1kT0NRBFfpLGjcjLJy4a9RChZW3nYBiJmsC9Jes3UvlnAnQX5yBYi5rAvSXtO5FWDjxV7CQhqr1FZYCAQhUJkB/oYVAoOcEli1bFnZmRLznzNizvAToL2l9j7Bw4C91rGAhTetggTwHoJjInoD6i0Q4MUnZu5oKOhDQgl8aDSfZgQNMTGRPYMmSJSErFDF8aVyNsHDgrgbc1tYWXpK48TsAxUT2BNRfJMQJRs3e1VTQgUB7e3sQ4gSjOsDERPYElOxA72UtLS3Z17URK4iwcPCKGrAUsl6UCEZ1AIqJ7Amov+hFieC67F1NBR0IKNmBRiwYEXeAiYnsCai/qDAinsbVCAsn7p2dneHGrzgLCgQgUJkA/YUWAoGeE6C/9JwVe0Jg+fLlAQLvY2naAsLCibuChdSIachOQDGTNQFelLJ2L5VzJkB/cQaKuawJqL+okEwnjZsRFg7c41QoZexgaocDUExkT0DBqPSX7N1MBZ0IaOqgPloxFcoJKGayJsBUqLTuRVg48JewaG1tDfP5yELgABQT2RMgeDt7F1NBRwIEbzvCxFT2BBYtWhTqSPB2GlcjLBy4k27WASImSkVA6WY1YsEX2FK5ncr2koBG+DRiQTBqLwFyWKkIaIRPhSydadyOsHDizoJfTiAxUwoC9JdSuJlKOhFQDJ+SgzBn3AkoZrImwAJ5ad2LsHDirywEuvHrhwIBCFQmQH+hhUCg5wQ0Kq7C86XnzNizvAToL2l9j7Bw4q+hN03tYB0LJ6CYyZqAguv09ZX+krWbqZwTgaVLlwZRQXIQJ6CYyZpAR0dHWCCPqbZp3IywcOCuBqzgOjVi5vQ5AMVE9gTUXyQqSHaQvaupoAMBrSSsGAtWqneAiYnsCSxevDjUkZXq07gaYeHAXcJCN359TSK4zgEoJrInoP6iET6EePaupoIOBPSiJGGBEHeAiYnsCSg5iApCPI2rERZO3DVUrakdelmiQAAClQloqFovSvQXWgoEqhNQf9FUKKYOVmfFHhBQchAV+kuatoCwSMOds0IAAhCAAAQgAAEIQCArAggLJ3cyYuEEEjOlIMCIRSncTCWdCDBi4QQSM6UgwIhFWjcjLBz4E2PhABETpSJAjEWp3E1l+0iAGIs+AuTwUhEgxiKtuxEWDvzJCuUAEROlIkBWqFK5m8r2kQBZofoIkMNLRYCsUGndjbBw4s86Fk4gMVMKAqxjUQo3U0knAqxj4QQSM6UgoP6iwjoWadyNsHDizkrCTiAxUwoC9JdSuJlKOhGgvziBxEwpCLDydlo3Iyyc+CtYSOkzlXKWAgEIVCZAf6GFQKDnBJYtWxbSzfJ86Tkz9iwvAfUXFdKZp2kDCAsH7lLHChbSAnn6oUAAApUJqL/ops9QNS0FAtUJvP/+++HDFQuwVmfFHhDQ1HQVFmBN0xYQFg7cJSxaW1vDTZ+VUR2AYiJ7AuovEuGsjJq9q6mgAwElO5CwaG5udrCGCQjkTWDRokWhgi0tLXlXtEFrh7BwcIyEhb4oaZVHRiwcgGIiewLqL5rWwYhF9q6mgg4E9AVWwoL+4gATE9kTUHIQFUb40ri6IYTF7bffbk8++eRKBHbaaSc75JBDuqWiVGJPP/20PfXUU2He6X777WcDBw6sOuw1c+ZMmzx5so0fP9769evnSlxz+nTj1w8FAhCoTKCzszP0XfoLLQUC1Qmov6gQY1GdFXtAgP6Stg00hLA4/vjj7f777zeJiVh22WUXO/roo7ul89xzz9mUKVPsmWeeCS8nY8aMsREjRthWW21VkWYthQUvSmkbMmcvFgH6S7H8xdWmJUB/ScufsxeLgLKoqfDhKo3fGkZYtLW12W233dYjCldccYVpDt3XvvY10zQkCY2dd97Z9t133yTCQtegoWpNhdIPBQIQqExA/UVfX5k6SEuBQHUCmtqhj2hMharOij0gwFSotG2gkMLi2GOPtU033dSOO+64oEhPPvlk23HHHe3EE09MJiwkjDSfjywEaRs0Zy8GAfUXiXCCt4vhL64yLQEFb0uINzU1pb0Qzg6BAhDQSvUqJDtI46yGERYPPfSQjR492rbccksbNWqUDR48eLUZlg466CAbNGiQnX/++eErjkYqhg0bZhMmTEgmLBT3oa+vfFFK05A5a7EIqL8o3SzBdcXyG1ebhoDSM+sjGh+u0vDnrMUioOQgKmTpTOO3hhAWt9xyiz366KOBgObGbb311mFq09ChQ7ulMnbs2LDPJZdcEv4uQaL9J06c+In9NTd19uzZYarS/PnzbdasWSZhsv7664cvQLpZa5+uwT564YnbdD36XftqypO2K1A7bpOw0e8aetM2iQvtq21dbWqbbGmb/tXv2r+rzbhNNuN59P94rrhNldS+3dmMwX3x+lU/HV/Npq6jaz3j9a9aT9mNCwF6sIt1ijZht6It9qTdFZmd+ovqGNtn7F+0uxX9iz77qY/ugdzvVjx7VOLzhmfFiucfz9n8nxXxObcm7yjxHUvvY319vyvyc7Y37Dze7+bOnRsypQ4ZMqSqslnrw3h3q7rrmu+gG4Ry2//pT3+yBx980LbZZhs777zzujV0wAEH2Ne//nX7+c9/Hm64++yzjw0fPtzOPffcT+zf0dFhd911ly1cuDA0MD2w9957b+vfv38YXdB0jKVLlwZhoKIvqNqu47Rd1xW3qbHGfeM2OUHbJFxkW1+UVmdTqw3rPF1t6vqjTZ1XP7phxm0xJae2x226jnh+2dJ2XW/cpnrEfdWxtL2rzbjQ0upsipOus6tNPdCiTdXPk10UZbpO2K1ZuysqO1232lPXtky7W3Efos+uuA9yv/v4WaE2oWeF7slqIzwreFas6TtKUZ8V3d0Hqj0r4uKr6jce73dlYufxfqePrBqgXgAAIABJREFU+GKWXFjE0Qq9oJ900knhZfi6667rVlgcccQRNmDAADvzzDODWDj88MNDRqnTTjvtE/vHwGq9wGi04rHHHrPDDjssjFjEL0Dap+sXIb1Er26bjtHftI9+4nXHhienrOnx8VzRXqXju7vO7q6paDajL2DXfVus5OMisovBqLG/RL/Xuy0Xkd2a3JtqeR+A3YpnQG/8sabs9BKpc6m/xOeP1/Orlm2kFs+0NWUXr6Fo9YRd7/tX7C8S4H19v/M6vhb+bFSbc+bMMU13bghhIWX5/PPP2zXXXGObbLKJXXrppd0Ki1/84hdBUGiqlL7Y//nPfw5TocaNG1dxqKRW6WbV8JSlSo2YOeNrPlrFEeUjoP6ir0rMgS2f76nxmhNQMKpGmkl2sObsOKJ8BPRSq0KygzS+b4gYC40izJgxI0yLePPNN4No2GGHHex73/teoKJF7fQCorUqVB5++GGbNm2azZs3L+yreIuRI0fatttum0xY6MYfpzCkcSVnhUBxCKi/xKmDxblqrhQCaQjoRUnCAiGehj9nLRYBJTtQQYin8VtDCIuLL77YJk2aFL74b7755nbggQcGEaE4CBVNedJK2THrk7523nPPPXbnnXeGv59xxhm23XbbVW1EtRqx0DVotCUG1KVxJWeFQHEIqL/oRYl1X4rjM640HYGuU6HSXQVnhkAxCKi/qLBOUhp/NYSwUFxFvHHqZUONQV8z46qJii7XXLKYai8GscXGo+lHMYtKJYy1FBZd572mcSVnhQAEIAABCEAAAhCAQDoCDSEs6lX9WgoLiZyYArZe9eE8ECgqAfWXmFa1qHXguiFQLwLKKBhTj9frnJwHAkUloP6iwoh4Gg8iLBy4a7SCBfIcQGKiNARYIK80rqaiDgRYIM8BIiZKQ4AF8tK6GmHhwF/Cor29PcSIsDKqA1BMZE9A/UVfkwhGzd7VVNCBgOIKNSJOMKoDTExkT0DJQVSam5uzr2sjVhBh4eSVuEAeQ29OQDGTNQGtYxHjqbKuKJWDgAMB+osDREyUhoCSg6joYy+l/gQQFk7MlSq366J5TmYxA4EsCdBfsnQrlaoRAfUXlZjQpEanwSwEsiBAf0nrRoSFE38t1KebPjd+J6CYyZqA+ouEuKZ3UCAAgcoE6C+0EAj0nIDWN1Ph+dJzZp57IiwcaCrGQsFCmgZF3mQHoJjInoCCUZUimqHq7F1NBR0I6Pmij1ZKrU6BAAQqE9DUdBViXtO0FISFA3cJi9bW1nDTJxjVASgmsieg/iIRTjBq9q6mgg4ElOxAwoJgVAeYmMiegJIdqLS0tGRf10asIMLCwSuMWDhAxESpCOgLrIapGbEoldupbC8J6Auspg4yYtFLgBxWKgJKdqD3MkYs0rgdYeHEnRgLJ5CYKQUB5oyXws1U0okAc8adQGKmFAToL2ndjLBw4q+GrKFqfVWiQAAClQmQtYMWAoGeE6C/9JwVe0KA/pK2DSAsHPhryE1DbwpGZR0LB6CYyJ6ApnZoKhTJDrJ3NRV0IKDniz5aMXXQASYmsifAOhZpXYywcOAvYdHW1hbmvzKnzwEoJrInwMrb2buYCjoSUDCqRsSbmpocrWIKAnkSYOXttH5FWDjwl7BYvHhx+PrKFyUHoJjInoD6i0b4CEbN3tVU0IGA0jNLWPDhygEmJrInoOQgKmTpTONqhIUT946OjjC1gwVZnIBiJmsCH3zwQZjaIXFBgQAEKhOgv9BCINBzAkoOog++TE3vOTPPPREWTjTViAncdoKJmewJqL+o0GeydzUVdCBAf3GAiInSEKC/pHU1wsKJfwze5gusE1DMZE1AwXWa2sEXpazdTOWcCGhEXCKc/uIEFDNZE1B/USE5SBo3IywcuEsdK7hO8RXMGXcAionsCai/6CWJOePZu5oKOhBQTJKEOHPGHWBiInsCiknSexnJDtK4GmHhwB1h4QARE6UigLAolbupbB8JKMuN4vcQFn0EyeGlIICwSOtmhIUTf6ZCOYHETCkIMBWqFG6mkk4EmArlBBIzpSDAVKi0bkZYOPEnWMgJJGZKQYD+Ugo3U0knAvQXJ5CYKQUB+ktaNyMsnPiTbtYJJGZKQYD0maVwM5V0IkB/cQKJmVIQIN1sWjcjLBz4s0CeA0RMlIoAC+SVyt1Uto8EWCCvjwA5vFQEWCAvrbsRFg78JSza2tpCRiiy3DgAxUT2BNRflBVqnXXWyb6uVBACfSWgZAfKCkWWm76S5PgyEFCyA5Xm5uYyVLfh6oiwcHCJhMWSJUvCixJ5xh2AYiJ7AuovynJDnvHsXU0FHQgoOYjWsVBKcwoEIFCZgJKDqNBf0rQUhIUT987OzvBFiZWEnYBiJmsC6i/qK+ozFAhAoDKB5cuXhx3oL7QUCFQnQH+pzqiWeyAsnOgqWEg3fW78TkAxkzUB+kvW7qVyzgQkxFU0ykeBAAQqE6C/pG0hCAsH/poKpWAhTYNiaocDUExkT0D9RS9JDFVn72oq6EBAUwc1wqc4PgoEIFCZgKYOqtBf0rQUhIUDdwmL1tbW0IhZGdUBKCayJ6D+IhFO8Hb2rqaCDgTa29vDaDjBqA4wMZE9ASU7UGlpacm+ro1YQYSFg1ckLJQOUC9KjFg4AMVE9gTUXz796U8zYpG9p6mgBwGN8ElY8AXWgyY2ciegET4VsnSm8TTCwom7FjDS1A5iLJyAYiZrAoqx0NQO5oxn7WYq50SA/uIEEjOlIKAYC33w1ccrSv0JICycmCsLgV6UyArlBBQzWROgv2TtXirnTIAsN85AMZc1AfpLWvciLJz4a+hN6ph1LJyAYiZrAgqu0+geUwezdjOVcyJAf3ECiZlSEGAdi7RuRlg48NeQm4LrlOGGOX0OQDGRPQH1F4lwkh1k72oq6EBAwaiaNkiyAweYmMiewOLFi8NUKJIdpHE1wsKBuxqwlpCXsCB9pgNQTGRPQDd+jfARjJq9q6mgAwElO9AIHx+uHGBiInsCSnagwoerNK5GWDhx7+joCF+UCEZ1AoqZrAmov+hFieC6rN1M5ZwIKDmI4vfoL05AMZM1AfUXFaamp3FzQwgLRfAr60VcLVE3T/2sLsOS5pvGfSM2zdWu1ohmzpxpkydPtvHjx1u/fv3SEOesEIAABCAAAQhAAAIQyJBAQwiL6dOn2wMPPGAPPfRQGOodM2aM7bHHHjZo0KBukZ9yyin25JNPrvS3o48+2o455piKLqqlsFCwkEYr+KKUYS+hSu4E6C/uSDGYMQGN8GnEotrHs4wRUDUI9JiARiw0RZ3kID1G5rpjQwgLiYoXX3wxxCe8++67tmDBAttuu+3shz/8YbeVlfDQHDr9G8uOO+5o+qlUaiUsYoyFGjFzxl3bJ8YyJaCYJIlw5oxn6mCq5UpAMUkawWfOuCtWjGVKQDFJKiQ7SOPghhAWU6dONX3B3GGHHWzevHl24YUX2vrrr29XXnnlaoXF4MGD7dJLL10jarUUFsraIWGEsFgjl7BzSQmov0hY8KJU0gZAtdeIgIS4hAUvSmuEjZ1LSkBCXKWpqamkBNJWuyGERUSgL/8aiTj11FPDS/rVV1+9WmGxxRZb2Pnnn//Ry4leUrpbnE42FZOhf//xj3/YI488YocffngQLio6Rn/TTzw+btPf4/b4t677dt3WdR2LrjbjOXpjsyfXpH2qXadnPbu7pp5cZ7zGSjxXvU7Yfbzg4ura3er80ejsYl7+2G+r9S/a3Vor3a+q9fnueNbjPtDo7a4n98tGZNd1KlRP76GrtpGe1L0ebaQvz9mi3u96wr4R211v7suN0O5if9F7pMf7He1uzZ4/c+bMMYm7IUOGVFU4a30Ye0fVXXu3g0TFlClTTCMYW221lR155JGrFRbTpk0LMRjbbLONHXHEEeHf7r7mqIHdfffdtnDhwhDwrY6yzz77WP/+/YN40ZxVjZboRUd/iyljNUcvBonHbVrNUfvqJ45O6Bj9LmEhWxqxiDa1Xci0TfvLprbp37hNf4/nj+dRrEbcpv/H7XGbjonbVCddp2zGa4o2tX+8pmhT23TN8fyxPrITt+n/cXvcFuupc8mmJ7s43x52K9rImrS7orJTHXXt6lO0u8+F+0Tsc/TZFWm7ud99/KzQNFu1ET3P1Hd4VvCcLcuzojfvKPpgpT6ifuPxflfU52xv2Hm8382fPz/cp5ILC02NmD17tk2aNMk0zWno0KFBXHRXHnzwQYtDLfq3paXFRo8ebaNGjfrE7npJnjVrVhgJeeutt8I5DjzwQNtggw1CsLWGl2NWKr08xwDsuE0387hN/4/Zq+I2HaNtUVioIetvXbNcxTS0eomSXf3b1WY8V1ebXa9JnUR/63pNcVu0qb/FbYLQ1WZ39axks2s9436xnqpXrI8XO12rbEW7sFsWXiJ60u6Kyk43HbXdmHJWdaXdrciOt7r+RZ/9VGgzavNlu9/Fb3rxPsyzYkVqd54V+T8runvvqfasiM9PvY95vN8V9TnbG3Ye73cKadA7d1JhoZcMvfwr05P+L5HwjW98Y7XpZvXirBuKgrz/+te/2k033WR77723nXjiiRWHSmoZY6Fgoaj0ejdew1EQKA8B9RfdwIhJKo/PqWnvCeghLVFBsoPeM+TI8hDQh16JC2L40vi8IWIsdBESFc8++6xNmDDBNtpoo4qiIqKK04D2339/GzFihJ177rnJhEVbW1t4SeLGn6Yhc9ZiEVB/kRAnGLVYfuNq0xBob28PQpxg1DT8OWuxCCjZgYSFZrNQ6k+gIYTFjTfeaHPnzrUDDjjAtt566x5/xXznnXfs8ccf/2jE4oQTTkgmLOJUKPKM178Rc8biEVB/0YsSecaL5zuuuP4EusYA1v/snBECxSKg/qLCiHgavzWEsDjrrLPs3nvvtYEDB4YvmJp3PXLkSDvqqKMCFWWHWnfdde2www4Lv19++eX2wgsvhKFhzbnVPK7ddtvNvv3tbycRFjppDAxf3WrhadzLWSHQmAToL43pF66qMQnQXxrTL1xVYxLQe6EK72Np/NMQwuLWW2+1J554YiUCw4YN+0hIaERDQ1rjxo0L+1xzzTU2Y8aMoEY33HBD23PPPYMoaW5uTiYsFPOhRkxDTtOQOWuxCPCiVCx/cbVpCdBf0vLn7MUioP6iolFxSv0JNISwqFe1axm8HdexYGpHvbzJeYpMQMGoynJEfymyF7n2ehHQ80UfrZRCkwIBCFQmwFSotC0EYeHAX0FCra2tYQSFLAQOQDGRPQGCt7N3MRV0JEDwtiNMTGVPQMsXqBC8ncbVCAsH7hIWpJt1AImJ0hBQf4mLGJWm0lQUAr0koBE+jVgQjNpLgBxWKgIa4VMhS2catyMsnLhrFcy4yJuTScxAIFsC9JdsXUvFakBAMXxxAdcamMckBLIioP6ioo9XlPoTQFg4MVcWAt349UOBAAQqE6C/0EIg0HMCXVfe7vlR7AmBchKgv6T1O8LCiX8M3mYdCyegmMmagILrlLGD/pK1m6mcE4GlS5eGj1YkO3ACipmsCXR0dIQF8kh2kMbNCAsH7mrACq5TI2ZOnwNQTGRPQP1FooJkB9m7mgo6ENBKwppqy0r1DjAxkT2BxYsXhzqyUn0aVyMsHLhLWOjGr69JBNc5AMVE9gTUXzT/FSGevaupoAMBvShJWCDEHWBiInsCSg6ighBP42qEhRN3DVVragfBQk5AMZM1AQ1V60WJ/pK1m6mcEwH1F02FYuqgE1DMZE1AyUFU6C9p3IywSMOds0IAAhCAAAQgAAEIQCArAggLJ3cyYuEEEjOlIMCIRSncTCWdCDBi4QQSM6UgwIhFWjcjLBz4E2PhABETpSJAjEWp3E1l+0iAGIs+AuTwUhEgxiKtuxEWDvzJCuUAEROlIkBWqFK5m8r2kQBZofoIkMNLRYCsUGndjbBw4s86Fk4gMVMKAqxjUQo3U0knAqxj4QQSM6UgoP6iwjoWadyNsHDizkrCTiAxUwoC9JdSuJlKOhGgvziBxEwpCLDydlo3Iyyc+CtYSOkzlXKWAgEIVCZAf6GFQKDnBJYtWxbSzfJ86Tkz9iwvAfUXFdKZp2kDCAsH7lLHChbSAnn6oUAAApUJqL/ops9QNS0FAtUJvP/+++HDFQuwVmfFHhDQ1HQVFmBN0xYQFg7cJSxaW1vDTZ+VUR2AYiJ7AuovEuGsjJq9q6mgAwElO5CwaG5udrCGCQjkTWDRokWhgi0tLXlXtEFrh7BwcIyEhb4oaZVHRiwcgGIiewLqL5rWwYhF9q6mgg4E9AVWwoL+4gATE9kTUHIQFUb40rgaYeHEXXP6dOPXDwUCEKhMoLOzM8wZp7/QUiBQnYD6iwoxFtVZsQcE6C9p2wDCwok/L0pOIDFTCgL0l1K4mUo6EaC/OIHETCkIKIuaCh+u0rgbYeHAXVOhNFStqVD6oUAAApUJqL/o6ytTB2kpEKhOQFM7NMLHVKjqrNgDAkyFStsGEBYO/CUs2trawnw+shA4AMVE9gTUXyTCCd7O3tVU0IGAgrclxJuamhysYQICeRPQSvUqJDtI42eEhQN3CQstIa+vr3xRcgCKiewJqL8o3SzBddm7mgo6EFB6Zk3r4MOVA0xMZE9AyUFUyNKZxtUICyfuLPjlBBIzpSBAfymFm6mkEwH1F02FYsEvJ6CYyZoAC+SldS/Cwok/S8g7gcRMKQjQX0rhZirpRID+4gQSM6UgQH9J62aEhRN/BQvpaxJflJyAYiZrAkuXLg1TO0h2kLWbqZwTAfUXjViQ7MAJKGayJtDR0RHqR39J42aEhQN3qWOt9Kj4CuaMOwDFRPYE1F8kwpkDm72rqaADAQWjSoiT7MABJiayJ6AYPhWSHaRxNcLCgbuEhW78UscICwegmMiegPqLhAXBqNm7mgo6ENCLkoQFQtwBJiayJ6BkByoI8TSuRlg4cddQtdIBMhXKCShmsibAVKis3UvlnAloaoemQjF10Bks5rIkwFSotG5FWDjx16iFbvwUCEAAAhCAAAQgAAEIlJEAwsLJ61LIGrHQDwUCEKhMQP1FUzsY4aOlQKA6AdLNVmfEHhCIBNRfVBjhS9MmEBYO3FkgzwEiJkpFgAXySuVuKttHAiyQ10eAHF4qAiyQl9bdCAsH/hIW7e3tISsUwagOQDGRPQH1F31NIhg1e1dTQQcCyqKm0XCCUR1gYiJ7AkoOotLc3Jx9XRuxgggLJ68sWbIkTOtg6M0JKGayJqB1XzQVijzjWbuZyjkRoL84gcRMKQgoOYiKPvZS6k8AYeHEfPny5SF4mwBuJ6CYyZoA/SVr91I5ZwLqLyoS4xQIQKAyAfpL2hbSEMJCX/tnzZplTzzxRHgx32mnnWzrrbde7bQizZ97/vnn7bnnngv777HHHrbFFltUVaczZ860yZMn2/jx461fv36u5JctWxZu+tz4XbFiLFMC6i/quyQ7yNTBVMuVAP3FFSfGMifQ2dkZasjzJY2jG0JYvPrqqzZlyhT7wx/+EF429t13Xxs9erQNHDiwWyoSCPfee6/95S9/CfsfeuihNnLkSPvKV75SkWKthIViLCR2NA2KqR1pGjJnLRYBBaNq6iBD1cXyG1ebhoCeL/poxQKsafhz1mIR0MdqFWJe0/itIYTFrbfeanrpHzRokOkl/bXXXgv/l2Dorlx99dX27rvv2uabbx72f/nll23EiBH2ne98J5mwaG1tDTd9glHTNGTOWiwC6i8S4QSjFstvXG0aAkp2IGFBMGoa/py1WASU7EClpaWlWBeeydU2hLC4+OKL7T//+Y+deeaZYQTiiiuusKamJjv//PO7xXzSSSdZ//797cgjjwz7n3POOTZ06FA79thjkwkLRiwy6RFUoy4E1F80TM2IRV1wc5KCE9AXWD3rGLEouCO5/LoQULIDfXRmxKIuuD9xkoYQFqeffrq1tbXZDTfcEG6eEg6aJnHVVVd1S2XcuHFhRONnP/tZ+Pt+++1nw4YNs7POOiuJsNBJibFI04A5azEJMGe8mH7jqtMQYM54Gu6ctZgE6C9p/dYQwkJCQsLi5ptvDjSOOuqo8DVTQqO7Mnbs2BDcfckll4Q/Kx5j5513tokTJ35id2UHmDdvnin9mP594YUXbMyYMfbFL37xoyxOUrZdswhI3Ghb/NHvGoZedVvMAqVj9bf4ezw+lU1BaLRrgl1jtZFa+IN213sfww52agN6ztTz+UO7o93R7ni/0z3H853x7bffDvaGDBlSVeGs9aHeRmpQJCw0h/R3v/tdeEH/0Y9+FEYsrr/++m7PphEKCQtNoVLZc889g7D46U9/+on9JSgUw6GKyqZiIBTkrSEy/S4Bo6+nWgJecLVNP1K82qbridv0/7ivArW1XcfEbbLV1aa265i4r2xqm/6N26JNbdfx2h5txq+68fz6PdqMa2Z0tRn3iyMo0Wa8JtWnmk3tq2vSvrIdbVaqJ+xWbiOwW9GWaXefDv2NPsv9jmcFz9mu9wGesytmefCO8vE7Z27Pig022MAGDx5cVTHUTFicffbZpmDOX/3qV0HlKNZCgZ2XX355txd1+OGH24ABA2zChAnh5VdB3kpRqylVqxa9JCvQW06Lyly245chbVtVL8W1KLqOWPRmW9dr6c3xXUdBoq2u19obm9FO1/U2Gq2etfDHqn6G3Yp2v2ob624b7HrGCXYf3/F607+4363gB7ue35t4VvT+3gQ72HVde2117wO9fY+VeO5JTFjNhMW1115rc+bMCWlmJRTuu+8+22STTeyUU07pVlhccMEFQRiMGjUqCIY77rjDhg8fbj/4wQ+qqiN2gAAEIAABCEAAAhCAAATSEqiZsHj88cdt6tSptmDBgiAslPFphx12sN122y3U+Omnnw7KZ9tttw2/S3goVkKjHNp/4403DutY6BgKBCAAAQhAAAIQgAAEINDYBGomLN577z178MEH7Ze//GUgoJgLrUmx7rrrht+POeYY03ytGFOhQO/bb7/dfvvb34a/X3jhhWEqlFLUUiAAAQhAAAIQgAAEIACBxiZQM2GhuIrFixeHWAiVfv36BZGg6U4q8+fPD4GgGslQ0SiFFkBRwLeKtisYO+7f2Bi5OghAAAIQgAAEIAABCJSbQM2ERbmxUnsIQAAC9SVw44032rRp0+p7UsezrbfeeiHJR/zY5GgaUxCAAAQgUCcCCIs6geY0EIAABGpJ4OCDD7bp06fb0KFDa3mamth+7bXXTD+KzVN2QAoEIAABCBSTAMKimH7jqiEAAQisREDCQouEXnfddYUjo/g6jVYgLArnOi4YAhCAwEoEEBY0CAhAAAIZEEBYZOBEqgABCECg4AQQFgV3IJcPAQhAQAQQFrQDCEAAAhBITQBhkdoDnB8CEICAAwGEhQNETEAAAhCAQJ8IICz6hI+DIQABCDQGAYRFY/iBq4AABCBQZgIIizJ7n7pDAALZEEBYZONKKgIBCECgsAQQFoV1HRcOAQhA4GMCqwoLLWnx17+a/f3vlSltuKHZTjuZDRtmtu66aYiSFSoNd84KAQhAwJsAwsKbKPYgAAEIJCCwqrD4wx/MbrvN7NFHK1/Mllsq8NvsBz8w+9KXEly4mSEs0nDnrBCAAAS8CSAsvIliDwIQgEACAqsKi9tvN/vf/zV76KHKF7PVVmZHHGF25JFmG2/8yX3feeedsHidytprr21aIXvTTTe1//mf/7HXX389bNtwww3tM5/5jC1btsy0//vvv29bbrmlLVmyxN5++2177733bJ111rG2tjbbeOONbaONNlrpRAiLBA2GU0IAAhCoAQGERQ2gYhICEIBAvQnUSljopf9IqQ4z++xnP2v777+/TZw40b7yla/Y2WefHQTDscceG8SFhMMtt9xiL730kl1//fX2yiuv2K9//Wt77LHHbLvttrPJkyfbT37yE/vxj3+MsKh3A+F8EIAABOpAAGFRB8icAgIQgECtCdRKWMyfPz8IBZWlS5eG0YsFCxbYWWedFaYwaURComH33Xe3qVOn2jPPPBP2Pe200+zyyy+3t956K4xwbL755jZhwgQ75phj7PTTT0dY1LpBYB8CEIBAAgIIiwTQOSUEIAABbwK1Ehaa1vTGG2/Y3/72N5s7d65Nnz49CItJkybZzJkzg5jQFCiJhd/85jfW3t5u3/rWt4LQOPTQQ+2rX/1qWLzvc5/7nI0dO9YOOeQQhIW387EHAQhAoEEIICwaxBFcBgQgAIG+EKiVsHj11VfDFKZp06bZwoUL7V//+pettdZa9vjjj9sHH3xgd911VxAdV111lV1wwQX2pS99KYiHzTbbzEaPHm2jRo0KIxUa7RgxYoQddNBBCIu+OJpjIQABCDQwAYRFAzuHS4MABCDQUwK1Ehb33HOPXXTRRXbyySfbbrvtZvfff79deeWVQVj0798/jFJoStTVV18dhIWmRcWpThq12HXXXe2cc84JgdwjR45EWPTUoewHAQhAoIAEEBYFdBqXDAEIQGBVArUSFnfffbddcskldtlll4UA7d///vd222232QMPPGBf/vKX7eGHHw7CQkHc//3vf22vvfay7373u+HyNHKhqVAHHHCALVq0yI477jgbP358iL/oWsgKRXuGAAQgkAcBhEUefqQWEIBAyQnUSlgoGPumm26yz3/+80FYvPnmmyGYe9iwYXbYYYdZZ2enSXxIbChjlH40aqFy3XXX2d///veQmlbpZzVd6oQTTrBTTjkFYVHy9kr1IQCBPAkgLPL0K7WCAARKRmBVYfHII2YPPGD27LOVQWy6qdmee5rttZfZ+ut/cl9ldZK40EiF1qBQDIVGJmbPnm0nnniiDRgwIMRY3HDDDSFYe/jw4WEfFYmK++67z2bMmGEDBw4MNpSaViMXjFiUrIFSXQhAoBQEEBalcDOVhAAEciewqrD4v/8z08+SJZVr/pnPmK2zjllTk9mczaWEAAABa0lEQVSnPuVLSQJEIxrKIvXUU0+Z4jU0TUoB3AgLX9ZYgwAEINAIBBAWjeAFrgECEIBAHwmsKiyWLzfr7DT78MPKhtdaSytqr/jR/z2LAr6VnvbDDz8MU6n2228/22OPPWzQoEEIC0/Q2IIABCDQIAQQFg3iCC4DAhCAQF8IrCos+mLL69g//vGP9vrrr9vaa69tTU1Ntv3224eAbwV6dy0Eb3sRxw4EIACBtAQQFmn5c3YIQAACLgQaUVj0tGIIi56SYj8IQAACjU0AYdHY/uHqIAABCPSIAMKiR5jYCQIQgAAEakgAYVFDuJiGAAQgUC8CCIt6keY8EIAABCCwOgIIC9oGBCAAgQwISFi88MILId1r0cqrr75qs2bNCqt5K30tBQIQgAAEikkAYVFMv3HVEIAABFYicO2114b1Jopa+vXrZ+eee25YhI8CAQhAAALFJICwKKbfuGoIQAACEIAABCAAAQg0FIH/B9jaKSY2zPPFAAAAAElFTkSuQmCC" width="668" />
We also saw a small improvement in GC metrics, but no clearly measurable reduction to long-pole query latencies.</div><div><br />
<h2>The iterations begin</h2><div><br /></div>
Our approach worked only for <code>commit</code>, when in practice the feature might also be useful for <code>refresh</code>, which is like <code>commit</code> minus the <code>fsync</code> for durability in case your computer or OS suddenly crashes. Unfortunately, these code paths are nearly entirely separate inside <code>IndexWriter</code>, so we aimed for “progress not perfection” and Michael <a href="https://github.com/apache/lucene-solr/pull/1155">opened an initial GitHub pull request that just worked for commit</a>.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuWwk_SERIExBMcs3CgtsShgtNmirUBye7gOxwJdoF99-BWZLKg7_hwJGPPBGXgq_lXpL-qmVKWdwX7EebAQj8nzYdJMtGtPqAucxZxfQjPRxKXK-VuwCbsqUc049zdGei3tUGnEyLbkqY/s1024/Random_walk_25000_not_animated.svg.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="914" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuWwk_SERIExBMcs3CgtsShgtNmirUBye7gOxwJdoF99-BWZLKg7_hwJGPPBGXgq_lXpL-qmVKWdwX7EebAQj8nzYdJMtGtPqAucxZxfQjPRxKXK-VuwCbsqUc049zdGei3tUGnEyLbkqY/s320/Random_walk_25000_not_animated.svg.png" width="320" /></a></div>There was good initial feedback there, but suddenly a <a href="https://github.com/apache/lucene-solr/pull/1155#issuecomment-579819741">“do not push!”</a> comment appeared on the PR, because David Smiley felt this new proposed feature <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17025367&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17025367">could be achieved in simpler ways based on an issue he previously opened</a>. Upon further discussion on the issue, we realized the two approaches were <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17028727&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17028727">indeed very different</a>. This shows a common challenge in open-source software development: when a new feature is proposed, and it is similar to another change being considered, how do you proceed? Merge the two efforts? Add both features? Cross-fertilize? Put on boxing gloves? In this case we proceeded separately. Finally, Michael’s PR received further feedback and he iterated, incorporating the feedback, and finally <a href="https://www.linkedin.com/in/mikesokolov/">Michael Sokolov</a> <a href="https://github.com/apache/lucene-solr/commit/043c5dff6f44c9bb2415005ac97db3c2c561ab45">pushed the change</a>, yay!<br />
<br />
Alas, shortly thereafter, Lucene’s <a href="http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html">excellent randomized tests</a>, running continuously on Apache’s public, and Elastic.co’s private, Jenkins build infrastructure, <a href="https://github.com/apache/lucene-solr/pull/1155#pullrequestreview-370072383">started failing in exotic ways</a>, leading us to <a href="https://github.com/apache/lucene-solr/commit/27b4fee074437a892c511898e9e86d62c127f1d7">revert the change five days later</a>. We found the root cause of those failures, and Michael Sokolov <a href="https://github.com/apache/lucene-solr/pull/1552">opened another pull request</a> to try again. This time we <a href="https://github.com/apache/lucene-solr/pull/1552#issuecomment-639524684">tried more carefully to “beast” Lucene’s unit tests before pushing</a> (basically, run them over and over again on a <a href="http://blog.mikemccandless.com/2021/01/apache-lucene-performance-on-128-core.html">highly concurrent computer, <code>beast3</code></a> to explore the random test space a bit). This uncovered even more exotic test failures, which we fixed and then re-iterated.</div><div>
<br />
At this point <a href="https://www.linkedin.com/in/simonwillnauer/?originalSubdomain=de">Simon Willnauer</a> suddenly engaged, with <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17053231&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17053231">an initial comment on the now massive LUCENE-8962</a>, and <a href="https://github.com/apache/lucene-solr/pull/1155#pullrequestreview-370173731">reviewed this PR</a> more closely, <a href="https://github.com/apache/lucene-solr/pull/1552#discussion_r438156235">asking for the new <code>IndexWriterEvents</code> change</a> to be <a href="https://issues.apache.org/jira/browse/LUCENE-9406">split off into a separate followon issue</a> which has now (months later) separately <a href="https://github.com/apache/lucene-solr/pull/2342">been committed</a> thanks to Zach Chen and Dawid Weiss! Simon also questioned the overall approach and value of the feature, as well as some <a href="https://github.com/apache/lucene-solr/pull/1552#discussion_r438153575">specific changes</a> in the PR. I pleaded with Simon to <a href="https://github.com/apache/lucene-solr/pull/1552#pullrequestreview-429836988">consider how helpful this feature is</a>.<br />
<br />
Finally, Simon, frustrated by the approach, and hearing my plea, rolled his sleeves up and <a href="https://github.com/apache/lucene-solr/pull/1576">prototyped a compelling alternative implementation</a>, yielding a more general simplification over the original approach. Simon’s cleaner approach paved the path to also eventually supporting <code>merge-on-refresh</code>, something we considered too difficult on the first PR (more on this later, a little epilogue). Lots of feedback and iterations and beasting ensued, and Simon iterated that PR to a <a href="https://github.com/apache/lucene-solr/pull/1576">committable pull request</a> and then factored out a <a href="https://github.com/apache/lucene-solr/pull/1585">base infrastructure pull request</a> first, and <a href="https://github.com/apache/lucene-solr/commit/59efe22ac29c95f9ba85b7214fcf5e30cc979222">pushed that first step</a>.<br />
<br />
There were also questions about <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17141583&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17141583">how Lucene should default</a>. This powerful feature is currently disabled by default, but we should consider enabling it by default, perhaps just during <code>commit</code>. Until then, brave Lucene users our there: it is your job to choose when to enable this feature for your usage!</div><div><br />
<h2>The last subtle, brutal, scary atomicity bug</h2><div><br /></div>
Simon then <a href="https://github.com/apache/lucene-solr/pull/1552">updated the 2nd pull request</a> to use the newly pushed base infrastructure and pushed it after more substantial test beasting, and we thought we were at last done! But, the computers disagreed: Lucene’s randomized tests <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17141809&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17141809">started failing in a different exotic way</a> leading to lots of great discussion on the issue and finally <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17142197&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17142197">Simon getting to the smoking gun root cause</a>, a horrible discovery: there was a <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17142388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17142388">subtle yet fatal flaw</a> in all of the attempts and fixes thus far!<br />
<br />
The change <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17142395&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17142395">broke Lucene’s atomicity guarantee for <code>updateDocument</code> in rare cases</a>, forcing us to <a href="https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5d43e73">revert for a second time</a>. At this point we were all rather dejected, after so much hard work, cross-team collaboration, iterations and beasting, as it was unclear exactly how we could fix this issue. Furthermore, this was a bug that was likely quietly impacting Amazon product search and our customers, since we heavily use and rely upon <code>updateDocument</code> to replace documents in the index as products in our catalog are frequently updated. Lucene’s atomicity ensures that the two separate operations done during <code>updateDocument</code>, <span style="font-family: monospace;">delete</span> and <span style="font-family: monospace;">add</span>, are never visible separately. When you refresh from another thread, you will either see the old document or the new one, but never both at the same time, and never neither. We take such a simple-sounding API guarantee for granted despite the <a href="https://blog.trifork.com/2011/05/03/lucene-indexing-gains-concurrency/">very complex under-the-hood implementation</a>.<br />
<br />
But, finally, after sleeping on it, Simon <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17142691&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17142691">boiled the problem down to a simple deterministic unit-test showing the bug</a> and had <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17143881&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17143881">an early idea on how to fix it</a>! Simon went off and coded as usual at the speed of light, pushing his fix to a <a href="https://github.com/apache/lucene-solr/tree/jira/lucene-8962?">feature branch for LUCENE-8962</a> (now deleted, how diligent). Many beasting and feedback iterations later, Simon opened <a href="https://github.com/apache/lucene-solr/pull/1617">one final PR</a>, our collective 3rd attempt. Finally, Simon <a href="https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7f352a9">pushed the final implementation</a> and <a href="https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9b1c928">backported to 8.6.0</a>, without subsequent reverts! The feature finally lives! It was first released in Lucene 8.6.0.</div><div><br />
<h2>And then there was refresh...</h2><div><br /></div>
Lucene applications typically call <code>refresh</code> far more frequently than <code>commit</code>! <code>refresh</code> makes recently indexed documents searchable in near-real-time, while <code>commit</code> moves all index changes onto durable storage so your index will be intact even if the OS crashes or the computer loses its precious electricity.<br />
<br />
Thanks to Simon finding a cleaner way to implement the original <code>merge-on-commit</code> feature, <code>merge-on-refresh</code> became surprisingly simple, relatively speaking, and Simon <a href="https://github.com/apache/lucene-solr/pull/1623">opened and iterated on this PR</a>. We proceeded with our usual iterative feedback, beasting tests, and finally Simon pushed the new feature for Lucene 8.7.0. No reverts needed! Though, we probably <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17184735&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17184735">should indeed have opened a separate dedicated issue</a> since <code>merge-on-refresh</code> was in a later release (8.7.0).</div><div><br />
<h2>Open-source sausage</h2><div><br /></div>
This hidden story, right under our collective digital noses, of how these two powerful new Lucene features, <code>merge-on-commit</code> (in Lucene 8.6.0) and <code>merge-on-refresh</code> (in Lucene 8.7.0), were created serves as a powerful example of open-source sausage making at its best.</div><div><br /></div><div>There are so many examples of strong open-source collaboration and lessons learned:<br />
<ul><li>Powerful changes emerge when diverse, cross-team, cross-corporation developers collaborate over open-source channels. If Amazon had built this feature and used it only internally, we might still have this subtle bug in <code>updateDocument</code> impacting our customers.</li><li>Complex projects unfold right under our noses. Features like <code>merge-on-refresh</code> take many tries to complete. Open-source development is rarely in a straight line.</li><li>Good changes take time: the <a href="https://issues.apache.org/jira/browse/LUCENE-8962">original issue was opened</a> Sep 3 2019, <span style="font-family: monospace;">merge-on-commit</span> was <a href="https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7f352a9">finally pushed (3rd time)</a> on June 27 2020, and <code><a href="https://github.com/apache/lucene-solr/commit/8294e1ae2068aa39e91c25cbaabf62afae40a02e">merge-on-refresh</a></code> on August 24, 2020, and finally this blog post, on March 19, 2021 -- 1.5 years total!</li><li><a href="https://github.com/apache/lucene-solr/tree/jira/lucene-8962">Feature branches</a> (now since deleted) under source control are helpful for big changes that require collaboration across multiple developers, over non-trivial amounts of time.</li><li>Iterative collaboration with harsh, raw and honest feedback that sometimes leads to complete rewrites by other developers to explore a different approach is normal.</li><li>Reverting is perfectly fine and useful development tool — we used it twice here! Committing first to mainline, letting that bake for a few weeks, before backporting to a feature branch (8.x) is healthy.</li><li>Complex features should be broken down into separable parts for easier iteration/consumption, especially when an initial proposed change is too controversial. In such cases we factor out separable, controversial parts into <a href="https://issues.apache.org/jira/browse/LUCENE-9406">their own issues</a> which are eventually developed later and <a href="https://github.com/apache/lucene-solr/pull/2342">perhaps committed</a>. Such open-source crumbling can also happen later in the iterations as more clarity surfaces, as it did with Simon's approach.</li><li>Developers sometimes try to <a href="https://github.com/apache/lucene-solr/pull/1155#issuecomment-579819741">block changes</a> because they might be too similar to other proposed changes, until the community can work out the way forward.</li><li>Some bugs last a long time before being discovered! Our initial attempt broke Lucene’s atomicity and we did not catch this until very late (third try) in the iterations.</li><li>When an exotic randomized unit test finally catches a failure, reproducible with a failing seed, we try to boil that precise failure down to small, self-contained deterministic (no randomness needed) unit test exposing the bug, then fix the bug and confirm the tests passes, and push both the new test case and the bug fix together.</li><li>Randomized tests are powerful: given enough iterations they will uncover all sorts of fun, latent bugs. Lucene likely has many bugs waiting to be discovered by our randomized tests just by uncovering precisely the right failing seeds. This seems similar to ₿itcoin mining, without the monetary value!</li><li>New features frequently begin life without being enabled by default, but discussions of <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17141583&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17141583">how the new feature should default</a> are important (it currently defaults to disabled).</li><li>We make many mistakes! Complex open-source software is difficult to improve without also breaking things. We <a href="https://issues.apache.org/jira/browse/LUCENE-8962?focusedCommentId=17184702&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17184702">really should have opened a separate issue for both features</a>.</li></ul>
And of course underlying all of the above is the strong passion of many diverse developers eager to continue improving Apache Lucene, bit by bit.</div><div><br /></div><div>Patches welcome!<br />
<br /><em style="background-color: white; color: #333333; font-family: Verdana, Geneva, sans-serif; font-size: 14.3px;">[I work at Amazon and the postings on this site are my own and do not necessarily represent Amazon's positions]</em></div>Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com0Lexington, MA, USA42.4473497 -71.227153114.137115863821151 -106.3834031 70.757583536178842 -36.070903099999995tag:blogger.com,1999:blog-8623074010562846957.post-11007880824865896172021-01-04T10:31:00.004-05:002021-01-04T11:56:30.879-05:00Apache Lucene performance on 128-core AMD Ryzen Threadripper 3990XAlmost a decade ago, I started running
<a href="http://blog.mikemccandless.com/2011/04/catching-slowdowns-in-lucene.html">Lucene's
nightly benchmarks</a>, and have been trying with mixed success to keep them running every night, through the numerous amazing changes relentlessly developed by the passionate Lucene community. The benchmarks run on the tip of <a href="https://github.com/apache/lucene-solr">Lucene's mainline branch</a> each night, which is understandably a volatile and high velocity code base.
<p>Sure, Lucene's
wonderful <a href="http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html">randomized
unit tests</a> will catch an accidental bug, API breakage or perhaps a subtle
corner-case issue during development. But nothing otherwise catches all-too-easy
unexpected performance regressions, nor helps us measure performance gains when we optimize.
</p><p>
As a recent example, it looks like <a href="https://home.apache.org/~mikemccand/lucenebench/BrowseMonthTaxoFacets.html">upgrading from JDK 12 to JDK 15 might have hurt Lucene's <tt>Month</tt> faceting queries/sec by ~5%</a> (look for annotation <tt>DG</tt> in that chart). However, that was not the only change in that period, benchmarks failed to run for a few nights, and other tasks don't seem to show such a drop, so it's possible (likely?) there is another root cause. Such is the challenge of benchmarking! WTFs suddenly sprout up all the time. </p><p>
Time flies when you are having fun: it has been almost five years since I last upgraded the custom hardware that runs <a href="https://people.apache.org/~mikemccand/lucenebench">Lucene's nightly benchmarks</a>, nearly an eternity in computer-years! Thanks to the fast paced technology market, computers keep getting relentlessly bigger, smaller, faster and cheaper.
</p><p>So, finally, as of a couple months ago, November 6, 2020, I have switched our nightly benchmarks to a new custom-built workstation, creatively named <tt>beast3</tt>, with these parts:</p><ul>
<li> Single socket AMD Ryzen Threadripper "desktop class" 3990X (64 cores, 128 with hyperthreading), clocked/volted at defaults
</li><li> 256 GB quad channel Multi-Bit ECC DDR 4 RAM, to reduce the chance of errant confusing bit flips <a href="https://issues.apache.org/jira/browse/LUCENE-6576">possibly wasting precious developer time</a> (plus <a href="https://www.realworldtech.com/forum/?threadid=198497&curpostid=198647">Linus agrees</a>!)
</li><li> <a href="https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/consumer-ssds/optane-ssd-9-series/optane-ssd-905p-series/905p-960gb-aic-20nm.html">Intel Optane SSD 905P Series, 960GB</a>
</li><li> RAID 1 array (mirror) of NVMe Samsung 970 pro 1 TB SSDs
</li><li> A spinning-magnets 16 TB Seagate IronWolf Pro
</li><li> Arch Linux, kernel 5.9.8-arch1-1
</li><li> OpenJDK 15.0.1+9-18
</li></ul>
All Lucene benchmarks use the Optane SSD to store their Lucene indices, though it is likely unimportant since the 256 GB of RAM ensures the indices are nearly entirely hot. All source documents are pulled from the RAID 1 SSD mirror to ensure reading the source documents is very fast and will not conflict with writing the Lucene indices.
<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3Jnb4IzhebVvTWZOSLEnPmc1MWkbs5xWNLnb3PVk-8D2LrX1a74YyWOOjzLIeIr74JyBXRMEG8lKIKc7tB1q-dVLJCTA6vxg15-1g4RLRIVU03Ql5DgQZbY1ELbMYJdfbHsjJzRQxHDJS/s620/amd-ryzen-die-shot.jpg" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="244" data-original-width="620" height="126" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3Jnb4IzhebVvTWZOSLEnPmc1MWkbs5xWNLnb3PVk-8D2LrX1a74YyWOOjzLIeIr74JyBXRMEG8lKIKc7tB1q-dVLJCTA6vxg15-1g4RLRIVU03Ql5DgQZbY1ELbMYJdfbHsjJzRQxHDJS/w320-h126/amd-ryzen-die-shot.jpg" width="320" /></a></div>
<tt>beast2</tt> was an impressive workstation five years ago, with dual socket Intel Xeon E5-2699 v3 "server class" CPUs, but this new workstation, now using a lower class "desktop class" CPU, in a single socket, is a even faster.
<p></p><p>
Watching <tt>top</tt> while running <tt>gradle test</tt> configured to use 64 JVMs is truly astounding. At times my whole terminal window is filled with only <tt>java</tt>! But, this also reveals the overall poor concurrency of Lucene's gradle/test-framework compiling and executing our numerous unit tests on highly concurrent hardware. Compilation of all main and test sources takes minutes and looks nearly single-threaded, with a single <tt>java</tt> process taking ~100% CPU. Most of the time my terminal is NOT full of <tt>java</tt> processes, and overall load is well below what the hardware could achieve. Patches welcome!
</p><p>
The gains across our various benchmarks are impressive:
</p><ul>
<li> <a href="https://home.apache.org/~mikemccand/lucenebench/indexing.html">Indexing</a>: ~42% faster for medium sized (~4 KB) docs, ~32% faster for small (~1 KB) docs
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/PKLookup.html">Primary Key Lookup</a>: ~49% faster
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/Term.html">TermQuery</a>: ~48% faster
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/AndHighHigh.html">BooleanQuery conjunctions of two high frequency terms</a>: ~38% faster
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/BrowseMonthTaxoFacets.html">Month faceting</a>: ~36% gain, followed by unexplained ~32% drop! (Edit: OK, it looks like it might be due to Lucene's default Codec <a href="https://issues.apache.org/jira/browse/LUCENE-9378?focusedCommentId=17234755&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17234755">no longer compressing BinaryDocValues by default</a> -- we can fix that!)
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/Fuzzy1.html">FuzzyQuery, edit distance 1</a>: ~35%
</li><li> <a href="https://home.apache.org/~mikemccand/geobench.html#search-polyRussia">Geo-spatial filtering by Russia Polygon, LatLonPoint</a>: ~31%
</li><li> <a href="https://home.apache.org/~mikemccand/geobench.html#index-times">LatLonPoint geo-spatial indexing</a>: ~48%
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/TermGroup10K.html">10K grouping with TermQuery</a>: ~39%
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/antcleantest.html">Time to run all Lucene unit tests</a>: ~43%
</li><li> <a href="https://home.apache.org/~mikemccand/lucenebench/checkIndexTime.html">Time to CheckIndex</a>: ~22%
</li></ul>
Most of these tasks are by design effectively testing single-core performance, showing each core of the new CPU is also substantially faster than one core of the older Xeon. The exceptions are Indexing, Primary Key Lookup and Time to run all Lucene unit tests, which do effectively use multiple cores.
<p>
I am happy to see the sizable jump in Lucene's indexing throughput, despite not yet increasing the number of indexing threads (still 36): it shows that Lucene's indexing implementation is indeed quite concurrent, allowing the faster cores to index more efficiently. However, smaller ~1 KB documents saw less gains than larger ~4 KB documents, likely due to some sort of locking contention in <tt>IndexWriter</tt> that is relatively more costly with smaller documents. Patches welcome!
</p><p>
The only serious wrinkle with upgrading to this new box is that rarely, a <tt>java</tt> process will simply hang, forever, until I notice, <tt>jstack</tt> and <tt>kill -9</tt> it. I have opened <a href="https://github.com/mikemccand/luceneutil/issues/89">this issue</a> to try to get to the bottom of it. It may be yet another classloader <a href="https://issues.apache.org/jira/browse/LUCENE-6482">deadlock</a> <a href="https://issues.apache.org/jira/browse/LUCENE-5573">bug</a>.</p><p>
Another small challenge is this is my first custom liquid cooling loop, and I am surprised how quickly (relatively speaking) the coolant "evaporates" despite being a closed loop with no obvious leaks. I just must remember to add more coolant periodically, or else the CPU might start thermal throttling and make everything go slowly!</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com0tag:blogger.com,1999:blog-8623074010562846957.post-22948822696790008822019-10-06T11:21:00.000-04:002019-10-07T13:17:07.179-04:00Concurrent query execution in Apache Lucene<a href="http://lucene.apache.org/">Apache Lucene</a> is a wonderfully
concurrent pure Java search engine, easily able to saturate the
available CPU or IO resources on your server, if you ask it to. The
concurrency model for a "typical" Lucene application is one thread per
query at search time, but did you know Lucene can also execute a single
query concurrently using multiple threads to greatly reduce how long
your slowest queries take?<br />
<br />
Lucene's <a href="https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/search/IndexSearcher.html">IndexSearcher</a>
class, responsible for executing incoming queries to find their top
matching hits from your index, accepts an
optional <a href="https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/Executor.html">Executor</a>
(e.g. a thread pool) during construction. If you pass an
<code>Executor</code> and your CPUs are idle enough (i.e. your server
is well below its red-line QPS throughput capacity), Lucene will use
multiple concurrent threads to find the top overall hits for each
query.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvLd4DSpd7gEJ49tMMxrGRKGt-sd7T25yuKcz1DR-7ZxWm46Agw2U7HgmVhQi20-290OmKe-oOtBXDq3keX2uF4d27Rolh7K3grM-_0WzgFIPfd8cFIHU1GuG46sdy_BCQ3jpJ6fN0Yt_T/s1600/threads1.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="300" data-original-width="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvLd4DSpd7gEJ49tMMxrGRKGt-sd7T25yuKcz1DR-7ZxWm46Agw2U7HgmVhQi20-290OmKe-oOtBXDq3keX2uF4d27Rolh7K3grM-_0WzgFIPfd8cFIHU1GuG46sdy_BCQ3jpJ6fN0Yt_T/s1600/threads1.jpg" /></a></div>
How does it do that? A Lucene index
is <a href="https://lucene.apache.org/core/8_2_0/core/org/apache/lucene/codecs/lucene80/package-summary.html#Segments">segmented</a>,
which makes searching it
an <a href="https://en.wikipedia.org/wiki/Embarrassingly_parallel">embarassingly
parallel</a> problem: each query must visit all segments in the index, collecting their globally competitive hits. When the query is
single-threaded, because you did not pass an <code>Executor</code>
to <code>IndexSearcher</code>, that one query thread must visit all
segments sequentially. If the index is large, and your queries are
costly, those queries will naturally require high CPU cost and wall
clock time to find the top hits. This will cause high long-pole
(P90+) query latencies even when you are running the server well
below its red-line QPS (throughput) capacity.
<br />
<br />
Instead, when you do pass an <code>Executor</code>
to <code>IndexSearcher</code>, the segments in the index are first grouped
up front into single thread work units called <em>thread slices</em>.
<a href="https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L306-L338">By
default</a>, large segments belong to their own thread slice and up to
5 smaller segments with at most 250K total documents will be coalesced
into a single thread slice, since they are presumably quick to
search sequentially by a single thread. You can easily customize how segments are
coalesced into thread slices by
subclassing <code>IndexSearcher</code> and overriding its
protected <a href="https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L299-L301"><code>slices</code></a>
method. Each incoming query is then executed concurrently, as long as
the server is idle enough to spend multiple CPU cores on one query,
with one thread working on each thread slice for that query.
<br />
<br />
This powerful feature
was <a href="https://issues.apache.org/jira/browse/LUCENE-169">originally
proposed almost 16 years ago by Jean-François Halleux</a> and
then <a href="https://github.com/apache/lucene-solr/commit/b1541ce02737a395fa040f23022a4877353c2ec4">committed
by Doug Cutting himself</a> (hello Doug!) and finally
<a href="https://issues.apache.org/jira/browse/LUCENE-2837">refactored
into IndexSearcher</a> almost 9 years ago, and has since undergone a bunch
of iterative improvements, many unfolding now thanks
to <a href="https://www.linkedin.com/in/atrisharma/">Atri Sharma</a>,
recently added <a href="https://markmail.org/thread/syk67t2lxefibw2l">new Lucene/Solr committer</a>. Such is the
distributed power of passionate open-source software development!
<br />
<br />
Concurrent query execution is a surprisingly little known sleeper
feature in Lucene, since it is not yet exposed
in <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> nor
<a href="https://lucene.apache.org/solr">Solr</a>, two popular
distributed search applications that build on Lucene. Their concurrency
model is instead concurrent search across index shards (usually on
different servers) for a single query, but using single-threaded search
within each shard.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr4U-8HNRvFO92D2zDgnYpvFrm7dYhPb6kLhQM5V9FaU5cYfR2RL8xC4x-6_wHCii47Nsr9MDlIe9SbPZ969-vZXlNHedtqy0mDV_0TXWskGHRRkPq4ktxyWRlbORFyep_hxfvXJcRBfnl/s1600/cpu.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="343" data-original-width="800" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhr4U-8HNRvFO92D2zDgnYpvFrm7dYhPb6kLhQM5V9FaU5cYfR2RL8xC4x-6_wHCii47Nsr9MDlIe9SbPZ969-vZXlNHedtqy0mDV_0TXWskGHRRkPq4ktxyWRlbORFyep_hxfvXJcRBfnl/s400/cpu.jpg" width="400" /></a></div>
This means many concurrent independent queries are required in order
to saturate cluster wide CPU or IO resources. Until the cluster sees
at least that minimum floor QPS, the full hardware resources cannot be
utilized. For use cases that often see high query rates, this
limitation is acceptable. But other common use-cases that have a
large index and lower query rate would benefit substantially from
concurrent query execution within a single cluster node if
Elasticsearch or Solr were to use this feature.
<br />
<br />
The real-world effects
of <a href="https://en.wikipedia.org/wiki/Moore%27s_law">Moore's
law</a> have shifted: modern server class computers are built with amazing and rapidly
increasingly concurrent hardware, not just in their CPUs where we now
see <a href="https://aws.amazon.com/ec2/instance-types/#Compute_Optimized">96
cores in the latest <code>c5.24xlarge</code> AWS EC2 instances</a>,
but also in their Graphic Processing Units (GPUs), memory bus and
DIMMs and solid-state disks (SSDs), which are in fact large concurrent RAID 0
arrays under-the-hood. The recent trend is for CPUs and GPUs to gain
more concurrency (cores), and less so for each individual core to get
too much faster. Why not use all this increasing concurrency to make
all queries faster, and saturate CPU/IO even at low query loads?
<br />
<br />
<b>Tricky Tradeoffs</b>
<br />
<br />
Unfortunately, even though searching a Lucene index is a naturally and
embarrassingly parallel problem, using multiple threads for one query
incurs inherent coordination overhead. To understand why, consider a
simple analogy: imagine you need apples, so you send your kids to the
local grocery store to buy them. If you have one only child, you send
her, she walks around the entire produce section and picks the ten
best apples, and brings them home.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjPIRQQU13-K7E-GUBAgiRX5BYeA6BKcbQ8xAL2Q3qUzLEyP93s-mJi_s_NnBr1z9iZQGoLy5svzEM207L1UuBYptG8WHC4XUnHQ0yP5wcZcqQh8p9u0kP8g2ViLhB85Y9_09m9vp36urw/s1600/apples.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjPIRQQU13-K7E-GUBAgiRX5BYeA6BKcbQ8xAL2Q3qUzLEyP93s-mJi_s_NnBr1z9iZQGoLy5svzEM207L1UuBYptG8WHC4XUnHQ0yP5wcZcqQh8p9u0kP8g2ViLhB85Y9_09m9vp36urw/s400/apples.jpg" width="400" /></a></div>
But if you have five children and you send all of them to the store,
will they come back five times faster, ignoring the "networking" time
for them to get to and from the store? How do they efficiently split
the work?<br />
<br />
Perhaps your children are clever and they first split up
all the apple sections in the store (there
are <a href="https://www.wholefoodsmarket.com/department/article/your-guide-apple-varieties">many
diverse apple choices these days</a>!) into five roughly equal
sections. Each runs around their own apple section, picking the ten
best apples she can find, and then they all meet up at the checkout
counter and work closely to choose the total ten best out of the fifty
apples they now have? This is somewhat wasteful, since the
children collected fifty apples overall just to choose the actual ten best in
the end, but it should indeed be faster than one child picking the ten
best overall.
<br />
<br />
This is effectively how Lucene implements concurrent search today:
each searcher thread works alone to find its own top N best hits from one thread
slice (the "map" phase), and then, once all query threads have
finished and joined back to the main thread, the main thread uses a <a href="https://en.wikipedia.org/wiki/Partial_sorting">partial
merge sort</a> to find the total top N best hits from the hits collected for each thread slice (the "reduce" phase). Lucene's <code>CollectorManager</code>,
<code>Collector</code> and <code>LeafCollector</code> abstractions all
work together to implement this. This means more total work is done
versus the single threaded case, since now <code>M * N</code> total
hits were collected and then reduced to just the top <code>N</code> in
the end, where <code>M</code> is the number of concurrent search threads
and <code>N</code> is the requested number of top hits to retrieve.
<br />
<br />
That added coordination cost will necessarily hurt the red-line QPS
capacity (throughput) of the search node, when running each query
concurrently, since Lucene is spending more total CPU cycles finding the
top hits. Yet at the same time, it can greatly improve the long-pole
query latencies when the search node has plenty of spare CPU
resources, since the hardest queries will now run concurrently.
Furthermore, that extra cost of collecting more hits and merging them
in the end is often a minor impact overall since it is usually the
matching and ranking of each hit that dominates the total query cost,
especially as the index grows larger, and that cost is efficiently
split across threads.<br />
<br />
You can further "amplify" this tradeoff by
limiting how many queries can run concurrently, thereby maximizing how
many CPU cores will be used for each query. You could also estimate
up front how costly each query will be and execute that query
concurrently only if its cost is large enough, so that easy queries that would run quickly with a single thread do not pay the overhead of synchronizing across multiple threads.<br />
<br />
This throughput versus latency tradeoff is frustrating, and it means
it might make sense to use a <em>modal</em> approach for your Lucene
application. When the cluster is lightly loaded, use multiple threads
per query by restricting how many queries may run concurrently,
reducing long-pole latencies. But when the cluster is running hot,
approaching its red-line capacity, shift to a single thread
per query, to maximize throughput. Be sure you are measuring
latencies correctly and
your <a href="https://www.youtube.com/watch?v=lJ8ydIuPFeU">load
testing client does not suffer from the all-too-common coordinated
omission bug</a>! Confirm
your <a href="http://highscalability.com/blog/2015/10/5/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html">load-testing
client is using open-loop testing</a> so you see the true latency
impact from, say, a long garbage collection pause, I/O hiccup or
swapping.
<br />
<br />
<b>Ongoing and future improvements</b>
<br />
<br />
Fortunately, there have been some recent exciting improvements to
reduce the added overhead for multi-threaded queries. Lucene
now <a href="https://issues.apache.org/jira/browse/LUCENE-8865">also
uses the incoming (calling) thread to help with concurrent
searching</a>. The algorithm for grouping small segments into slices
(thread work units)
<a href="https://issues.apache.org/jira/browse/LUCENE-8757">has
improved</a>. Early termination
now <a href="https://issues.apache.org/jira/browse/LUCENE-8939">uses a
single shared global hit counter</a> across multiple search threads
for one query, reducing the total cost for the query. Query caching will
soon <a href="https://github.com/apache/lucene-solr/pull/815">use the
Executor to cache concurrently</a> and can even be more efficient in some
cases when an <code>Executor</code> is used. Instead of each search
thread working fully independently and merging top hits only in the end, they
should share information while they concurrently collect such as <a href="https://issues.apache.org/jira/browse/LUCENE-8974">
their worst scoring top hit collected so far</a> or even
<a href="https://github.com/apache/lucene-solr/pull/854">use a
single shared priority queue across all threads</a>. The shared
priority queue may incur too much locking, so as a compromise,
searching
now <a href="https://issues.apache.org/jira/browse/LUCENE-8978">efficiently
shares the best of the worst collected hit across searcher
threads</a>, which showed
impressive <a href="https://github.com/mikemccand/luceneutil">luceneutil</a> <a href="https://issues.apache.org/jira/browse/LUCENE-8978?focusedCommentId=16936554&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16936554">benchmark
results</a>.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghM1_O6oWmqjhosQgPivsz3zFaixNZMI53-D0whGUgxNiylCAQa72GKAqURwuX_j6fe1vcwXCG7aTO-6BZrP0sN9UI8PVyHGF21mSBv4YWoswlvIF2Lu81w83ECKSXzLTr4Qxes_aGW3zT/s1600/concurrency.jpeg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="360" data-original-width="640" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghM1_O6oWmqjhosQgPivsz3zFaixNZMI53-D0whGUgxNiylCAQa72GKAqURwuX_j6fe1vcwXCG7aTO-6BZrP0sN9UI8PVyHGF21mSBv4YWoswlvIF2Lu81w83ECKSXzLTr4Qxes_aGW3zT/s400/concurrency.jpeg" width="400" /></a></div>
These improvements are reducing the extra cost of concurrent search
but that cost can never be zero as there is an inherent natural cost
to more frequent thread context switching, lock contention for shared
priority queues, hit counters and priority queue bottoms and possibly difficult
effects due to modern
<a href="https://en.wikipedia.org/wiki/Non-uniform_memory_access">non-uniform
memory architectures (NUMA)</a>.
<br />
<br />
One curious and disappointing limitation of Lucene's concurrent search
is that a fully merged index, down to a single segment, loses all
concurrency! This
is <a href="https://en.wikipedia.org/wiki/Bizarro_World">Bizarro
World</a>, since normally one merges their index down to a single
segment in order to improve query performance! But when you are
looking at long-pole query latencies, a fully merged index is
unfortunately slower since all queries are now single threaded
again even when you pass an <code>Executor</code>
to <code>IndexSearcher</code>. Even a single large newly completed
merge will cause a sawtooth pattern in your long pole latencies as
it reduces the net query concurrency, though the red-line cluster throughput capacity still improves with such merges. One simple idea to address this is
to <a href="https://issues.apache.org/jira/browse/LUCENE-8675">allow
multiple threads to search a single large segment</a>, which should
work well since Lucene has natural APIs for searching separate regions
in "docid space" of the segment.
<br />
<br />
Concurrent search has come a long way since Jean-François Halleux
first proposed it for Lucene, and I expect it still has a long way to
go, to get the point where we truly minimize the added overhead of
using multiple threads for costly queries. As Lucene improves its query planning and optimization we will reach a point where easy queries run single-threaded but costly queries run concurrently and efficiently. These improvements must come to Lucene: modern servers continue to add more and more cores but are not
making those cores too much faster, so it is inevitable that modern
software, including Lucene, must find ways to efficiently tap into all
this concurrency.
<br />
<br />
<em>[I work at Amazon and the postings on this site are my own and do not necessarily represent Amazon's positions]</em>
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com3tag:blogger.com,1999:blog-8623074010562846957.post-72853156746291684142017-09-04T18:29:00.003-04:002017-09-26T11:59:36.029-04:00Lucene's near-real-time segment index replication<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGU6frtMExi5URaV7B87J8FiBP1hGdgLyVv5u0cV7Gr6d2TqvRMU9Js_EULw-kfF4MPm0HOvB4QGwXDcMrwtP0Rv6odGzZZpbo9Tf-7zyO1YvsTlchVMAIkDVUMJyrKyCrh6QlFpwdbi5x/s1600/0.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="170" data-original-width="170" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGU6frtMExi5URaV7B87J8FiBP1hGdgLyVv5u0cV7Gr6d2TqvRMU9Js_EULw-kfF4MPm0HOvB4QGwXDcMrwtP0Rv6odGzZZpbo9Tf-7zyO1YvsTlchVMAIkDVUMJyrKyCrh6QlFpwdbi5x/s200/0.png" width="200" /></a></div>
<em>[<b>TL;DR</b>: <a href="http://lucene.apache.org/">Apache Lucene
6.0</a> quietly introduced a powerful new feature
called <a href="https://issues.apache.org/jira/browse/LUCENE-5438">near-real-time
(NRT) segment replication</a>, for efficiently and reliably
replicating indices from one server to another, and taking advantage
of ever faster and cheaper local area networking technologies.
Neither of the popular search servers
(<a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a>,
<a href="http://lucene.apache.org/solr">Solr</a>) are using it yet,
but it should bring a big increase in indexing and searching
performance and robustness to them.]</em>
<br />
<br />
Lucene has
a <a href="https://www.slideshare.net/lucenerevolution/what-is-inaluceneagrandfinal">unique
write-once segmented architecture</a>: recently indexed documents are
written to a new self-contained <em>segment</em>, in append-only,
write-once fashion: once written, those segment files will never again
change. This happens either when too much RAM is being used to hold
recently indexed documents, or when you ask Lucene to refresh your
searcher so you can search all recently indexed documents.
<br />
<br />
Over time, smaller segments
are <a href="http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">merged
away into bigger segments</a>, and the index has a logarithmic
"staircase" structure of active segment files at any time. This is an
unusual design, when compared with databases which continuously update
their files in-place, and it bubbles up to all sorts of nice
high-level features in Lucene. For example:
<br />
<ul>
<li>Efficient
<a href="http://blog.mikemccandless.com/2012/03/transactional-lucene.html">ACID
transactions</a>.
<br /><br />
</li>
<li>Point-in-time view of the index for searching that will never
change, even under concurrent indexing, enabling stable user interactions like
pagination and drill-down.
<br /><br />
</li>
<li>Multiple point-in-time snapshots (commit points) can be
indefinitely preserved in the index,
even under concurrent indexing, useful for taking <a href="http://freecontent.manning.com/hot-backups-with-lucene/">hot backups
of your Lucene index</a>.
</li>
</ul>
The <a href="http://blog.mikemccandless.com/2009/07/ive-been-test-driving-suns-now-oracles.html">ZFS
filesystem</a>'s has similar features, such as efficient
whole-filesystem snapshots, which are possible because it also uses a
write-once design, at the file-block level: when you change a file in
your favorite text editor and save it, ZFS allocates new blocks and
writes your new version to those blocks, leaving the original blocks
unchanged.
<br />
<br />
This write-once design also empowers Lucene to use optimized
data-structures
and <a href="https://issues.apache.org/jira/browse/LUCENE-6863">apply
powerful compression techniques when writing index files</a>, because
Lucene knows the values will not later change for this segment. For
example, you never have to tell Lucene that your doc values field
needs only 1 byte of storage, nor that the values are sparse, etc.:
Lucene figures that out by itself by looking at all values it is about
to write to each new segment.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVDITfPuG6X8QhUgpO2NMODLRBwhZ9HToNdG7rQ77efqfZnz2mNK_AGophF8-0FqGIsTkSnmGUmRfcJBW1gNQ87RYp8XQaIN10fdBGHvU7cckGhOr16F-YB2hqw6HPHcQIfbbNK5Vfv2rg/s1600/1.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="170" data-original-width="170" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVDITfPuG6X8QhUgpO2NMODLRBwhZ9HToNdG7rQ77efqfZnz2mNK_AGophF8-0FqGIsTkSnmGUmRfcJBW1gNQ87RYp8XQaIN10fdBGHvU7cckGhOr16F-YB2hqw6HPHcQIfbbNK5Vfv2rg/s200/1.png" width="200" /></a></div>
Finally, and the topic of this post: this design enables Lucene to
efficiently replicate a search index from one server ("primary") to
another ("replica"): in order to sync recent index changes from
primary to replica you only need to look at the file names in the
index directory, and not their contents. Any new files must be
copied, and any files previously copied do not need to be copied
again because they are never changed!
<br />
<br />
Taking advantage of this, we long
ago <a href="http://shaierera.blogspot.com/2013/05/the-replicator.html">added
a replicator module to Lucene</a> that used exactly this approach, and
it works well. However, to use those APIs to replicate recent index
changes, each time you would like to sync, you must
first <em>commit</em> your changes on the primary index. This is
unfortunately a costly operation, invoking <code>fsync</code> on
multiple recently written files, and greatly increases the sync
latency when compared to a local index opening a new NRT searcher.
<br />
<br />
<h2>
Document or segment replication?</h2>
<div>
<br /></div>
<!--<img src=replicator.svg width=200 align=right></img>-->
The requirement to commit in order to replicate Lucene's recent index
changes was such a nasty limitation that when the popular distributed search
servers <a href="http://www.elasticsearch.org/">Elasticsearch</a> and
<a href="http://lucene.apache.org/solr">Solr</a> added distributed
indexing support, they chose not to use Lucene's replication module at
all, and instead created their own <em>document replication</em>,
where the primary and all replicas redundantly index and merge all
incoming documents.
<br />
<br />
While this might seem like a natural approach to keeping replicas in
sync, there are downsides:
<br />
<ul>
<li><em>More costly CPU/IO resource usage across the cluster</em>:
all nodes must do the same redundant indexing and merging work,
instead of just one primary node. This is especially painful when
large merges are running and interfere with concurrent searching,
and is more costly on clusters that need many replicas to support
high query rates. This alone would give Elasticsearch and Solr a
big increase in cluster wide indexing and searching throughput.
<br /><br />
</li>
<li><em>Risk of inconsistency</em>: ensuring that precisely the same
set of documents is indexed in primary and all
replicas is
tricky and <a href="https://aphyr.com/posts/317-jepsen-elasticsearch">contributed to the problems Elasticsearch has had in
the past losing documents when the network is mis-behaving</a>. For example, if one of the replicas throws an
exception while indexing a document, or if a network hiccup
happens, that replica is now missing a document that the other
replicas and primary contain.
<br /><br />
</li>
<li><em>Costly node recovery after down time</em>: when a replica
goes down for a while and then comes back up, it must replay
(reindex) any newly indexed documents that arrived while it was
down. This can easily be a very large number, requiring a large
transaction log and taking a long time to catch up, or it must
fallback to a costly full index copy. In contrast, segment based
replication only needs to copy the new index files.
<br /><br />
</li>
<li><em>High code complexity</em>: the code to handle the numerous
possible cases where replicas can become out of sync quickly
becomes complex, and handling a primary switch (because the old
primary crashed) especially so.
<br /><br />
</li>
<li><em>No "point in time" consistency</em>: the primary and replicas refresh
on their own schedules, so they are all typically searching
slightly different and incomparable views of the index.
</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgArK-GL1nGp7kWRn5XurSyT3KI9uzX5tLjwuGHZkdjLGz2cFb1FWhnyGghpr21oCYLDU2OJuP1OSAJxwObhMGk3Gy0kNWP80yDIS66YI9KFGUw29_jwWdnjOGgUkLZetMIdYriZvdv2ATQ/s1600/2.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="170" data-original-width="170" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgArK-GL1nGp7kWRn5XurSyT3KI9uzX5tLjwuGHZkdjLGz2cFb1FWhnyGghpr21oCYLDU2OJuP1OSAJxwObhMGk3Gy0kNWP80yDIS66YI9KFGUw29_jwWdnjOGgUkLZetMIdYriZvdv2ATQ/s200/2.png" width="200" /></a></div>
Finally, in Lucene
6.0, <a href="https://www.elastic.co/blog/lucene-points-6.0">overshadowed
by other important features like dimensional points</a>, we quietly
improved Lucene's replication module with support
for <a href="https://issues.apache.org/jira/browse/LUCENE-5438">NRT
segment replication</a>, to copy new segment files after a refresh
without first having to call commit. This feature is an especially
compelling when combined with the neverending trend towards faster and
cheaper local area networking technologies.
<br />
<br />
<h2>
How does it work?</h2>
<div>
<br /></div>
While the logical design is straightforward ("just copy over the new
segment files from primary to replicas"), the implementation is
challenging because this adds yet another concurrent operation
(replicas slowly copying index files over the wire), along side the
many operations that already happen in <code>IndexWriter</code> such
as opening new readers, indexing, deleting, segment merging and
committing. Fortunately we were able to build on pre-existing Lucene
capabilities like NRT readers, to write new segments and identify all
their files, and snapshotting, to ensure segment files are not deleted
until replicas have finished copying them.
<br />
<br />
The two main APIs are <code>PrimaryNode</code>, which holds all state for
the primary node including a local <code>IndexWriter</code> instance,
and <code>ReplicaNode</code> for the replicas. The replicas act like
an index writer, since they also create and delete index files, and so
they acquire Lucene's index write lock when they start, to detect
inadvertent misuse. When you instantiate each replica, you provide a
pointer to where its corresponding primary is running, for example a
host or IP address and port.
<br />
<br />
Both the primary and replica nodes expose
a <code>SearcherManager</code> so you can acquire and release the
latest searcher at any time. Searching on the primary node also
works, and would match the behavior of all Elasticsearch and Solr
nodes today (since they always do both indexing and searching), but
you might also choose to dedicate the primary node to indexing only.
<br />
<br />
You use <code>IndexWriter</code> from the primary node to make index
changes as usual, and then when you would like to search recently
indexed documents, you ask the primary node to refresh. Under the
hood, Lucene will open a new NRT reader from its
local <code>IndexWriter</code>, gather the index files it references,
and notify all connected replicas. The replicas then compute which
index files they are missing and then copy them over from the primary.
<br />
<br />
Document deletions, which normally carry in memory as a bitset
directly from <code>IndexWriter</code> to an
NRT <code>IndexReader</code>, are instead written through to the
filesystem and copied over as well. All files are first copied to
temporary files, and then renamed (atomically) in the end if all
copies were successful. Lucene's existing end-to-end checksums are
used to validate no bits were flipped in transit by a flaky network
link, or bad RAM or CPU. Finally, the in-memory segments file
(a <code>SegmentInfos</code> instance) is serialized on the wire and
sent to the replicas, which then deserialize it and open an NRT
searcher via the local <code>SearcherManager</code>. The resulting
searcher on the replica is guaranteed to search the exact same
point-in-time view as the primary.
<br />
<br />
This all happens concurrently with ongoing searches on the replica,
and optionally primary, nodes. Those searches see the old searcher
until the replication finishes and the new searcher is opened. You
can also use Lucene's existing <code>SearcherLifetimeManager</code>,
which tracks each point-in-time searcher using its long version, if
you need to keep older searchers around for a while as well.
<br />
<br />
The replica and primary nodes both expose independent <em>commit</em>
APIs; you can choose to call these based on your durability
requirements, and even stagger the commits across nodes to reduce
cluster wide search capacity impact.
<br />
<br />
<h2>
No transaction log</h2>
<div>
<br /></div>
Note that Lucene does not provide a transaction log! More generally,
a cluster of primary + replicas linked by NRT replication behave a lot
like a single <code>IndexWriter</code> and NRT reader on a single JVM.
This means it is your responsibility to be prepared to replay
documents for reindexing since the last commit point, if the whole
cluster crashes and starts up again, or if the primary node crashes
before replicas were able to copy the new point-in-time refresh.
<br />
<br />
Note that at the filesystem level, there is no difference between the
Lucene index for a primary versus replica node, which makes it simple
to shut down the old primary and promote one of the replicas to be a
new primary.
<br />
<br />
<h2>
Merging</h2>
<div>
<br /></div>
With NRT replication, the primary node also does all segment merging.
This is important, because merging is a CPU and IO heavy operation,
and interferes with ongoing searching.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyA0Xdt_UKhJ9epnnnK-gekBixN_SoEuSXEmVGIW8hj9QJkEKAXTln4PkpLyA9Hy7d5rLb5LG3Rg_yhlbcSwbTqcVvTBZk3ArRkTCQX8SbtFQEB-xoVoZQCZOOYSyDCwSJBS-C9klkVxGB/s1600/3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="170" data-original-width="170" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyA0Xdt_UKhJ9epnnnK-gekBixN_SoEuSXEmVGIW8hj9QJkEKAXTln4PkpLyA9Hy7d5rLb5LG3Rg_yhlbcSwbTqcVvTBZk3ArRkTCQX8SbtFQEB-xoVoZQCZOOYSyDCwSJBS-C9klkVxGB/s200/3.png" width="200" /></a>Once the primary has finished a merge, and before it installs the
merged segment in its index, it uses Lucene's merged segment warming
API to give all replicas a chance to pre-copy the merged segment.
This means that a merge should never block a refresh, so Lucene will
keep fast refreshes even as large merged segments are still copying.
Once all running replicas have pre-copied the merge, then the primary
installs the merged segment, and after the next refresh, so do all the
replicas.
<br />
<br />
We have discussed having replicas perform their own merging, but I
suspect that will be a poor tradeoff. Local area networks (e.g. 10
gigabit ethernet) are quickly becoming plenty fast and cheap, and
asking a replica to also do merging would necessarily impact search
performance. It would also be quite complex trying to ensure replicas
perform precisely the same merges as the primary at the same time, and would otherwise break the same point-in-time view across replicas.<br />
<br />
<h2>
Abstractions</h2>
<div>
<br /></div>
The primary and replica nodes are abstract: you must implement certain
functions yourself. For example, the low level mechanics of how to copy bytes
from primary to replica is up to you. Lucene does not provide that,
except in its unit tests which use simple point-to-point TCP "thread
per connection" servers. You could choose to use rsync, robocopy,
<a href="https://netty.io/">netty</a> servers, a central file server,
carrier pigeons, UDP multicast (likely helpful if there are many
replicas on the same subnet), etc.
<br />
<br />
Lucene also does not provide any distributed leader election
algorithms to pick a new primary when the current primary has crashed,
nor heuristics to detect a downed primary or replica. But once you
pick the new primary, Lucene will take care of having all replicas
cutover to it, removing stale partially copied files from the old
primary, etc.
<br />
<br />
Finally, Lucene does not provide any load-balancing to direct queries
to the least loaded replica, nor any cluster state to keep track of
which node is the primary and which are the replicas, though <a href="https://zookeeper.apache.org/">Apache Zookeeper</a> is useful for such shared distributed state. These parts are
all up to you!<br />
<br />
<h2>
Expected failure modes</h2>
<div>
<br /></div>
There are many things that can go wrong with a cluster of servers
indexing and searching with NRT index replication, and we wrote
an <a href="https://github.com/apache/lucene-solr/blob/master/lucene/replicator/src/test/org/apache/lucene/replicator/nrt/TestStressNRTReplication.java">evil
randomized stress test case</a> to exercise Lucene's handling in such
cases. This test creates a cluster by spawning JVM subprocesses for a
primary and multiple replica nodes, and begins indexing and
replicating, while randomly applying conditions like an unusually slow
network to specific nodes, a network that randomly flips bits, random
JVM crashes (SIGSEGV!) of either primary or replica nodes, followed by
recovery, etc. This test case uncovered all sorts of fun corner
cases!
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAqTIaJBiuTzh0RHvXQpRTh9cTXd1HoxmNtThlLOVtjRKEFWNpH4JTtH-D2YY-CEoi8fSX6x2RD8vfpdmra0mUjPOkO-bVNMpLrjhr8jZ3nLpGUIvEEtnuDj-jee2OA6zKxmFpho6aKG31/s1600/4.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="170" data-original-width="170" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAqTIaJBiuTzh0RHvXQpRTh9cTXd1HoxmNtThlLOVtjRKEFWNpH4JTtH-D2YY-CEoi8fSX6x2RD8vfpdmra0mUjPOkO-bVNMpLrjhr8jZ3nLpGUIvEEtnuDj-jee2OA6zKxmFpho6aKG31/s200/4.png" width="200" /></a>An especially important case is when the primary node goes down
(crashes, loses power or is intentionally killed). In this case, one
of the replicas, ideally the replica that was furthest along in
copying files from the old primary as decided by a distributed
election, is promoted to become the new primary node. All other
replicas then switch to the new primary, and in the process must
delete any files copied or partially copied from the old primary but
not referenced by the new primary. Any documents indexed into the
primary but not copied to that replica will need to be indexed again,
and this is the caller's responsibility (no transaction log).
<br />
<br />
If the whole cluster crashes or loses power, on restart you need to
determine whichever index (primary or replica) has the "most recent"
commit point, and start the primary node on that host and replica
nodes on the other hosts. Those replica nodes may need to delete some
index files in order to switch to the new primary, and Lucene takes
care of that. Finally, you will have to replay any documents that
arrived after the last successful commit.
<br />
<br />
Other fun cases include: a replica crashing while it was still pre-copying
a merge; a replica that is falling behind because it is still copying
files from the previous refresh when a new refresh happens; a replica going down and coming back up later after the primary node has also changed.<br />
<br />
<br />
<h2>
Downsides</h2>
<div>
<br /></div>
There are also some minor downsides to segment replication:
<br />
<ul>
<li><em>Very new code</em>: this feature is quite new and also quite
complex, and not widely used yet. Likely there exciting bugs!
Patches welcome!
<br /><br />
</li>
<li><em>Slightly slower refresh time</em>: after refreshing on the
primary, including writing document deletions to disk as well, we
must then copy all new files to the replica and open a new
searcher on the replica, adding a bit more time before documents
are visible for searching on the replica when compared to a straight NRT reader from an <span style="font-family: monospace;">IndexWriter</span>. If this is really a problem,
you can use the primary node for searching and have refresh
latency very close to what a straight NRT reader provides.
<br /><br />
</li>
<li><em>Index problems might be replicated</em>: if something goes
wrong, and the primary somehow writes a broken index file, then
that broken file will be replicated to all replicas too. But this
is uncommon these days, especially with
Lucene's <a href="https://issues.apache.org/jira/browse/LUCENE-5842">end
to end checksums</a>.</li>
</ul>
<h2>
</h2>
<h2>
Concluding</h2>
<div>
<br /></div>
NRT segment replication represents an opportunity for sizable
performance and reliability improvements to the popular distributed
search servers, especially when combined with the ongoing trend
towards faster and cheaper local area networks. While this feature unfortunately came too late for Elasticsearch and
Solr, I am hopeful that the next popular distributed search server, and
its users, can benefit from it!<br />
<br />
<em style="background-color: white; color: #333333; font-family: Verdana, Geneva, sans-serif; font-size: 14.3px;">[I work at Amazon and the postings on this site are my own and don't necessarily represent Amazon's position]</em>Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com6tag:blogger.com,1999:blog-8623074010562846957.post-85702217832902862692017-07-02T13:28:00.000-04:002017-09-26T11:59:49.208-04:00Lucene gets concurrent deletes and updates!Long ago, Lucene could only use a single thread to write new segments
to disk. The actual indexing of documents, which is the costly
process of inverting incoming documents into in-memory segment data
structures, could run with multiple threads, but back then, the
process of writing those in-memory indices to Lucene segments was
single threaded.
<br />
<br />
We fixed that,
<a href="https://issues.apache.org/jira/browse/LUCENE-3023">more than 6 years ago now</a>, yielding <a href="http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html">big indexing throughput gains on
concurrent hardware</a>.
<br />
<br />
Today, hardware has only become even more concurrent, and we've
finally done the same
thing <a href="https://issues.apache.org/jira/browse/LUCENE-7868">for
processing deleted documents and updating doc values</a>!
<br />
<br />
This change, in time for
Lucene's <a href="http://blog.mikemccandless.com/2017/03/apache-lucene-70-is-coming-soon.html">next
major release (7.0)</a>, shows a 53% indexing throughput speedup when
updating whole documents, and a 7.4X - 8.6X speedup when updating doc
values, on a private test corpus using highly concurrent hardware (an
<a href="https://aws.amazon.com/ec2/instance-types/#i3">i3.16xlarge EC2 instance</a>).
<br />
<br />
<h3>
Buffering versus applying</h3>
<br />
When you ask Lucene's <code>IndexWriter</code> to delete a document,
or update a document (which is an atomic delete and then add), or to
update a doc-values field for a document, you pass it
a <code>Term</code>, typically against a primary key field
like <code>id</code>, that identifies which document to update.
But <code>IndexWriter</code> does not perform the deletion right away.
Instead, it buffers up all such deletions and updates, and only
finally applies them in bulk once they are using too much RAM, or you
refresh your
<a href="http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html">near-real-time
reader</a>, or call commit, or a merge needs to kick off.
<br />
<br />
The process of resolving those terms to actual Lucene document ids is
quite costly as Lucene must visit all segments and perform a primary
key lookup for each term. Performing lookups in batches gains some
efficiency because we sort the terms in unicode order so we can do a
single sequential scan through each segment's terms dictionary and postings.
<br />
<br />
We have also
optimized <a href="https://home.apache.org/~mikemccand/lucenebench/PKLookup.html">primary
key lookups</a> and the buffering of deletes and updates quite a bit
over time, with issues
like <a href="https://issues.apache.org/jira/browse/LUCENE-6161">LUCENE-6161</a>,
<a href="https://issues.apache.org/jira/browse/LUCENE-6161">LUCENE-2897</a>,
<a href="https://issues.apache.org/jira/browse/LUCENE-6161">LUCENE-2680</a>,
<a href="https://issues.apache.org/jira/browse/LUCENE-6161">LUCENE-3342</a>.
Our fast <a href="https://issues.apache.org/jira/browse/LUCENE-3030">
BlockTree terms dictionary</a> can sometimes save a disk seek for each
segment if it can tell from
the <a href="http://blog.mikemccandless.com/2013/06/build-your-own-finite-state-transducer.html">finite
state transducer</a> terms index that the requested term cannot
possibly exist in this segment.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzMaw-sy5NyC13qsT5jsM3LwiOp2LY2eXX2PH2GOLQngO5C13LohyFfRheRSo4Itkd51zRtIgjUs9pkj71kWCF5jfJPaQdL81d9oowYXl3jEHJiHRwTyhPwrQxJPYBXnIhKH3THueYijuV/s1600/refresh.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1266" data-original-width="1280" height="317" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzMaw-sy5NyC13qsT5jsM3LwiOp2LY2eXX2PH2GOLQngO5C13LohyFfRheRSo4Itkd51zRtIgjUs9pkj71kWCF5jfJPaQdL81d9oowYXl3jEHJiHRwTyhPwrQxJPYBXnIhKH3THueYijuV/s320/refresh.png" width="320" /></a></div>
<br />
Still, as fast as we have made this code, only one thread is allowed
to run it at a time, and for update-heavy workloads, that one thread
can become a major bottleneck. We've seen users asking about this in
the past, because while the deletes are being resolved it looks as
if <code>IndexWriter</code> is hung since nothing else is happening.
The larger your indexing buffer the longer the hang.
<br />
<br />
Of course, if you are simply appending new documents to your Lucene
index, never updating previously indexed documents, a common use-case
these days with
the <a href="https://www.elastic.co/webinars/using-elasticsearch-for-log-search-and-analysis">broad
adoption of Lucene for log analytics</a>, then none of this matters to
you!
<br />
<br />
<h3>
Concurrency is hard</h3>
<br />
With this change, <code>IndexWriter</code> still buffers deletes and updates into
<em>packets</em>, but whereas before, when each packet was also
buffered for later single-threaded application, instead <code>IndexWriter</code> now
immediately resolves the deletes and updates in that packet to the
affected documents using the current indexing thread. So you gain as
much concurrency as indexing threads you are sending
through <code>IndexWriter</code>.
<br />
<br />
The change was overly difficult because of <code>IndexWriter</code>'s
terribly complex concurrency, a technical debt I am now convinced we
need to address head-on by somehow
refactoring <code>IndexWriter</code>. This class is challenging to
implement since it must handle so many complex and costly concurrent
operations: ongoing indexing, deletes and updates; refreshing new
readers; writing new segment files; committing changes to disk;
merging segments and adding indexes. There are numerous locks, not
just <code>IndexWriter</code>'s monitor lock, but also many other
internal classes, that make it easy to accidentally trigger a deadlock
today. Patches welcome!
<br />
<br />
The original change also led to
some <a href="https://issues.apache.org/jira/browse/LUCENE-7888">cryptic</a> <a href="https://issues.apache.org/jira/browse/LUCENE-7887">test</a> <a href="https://issues.apache.org/jira/browse/LUCENE-7894">failures</a>
thanks to our extensive randomized tests, which we are working through
for 7.0.
<br />
<br />
That complex concurrency unfortunately prevented me from making the
final step of deletes and updates fully concurent: writing the new
segment files. This file writing takes the in-memory resolved doc ids
and writes a new per-segment bitset, for deleted documents, or a whole
new doc values column per field, for doc values updates.
<br />
<br />
This is typically a fast operation, except for large indices where a
whole column of doc-values updates could be sizable. But since we
must do this for every segment that has affected documents, doing this
single threaded is definitely still a small bottleneck, so it would be
nice, once we succeed in simplifying <code>IndexWriter</code>'s
concurrency, to also make our file writes concurrent.
<br />
<br />
<em style="background-color: white; color: #333333; font-family: Verdana, Geneva, sans-serif; font-size: 14.3px;">[I work at Amazon and the postings on this site are my own and don't necessarily represent Amazon's position]</em>Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com2tag:blogger.com,1999:blog-8623074010562846957.post-76139735981990079812017-03-14T20:47:00.000-04:002017-03-14T20:47:05.319-04:00Apache Lucene 7.0 Is Coming Soon!The <a href="http://lucene.apache.org/">Apache Lucene project</a> will likely release its next major release, 7.0, <a href="http://markmail.org/thread/o5tstnrh2i5p7viw">in a few
months</a>!
<br />
<br />
Remember that Lucene developers generally try hard to backport new features for the next non-major (feature) release, and the upcoming <a href="http://jirasearch.mikemccandless.com/search.py?chg=ddm&text=&a1=issueType&a2=New+Feature&page=0&sort=recentlyUpdated&format=list&id=sdwn0erq6h8t&dd=fixVersions%3A6.5&dd=project%3ALucene&dd=issueType%3AImprovement&newText=">6.5 already has many great changes</a>, so a new major release is exciting because it means the 7.0-only features, which I now describe, are the particularly big ones that we felt could not be backported for 6.5.
<br />
<br />
Of course, with every major release, we also do more mundane things like remove deprecated 6.x APIs, and drop support for old indices (written with Lucene 5.x or earlier).
<br />
<br />
This is only a subset of the new 7.0 only features; for the full list <a href="https://github.com/apache/lucene-solr/blob/master/lucene/CHANGES.txt">please see the 7.0.0 section in the upcoming <code>CHANGES.txt</code></a>.
<br />
<br />
<b>Doc values as iterators</b>
<br />
<br />
The biggest change in 7.0 is <a href="https://issues.apache.org/jira/browse/LUCENE-7407">changing doc values from a random access API to a more restrictive iterator API</a>.
<br />
<br />
Doc values are Lucene's column-stride numeric, sorted or binary per-document field storage across all documents. They can be used to hold scoring signals, such as the single-byte (by default) document length encoding or application-dependent signals, or for sorting, faceting or grouping, or even numeric fields that you might use for range filtering in some queries. Their column-stride storage means it's efficient to visit all values for the one field across documents, in contrast to row-stride storage that stored fields use to retrieve all field values for a single document.
<br />
<br />
Postings have long been consumed through an iterator, so this was a relatively natural change to make, and the two share the same base class, <code>DocIdSetIterator</code>, to step through or seek to each hit.
<br />
<br />
The <a href="https://issues.apache.org/jira/browse/LUCENE-7407">initial rote switch to an iterator API</a> was really just a plumbing swap and less interesting than all the subsequent user-impacting improvements that became possible thanks to the more restrictive API:
<br />
<ul>
<li>The 7.0 codec <a href="http://issues.apache.org/jira/browse/LUCENE-7489">now sparsely encodes sparse doc values</a> and <a href="http://issues.apache.org/jira/browse/LUCENE-7475">length normalization factors ("norms")</a>
</li>
<li> Outlier values <a href="https://issues.apache.org/jira/browse/LUCENE-7589">no longer consume excessive space</a>
</li>
<li> Our doc values based queries <a href="https://issues.apache.org/jira/browse/LUCENE-7461">take advantage of the new API</a>
</li>
<li> Both <a href="https://issues.apache.org/jira/browse/LUCENE-7589">top-level browse-only facet counts</a> and <a href="https://issues.apache.org/jira/browse/LUCENE-7506">facet counts for hits in a query</a> are now faster in sparse cases
</li>
<li> <a href="https://issues.apache.org/jira/browse/LUCENE-7462">A new <code>advanceExact</code> method</a> enables more efficient skipping
</li>
</ul>
With these changes, you finally only pay for what you actually use with doc values, in index size, indexing performance, etc. This is the same as other parts of the index like postings, stored fields, term vectors, etc., and it means users with very sparse doc values <a href="https://issues.apache.org/jira/browse/LUCENE-7253">no longer see merges taking unreasonably long time or the index becoming unexpectedly huge while merging</a>.
<br />
<br />
Our <a href="https://home.apache.org/~mikemccand/lucenebench/sparseResults.html">nightly sparse benchmarks</a>, based on the <a href="http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml">NYC Trip Data corpus</a>, show the impressive gains each of the above changes (and more!) accomplished.
<br />
<br />
<b>Goodbye index-time boosts</b>
<br />
<br />
Index-time boosting, which lets you increase the a-priori score for a particular document versus other documents, <a href="https://issues.apache.org/jira/browse/LUCENE-6819"> is now deprecated and will be removed in 7.0</a>.
<br />
<br />
This has always been a fragile feature: it was encoded, along with the field's length, into a single byte value, and thus had very low precision. Furthermore, it is now straightforward to write your custom boost into your own doc values field and use function queries to apply the boost at search time. Finally, with index time boosts gone, <a href="https://issues.apache.org/jira/browse/LUCENE-6819?focusedCommentId=15870559&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15870559">length encoding is more accurate</a>, and in particular the first nine length values (1 to 9) are distinct.
<br />
<br />
<b>Query scoring is simpler</b>
<br />
<br />
<code>BooleanQuery</code> has long exposed a confusing scoring feature called the <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/practical-scoring-function.html#coord">coordination factor</a> (<code><b>coord</b></code>), to reward hits containing a higher percentage of the search terms. However, this hack is only necessary for scoring models like <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF/IDF</a> which have "weak" term saturation such that many occurrences of a single term in a document would be more powerful than adding a single occurence of another term from the query. Since this was specific to one scoring model, <code>TFIDFSimilarity</code>, and since Lucene has now switched to the better <a href="https://en.wikipedia.org/wiki/Okapi_BM25">Okapi BM25</a> scoring model by default, we have now <a href="https://issues.apache.org/jira/browse/LUCENE-7369">fully removed coordination factors</a> in 7.0 from both <code>BooleanQuery</code> and <code>Similarity</code>.
<br />
<br />
Likewise, the query normalization phase of scoring <a href="http://issues.apache.org/jira/browse/LUCENE-7368">will be removed</a>. This phase tried to equalize scores across different queries and indices so that they are more comparable, but didn't alter the sort order of hits, and was also TF/IDF specific.
<br />
<br />
With these scoring simplifications <a href="https://issues.apache.org/jira/browse/LUCENE-7416"><code>BooleanQuery</code> now makes more aggressive query optimizations when the same sub-clause occurs with different <code>Occur</code> constraints</a>, previously not possible since the scores would change.
<br />
<br />
<b>Classic Query Parser no longer splits on whitespace</b>
<br />
<br />
Lucene's original, now called "classic", query parser, always pre-splits the incoming query text on whitespace, and then separately sends those single tokens to the query-time analyzer. This means multi-token filters, such as <a href="https://issues.apache.org/jira/browse/LUCENE-6664"><code>SynonymGraphFilter</code></a> or <code>ShingleFilter</code>, will not work.
<br />
<br />
For example, if the user asks for "denial of service attack" and you had a synonym mapping "denial of service" to DOS, the classic query parser would separately analyze "denial", "of" and "service" so your synonym would never match.
<br />
<br />
We have already <a href="https://issues.apache.org/jira/browse/LUCENE-2605">added an option to the query parser</a> to not pre-split on whitespace but left the default unchanged for 6.x releases to preserve backwards compatibility. Finally, with 7.0, <a href="https://issues.apache.org/jira/browse/LUCENE-2605">we fix that default</a> so that analyzers can see multiple tokens at once, and synonyms will work.
<br />
<br />
<b>More stuff</b>
<br />
<br />
As of 7.0 Lucene will (finally!) <a href="https://issues.apache.org/jira/browse/LUCENE-7703">record into the index metadata which Lucene version was used to originally create it</a>. This knowledge can help us implement future backwards compatibility.
<br />
<br />
<a href="http://blog.mikemccandless.com/2011/01/finite-state-transducers-part-2.html">Finite state transducers</a>, used in many ways in Lucene, used to have a complex method call <code>pack</code> which would eek out a few more bytes to further shrink the already small size of the FST. But the code was complex and rarely used and sometimes even made the FST larger so <a href="https://issues.apache.org/jira/browse/LUCENE-7531">we have removed it for 7.0</a>.
<br />
<br />
<code>IndexWriter</code>, used to add, update and delete documents in your index, <a href="https://issues.apache.org/jira/browse/LUCENE-7629">will no longer accept broken token offsets</a> sometimes produced by mis-behaving token filters. Offsets are used for highlighting, and broken offsets, where the end offset for a single token comes before the start offset, or the start offset of a token goes backwards versus the previous token, can only break search-time highlighting. So with this change, Lucene prevents such mistakes at index time by throwing an exception. To ease this transition for cases where users didn't even know their analyzer was producing broken offsets, we've also added a few token filters to "correct" offsets before they are passed to <code>IndexWriter</code>.
<br />
<br />
Advanced users of Lucene often need to cache something custom for each segment at search time, but the APIs for this are trappy and can lead to unexpected memory leaks so <a href="https://issues.apache.org/jira/browse/LUCENE-7410">we have overhauled these APIs to reduce the chance of accidental misuse</a>.
<br />
<br />
Finally, the dimensional points API <a href="https://issues.apache.org/jira/browse/LUCENE-7494">now takes a field name up front to offer per-field points access</a>, matching how the doc values APIs work.
<br />
<br />
Lucene 7.0 has not been released, so if you have ideas on any additional major-release-worthy changes you'd like to explore please <a href="mailto:dev@lucene.apache.org">reach out</a>!
<br />
<br />
<em>[I work at Amazon and the postings on this site are my own and don't necessarily represent Amazon's position]</em>
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com13tag:blogger.com,1999:blog-8623074010562846957.post-41993334983901163852016-10-20T16:33:00.001-04:002016-10-21T11:29:11.589-04:00Jirasearch 2.0 dog food: using Lucene to find our Jira issues<a href="http://blog.mikemccandless.com/2014/03/using-lucenes-search-server-to-search.html">A
few years ago</a> I first
built and
released <a href="http://jirasearch.mikemccandless.com">Jirasearch</a>
as a fun dog-food test case for the thin-wrapper <a href="http://github.com/mikemccand/luceneserver">Lucene
server</a>, to expose a powerful search UI over our Jira issues.
<p>
This is a great showcase of a number of Lucene's important features:
<ul>
<li>Using block join queries to model parent (the original Jira issue) and
children (each comment) documents. This basic relational structure is
<a href="http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html">also common in
e-commerce applications</a>, where you have a product (e.g. a specific
shirt) and then individual SKUs (size/color combinations) under
that shirt<br><br>
<li>Highlighting
with <a href="http://blog.mikemccandless.com/2012/12/a-new-lucene-highlighter-is-born.html"><code>PostingsHighlighter</code></a><br><br>
<li>Faceting, with flat, hierarchical, and <a href="http://blog.mikemccandless.com/2013/05/dynamic-faceting-with-lucene.html">dynamic numeric range</a>
fields. Remember you can <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=ddm&a1=project&a2=Solr&sort=recentlyUpdated&dd=project%3ALucene">pick multiple facet values (multi-select)</a> with shift+click!<br><br>
<li><a href="http://blog.mikemccandless.com/2013/02/drill-sideways-faceting-with-lucene.html"><code>DrillSideways</code></a>
facet counts, so you don't lose facet counts
of other labels just because you drilled down to one of them<br><br>
<li><a href="http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html"><code>AnalyzingInfixSuggester</code> for auto-suggest</a>,
including <a href="https://issues.apache.org/jira/browse/LUCENE-5477">near-real-time
updates</a>. Suggestions are project specific: if you have drilled down to specific project/s,
then the suggestions will only be from those projects, thanks
to <code>AnalyzingInfixSuggester</code> <a href="https://issues.apache.org/jira/browse/LUCENE-5528">now
supporting contexts</a><br><br>
<li>Near real time indexing and searching<br><br>
<li><code>WordDelimiterFilter</code> so camel case tokens are split
(try <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=new&text=infix">searching for infix</a>)<br><br>
<li>Synonyms<br><br>
<li>Using <a href="http://shaierera.blogspot.com/2014/04/expressions-with-lucene.html">expressions</a>
to dynamically compute a blend of recency and relevance for the
sort order score for hits
</ul>
Curiously, spell correction, or even fuzzy infix suggestions, is still missing (pull requests welcome!).
<p>
Since the initial release
of <a href="http://jirasearch.mikemccandless.com">Jirasearch</a> it has seen substantial usage and interest from users and
developers. Building this and keeping it running all this time has been an
awesome and humbling exercise for me because I get to experience life
as a "production" user of our software. At the same time, we all get a nice search
UI for finding issues.
<p>
<h2>Upgrading from Lucene 4.6.x to 6.x</h2>
<p>
For the past week or so I had another similarly humbling experience,
this time
upgrading <a href="http://jirasearch.mikemccandless.com">Jirasearch</a>
from the very-old Lucene 4.6.x release, to the latest 6.x release.
Small (yet vital!) things changed, such as
the <a href="https://issues.apache.org/jira/browse/LUCENE-7497">new
requirement to use a special index searcher</a>
with <code>ToParentBlockJoinQuery</code>, which conflicts with how you
must use <code>DrillSideways</code>.
I <a href="https://issues.apache.org/jira/browse/LUCENE-7505">hit this
bug</a> in the infix suggester. Something changed about pure negative
boolean queries, but I am still not sure what (I have worked around it
for now)!
<p>
I had already previously
upgraded <a href="http://github.com/mikemccand/luceneserver">Lucene
server</a> to dimensional points so I got that "for free" for the existing
numeric fields in <a href="http://jirasearch.mikemccandless.com">Jirasearch</a>.
<p>
<h2>New Jirasearch features</h2>
<p>
Besides "merely" upgrading from Lucene 4.6.x to 6.x, and switching all
numeric fields to the new dimensional points, I also added some
compelling user-visible improvements (thank you
to <a href="http://www.outerthoughts.com/">Alexandre Rafalovitch</a>
for suggesting some of these, thus kick-starting my unexpectedly
challenging upgrade-and-improve effort):
<ul>
<li> cutting@apache.org is finally presented as Doug Cutting! Plus,
the auto-suggest now works if you type "Doug".
<li> The new <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=dds&a1=updatedAgo&a2=%3E+1+month+ago">Updated ago</a> facet dimension lets you drill
down to issues that have not been updated for some time.
<li> The new <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=dds&a1=lastContributor&a2=Michael+McCandless">Last comment user</a> facet dimension is the
user who last commented on an issue.
<li> The new <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=dds&a1=committedBy&a2=Michael+McCandless">Committed by</a> facet dimension lets you
drill down to those issues a given developer has committed changes
for.
<li> The <b>Committed paths</b> hierarchical facet dimension,
letting you find issues according to which paths in the source
tree were changed for that issue, was broken since
we <a href="https://issues.apache.org/jira/browse/LUCENE-6938">switched
from Subversion to Git</a>.
<li> The <a href="https://issues.apache.org/jira/browse/INFRA/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel">Infrastructure</a>
project issues are now included as well.
<li> The per-comment text processing sees some minor improvements, e.g. expanding
a referenced user name to their display name,
mapping <code>commitbot</code> comment link directly to the change
set and including the branch name, plus a few new synonyms (try <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=new&text=pnp">pnp</a>!)
</ul>
<p>
The new facet fields are especially fun: you can now find issues that
<a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=dds&text=&a1=project&a2=Lucene&page=0&searcher=485&sort=recentlyUpdated&format=list&id=dqbd7opzzk1z&dd=project%3ALucene%2CSolr&dd=lastContributor%3AMichael+McCandless&dd=updatedAgo%3A%3E+1+week+ago&dd=status%3AOpen%2CReopened&newText=">you perhaps killed</a>, by drilling down on <b>Updated ago > 1 month
ago</b> and <b>Last comment user = you</b> (this was the use case
suggested by Alexandre).
<p>
Another fun one
is to see issues a given developer committed (<b>Committed by</b>) to
an unusual part of the source tree (<b>Committed paths</b>), e.g. the
issues
where <a href="http://jirasearch.mikemccandless.com/search.py?chg=dds&a1=project&a2=Lucene&page=0&dd=committedBy%3AMichael+McCandless&dd=project%3ASolr&dd=committedPaths%3Asolr">I
committed changes to Solr for a Lucene Jira issue</a>.
<p>
<h2>Open source Jirasearch</h2>
<p>
With this update I am also making all the sources behind jirasearch
open-source under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2 license</a>, in
the <code>examples/jirasearch</code> <a href="https://github.com/mikemccand/luceneserver/tree/master/examples/jirasearch">sub-directory
of the luceneserver github project</a>.
<p>
While <a href="https://github.com/mikemccand/luceneserver">Luceneserver
itself is entirely Java</a>, the sources for the Jirasearch
application, to extract details of all issues from the Apache Jira
instance, to convert those documents into Lucene server documents, to
do a full and near-real-time indexing, building suggestest, and the
search UI, are entirely Python.
<p>
Please note the Python sources are
not particularly pretty. Yet, they are functional, and as always:
patches welcome!
<p>
It's likely I broke things during this upgrade process; please let me know (add a comment here, or shoot me an email) if so.Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com10tag:blogger.com,1999:blog-8623074010562846957.post-87972092332267172752015-10-25T08:02:00.001-04:002015-10-25T08:02:16.072-04:00Where are my new blog posts?<p>Some of you have noticed that I'm not writing much in this blog lately.
<p>But fear not: exciting changes are still happening in <a href="http://lucene.apache.org">Lucene</a>, and I am still writing about them!
<p>It's just that most of what I write is now appearing at either the <a href="https://www.elastic.co/blog">Elastic blogs</a> or on my <a href="https://plus.google.com/u/0/+MichaelMcCandless">Google+ feed</a>, so please head over to those two sources to keep reading about the fun changes in <a href="http://lucene.apache.org">Apache Lucene</a> and <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a>.Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com0tag:blogger.com,1999:blog-8623074010562846957.post-35615925349770211592014-11-17T04:31:00.000-05:002014-11-17T06:11:32.041-05:00Apache Lucene™ 5.0.0 is coming!At long last, after a strong series of 4.x feature releases, most
recently <a href="http://lucene.apache.org">4.10.2</a>, we are finally
working towards another
major <a href="http://lucene.apache.org">Apache Lucene</a> release!
<p>
There are no promises for the exact timing (it's done when it's
done!), but we already have
a <a href="http://markmail.org/message/nbo4cpenjqncymje">volunteer
release manager</a> (thank you Anshum!).
<p>
A major release in Lucene means all deprecated APIs (as of 4.10.x) are
dropped, support for 3.x indices is removed while
the <a href="https://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x/lucene/backward-codecs/src/java/org/apache/lucene/codecs/">numerous
4.x index formats</a> are still supported for index backwards
compatibility, and the 4.10.x branch becomes our bug-fix only release
series (no new features, no API changes).
<p>
5.0.0 already contains a number of exciting changes, which I describe
below, and they are still rolling in with ongoing active development.
<p>
<b>Stronger index safety</b>
<p>
Many of the 5.0.0 changes are focused on providing stronger protection
against index corruption.
<p>
All file
access <a href="https://issues.apache.org/jira/browse/LUCENE-5945">now uses</a>
<a href="https://en.wikipedia.org/wiki/Non-blocking_I/O_(Java)#JDK_7_and_NIO.2">Java's
NIO.2 APIs</a>, giving us better error handling
(e.g., <code>Files.delete</code> returns a meaningful exception) along
with atomic rename for
<a href="https://issues.apache.org/jira/browse/LUCENE-5925">safer
commits</a>, reducing the risk of hideous "your entire index is gone"
bugs
like <a href="https://issues.apache.org/jira/browse/LUCENE-4870">this
doozie</a>.
<p>
Lucene's <a href="http://shaierera.blogspot.com/2013/05/the-replicator.html">replication
module</a>, along with distributed servers on top of Lucene such
as <a href="http://elasticsearch.org">Elasticsearch</a>
or <a href="http://lucene.apache.org/solr">Solr</a>, must copy index
files from one place to another. They do this for backup purposes
(e.g., <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html">snapshot
and restore</a>), for migrating or recovering a shard from one node
to another or when adding a new replica. Such replicators try to be
incremental, so that if the same file name is present, with the same
length and checksum, it will not be copied again.
<p>
Unfortunately, these layers sometimes have subtle bugs (they are
complex!). Thanks
to <a href="https://issues.apache.org/jira/browse/LUCENE-2446">checksums</a>
(added in 4.8.0), Lucene already detects if the replicator caused any
bit-flips while copying, and this revealed a long
standing <a href="https://github.com/elasticsearch/elasticsearch/pull/7210">nasty
bug</a> in the compression
library <a href="http://elasticsearch.org">Elasticsearch</a> uses.
<p>
With 5.0.0 we take this even further and now detect if whole files
were copied to the wrong file name, by assigning
<a href="https://issues.apache.org/jira/browse/LUCENE-5925">a unique
id</a> to every segment and commit (<code>segments_N</code> file).
Each index file now records
the <a href="https://issues.apache.org/jira/browse/LUCENE-5969">segment
id in its header</a>, and then these ids are cross-checked when the
index is opened.
<p>
The new <a href="https://issues.apache.org/jira/browse/LUCENE-5969">Lucene50Codec</a>
also includes further index corruption detection.
<p>
Even <code>CorruptIndexException</code> itself is improved! It will
now always refer to the file or resource where the corruption was
detected, as this is now a required argument to its constructors.
When corruption is detected higher up (e.g., a bad field number in the
field infos file), the resulting <code>CorruptIndexException</code>
will now state whether there was also a checksum mismatch in the file,
helping to narrow the possible source of the corruption.
<p>
Finally, during merge, <code>IndexWriter</code> now always checks the
incoming segments for corruption before merging. This can mean, on
upgrading to 5.0.0, that merging may uncover long-standing latent
corruption in an older 4.x index.
<p>
<b>Reduced heap usage</b>
<p>
5.0.0 also includes several changes to reduce heap usage during
indexing and searching.
<p>
If your index has 1B docs, then caching a
single <code>FixedBitSet</code>-based filter in <code>4.10.2</code>
costs a non-trivial 125 MB of heap! But with 5.0.0, Lucene now
supports random-writable and advance-able sparse bitsets
(<a href="https://issues.apache.org/jira/browse/LUCENE-5983"><code>RoaringDocIdSet</code></a>
and <a href="https://issues.apache.org/jira/browse/LUCENE-5938"><code>SparseFixedBitSet</code></a>),
so the heap required is in proportion to how many bits are set, not how
many total documents exist in the index. These bitsets also greatly
simplify how <code>MultiTermQuery</code> is rewritten (no
more <code>CONSTANT_SCORE_AUTO_REWRITE_METHOD</code>), and they
provide faster advance implementations than <code>FixedBitSet</code>'s
linear scan. Finally, they provide a more
accurate <code>cost()</code> implementation, allowing Lucene to make
better choices about how to drive the intersection at query time.
<p>
Heap usage during <code>IndexWriter</code> <a href="http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">merging</a> is
also <a href="https://issues.apache.org/jira/browse/LUCENE-5969">much
lower with the new Lucene50Codec</a>, since doc values and norms for
the segments being merged are no longer fully loaded into heap for all
fields; now they are loaded for the one field currently being merged, and then
dropped.
<p>
The default norms format now uses sparse encoding when appropriate, so
indices that enable norms for many sparse fields will see a
large reduction in required heap at search time.
<p>
<b>An explain API for heap usage </b>
<p>
If you still find Lucene using more heap than you expected, 5.0.0
has a <a href="https://issues.apache.org/jira/browse/LUCENE-5949">new API to
print a tree structure</a> showing a recursive breakdown of which
parts are using how much heap. This is analogous to
Lucene's <a href="https://lucene.apache.org/core/4_10_2/core/org/apache/lucene/search/IndexSearcher.html#explain(org.apache.lucene.search.Query, int)">explain API</a>, used to understand why a
document has a certain relevance score, but applied to heap usage
instead.
<p>
It produces output like this:
<pre>
_cz(5.0.0):C8330469: 28MB
postings [...]: 5.2MB
...
field 'latitude' [...]: 678.5KB
term index [FST(nodes=6679, ...)]: 678.3KB
</pre>
This is a much faster way to see what is using up your heap than trying to stare
at a Java heap dump.
<p>
<b>Further changes</b>
<p>
There is a long tail of additional 5.0.0 changes; here are some of them:
<ul>
<li>Old experimental postings formats
(<code>Sep/Fixed/VariableIntPostingsFormat</code>) have been
removed. PulsingPostingsFormat has also been removed,
since the default postings format already pulses unique terms.
<p>
<li><code>FieldCache</code> is gone (moved to a
dedicated <code>UninvertingReader</code> in the <code>misc</code>
module). This means when you intend to sort on a field, you
should index that field using doc values, which is much faster and
less heap consuming than <code>FieldCache</code>.
<p>
<li> <code>Tokenizer</code>s and <code>Analyzer</code>s <a href="https://issues.apache.org/jira/browse/LUCENE-5388">no longer require</a> <code>Reader</code> on init.
<p>
<li> <code>NormsFormat</code> now gets its own dedicated <code>NormsConsumer/Producer</code>
<p>
<li> Simplifications to <code>FieldInfo</code> (Lucene's "low schema"): no more <code>normType</code>
(it is always a <code>DocValuesType.NUMERIC</code>), no more <code>isIndexed</code> (just check
<code>IndexOptions</code>)
<p>
<li> Compound file handling is simpler, and is now under codec control.
<p>
<li> <code>SortedSetSortField</code>, used to sort on a multi-valued
field, is promoted from sandbox to Lucene's core
<p>
<li> <code>PostingsFormat</code> now uses a "pull" API when writing
postings, just like doc values. This is powerful because you can
do things in your postings format that require making more than one
pass through the postings such as iterating over all postings for
each term to decide which compression format it should use.
<p>
<li> Version is no longer required on init to classes
like <code>IndexWriterConfig</code> and analysis components.
<p>
</ul>
The changes I've described here are just a snapshot of what we have
lined up today for a 5.0.0 release. 5.0.0 is still under active
development (patches welcome!) so this list will change by the time
the actual release is done.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com8tag:blogger.com,1999:blog-8623074010562846957.post-46550358485217010492014-08-30T15:51:00.002-04:002014-12-09T04:31:42.574-05:00Scoring tennis using finite-state automataFor some reason having to do
with <a href="http://en.wikipedia.org/wiki/France_in_the_Middle_Ages">the medieval
French</a>,
the <a href="http://en.wikipedia.org/wiki/Tennis_scoring_system">scoring
system for tennis is very strange</a>.
<br><br>
In actuality, the game is easy to explain: to win, you must score at
least 4 points and win by at least 2. Yet in practice, you are
supposed to use strange labels like "love" (0 points), "15" (1 point),
"30" (2 points), "40" (3 points), "deuce" (3 or more points each, and
the players are tied), "all" (players are tied) instead of simply
tracking points as numbers, as other sports do.
<br><br>
This is of course wildly confusing to newcomers. Fortunately, the
convoluted logic is easy to express as
a <a href="http://blog.mikemccandless.com/2012/05/finite-state-automata-in-lucene.html">finite-state
automaton</a> (FSA):
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKbo20tG8v6jh3ZVILkRwOb4Y1e9RRI3XGDyNtWErepzN8SXd4sV3OKXa0XbvqkPqbpiSwj8KaiSUSZ-9YHN4Y7UdhQNUoQO2BLLjAVy3UW3FiZr4ind-OcCTzb6Ez88tLfvevbnFjF8Lw/s1600/normal.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKbo20tG8v6jh3ZVILkRwOb4Y1e9RRI3XGDyNtWErepzN8SXd4sV3OKXa0XbvqkPqbpiSwj8KaiSUSZ-9YHN4Y7UdhQNUoQO2BLLjAVy3UW3FiZr4ind-OcCTzb6Ez88tLfvevbnFjF8Lw/s1600/normal.png" /></a>
<br><br>
The game begins in the left-most (unlabeled) state, and then each time
either player 1 (<font color=red>red</font>) or player 2
(<font color=blue>blue</font>) scores, you advance to the
corresponding state to know how to say the score properly in
tennis-speak. In each state, player 1's score is first followed by
player 2's; for example "40 30" means player 1 has scored 3 points and
player 2 has scored 2 and "15 all" means both players have scored
once. "adv 2" means player 2 is ahead by 1 point and will win if s/he
scores again.
<br><br>
There are only 20 states, and there are cycles
which means a tennis game can in fact go on indefinitely, if the
players pass back and forth through the "deuce" (translation: game is
tied) state.
<br><br>
This FSA is correct, and if you watch
a <a href="http://www.wimbledon.com/index.html">Wimbledon match</a>,
for example, you'll see the game advance through precisely these
states.
<br><br>
<font size=+2><b>Minimization</b></font>
<br><br>
Yet for an FSA, merely being correct is not good enough!
<br><br>
It should
also strive to be minimal, and surprisingly this FSA is not: if you
build this Automaton in <a href="http://lucene.apache.org">Lucene</a>
and minimize it, you'll discover that there are some wasted states!
This means 20 states is overkill when deciding who won the game.
<br><br>
Specifically, there is no difference between the "30 all" and "deuce"
states, nor between the "30 40" and "adv 2" states, nor between the
"40 30" and "adv 1" states. From either state in each of these pairs,
there is no sequence of player 1 / player 2 scoring that will result
in a different final outcome (this is in principle how the minimization
process identifies indistinguishable states).
<br><br>
Therefore, there's no point in keeping those states, and you can
safely use this smaller 17-state FSA (15% smaller!) to score your
tennis games instead:
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXOq3e0oR9xbH2WZy4RRRgOacmxEwZn-Va2LaExQPc-1HcJJZojCBTHi0BRsK_1njI3A1cuZaqKLIZJnta9V9cWj2wWVcWjrps1ozc7RxLcpZrMMiD-tM9USWxt69W-9Y6xv9qlhO07XgN/s1600/min.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXOq3e0oR9xbH2WZy4RRRgOacmxEwZn-Va2LaExQPc-1HcJJZojCBTHi0BRsK_1njI3A1cuZaqKLIZJnta9V9cWj2wWVcWjrps1ozc7RxLcpZrMMiD-tM9USWxt69W-9Y6xv9qlhO07XgN/s1600/min.png" /></a>
<br><br>
For example, from "15 30", if player 1 scores, you go straight to
"deuce" and don't bother with the redundant "30 30" state.
<br><br>
Another (simpler?) way to understand why these states are wasted is to
recognize that the finite state machine is tracking two different pieces
of information: first, how many points ahead player 1 is (since a
player must win by 2 points) and second, how many points have been
scored (since a player must score at least 4 points to win).
<br><br>
Once enough points (4 or more) have been scored by either player,
their absolute scores no longer matter. All that matters is the
relative score: whether player 1 is ahead by 1, equal, or behind by 1.
For example, we don't care if the score is 197 to 196 or 6 to 5: they
are the same thing.
<br><br>
Yet, early on, the FSA must also track the absolute scores, to ensure
at least 4 points were scored by the winner. With the original
20-state FSA, the crossover between these two phases was what would
have been "40 40" (each player scored 3 points). But in the minimal
machine, the crossover became "30 30" (each player scored 2 points),
which is safe since each player must still "win by 2" so if player 1
scores 2 points from "30 30", that means player 1 scored 4 points
overall.
<br><br>
FSA minimization saved only 3 states for the game of tennis, resulting
in a 15% smaller automaton, and maybe this simplifies keeping track of
scores in your games by a bit, but in
other <a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">FSA
applications in Lucene</a>, such
as <a href="http://blog.mikemccandless.com/2012/09/lucenes-new-analyzing-suggester.html">the
analyzing
suggester</a>, <a href="http://blog.mikemccandless.com/2011/06/primary-key-lookups-are-28x-faster-with.html">MemoryPostingsFormat</a>
and the terms index, minimization is vital since it saves substantial disk and RAM for
Lucene applications!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com7tag:blogger.com,1999:blog-8623074010562846957.post-10552457983284427972014-08-03T05:43:00.000-04:002014-08-03T05:43:31.217-04:00A new proximity query for Lucene, using automatonsThe simplest <a href="http://lucene.apache.org">Apache Lucene</a> query, <code>TermQuery</code>, matches any document that contains the specified term, regardless of <em>where</em> the term occurs inside each document. Using <code>BooleanQuery</code> you can combine multiple <code>TermQuery</code>s, with full control over which terms
are optional (<code>SHOULD</code>) and which are required (<code>MUST</code>) or required not to be present (<code>MUST_NOT</code>), but still the matching ignores the relative positions of each term inside the document.
<br><br>
Sometimes you do care about the positions of the terms, and for such cases Lucene has various so-called <em>proximity</em> queries.
<br><br>
The simplest proximity query is <code>PhraseQuery</code>, to match a specific sequence of tokens such as "Barack Obama". Seen as a graph, a <code>PhraseQuery</code> is a simple linear chain:
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK7gCj74ProLJp_x5BTId605iE4_5-uMsh-IV6aTP2Av2fwgzpYgflH_c4WvPxh-Y33fC1aIRxOskQVqzYRBJf5_AORtcvgvgbrJ3UpxptsRL_NPY5MAEe6LoMbrmdivwUqexWaJ-JJh16/s1600/barack_obama1.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK7gCj74ProLJp_x5BTId605iE4_5-uMsh-IV6aTP2Av2fwgzpYgflH_c4WvPxh-Y33fC1aIRxOskQVqzYRBJf5_AORtcvgvgbrJ3UpxptsRL_NPY5MAEe6LoMbrmdivwUqexWaJ-JJh16/s1600/barack_obama1.png" /></a>
<br><br>
By default the phrase must precisely match, but if you set a non-zero <em>slop factor</em>, a document can still match even when the tokens are not exactly in sequence, as long as the <a href="http://en.wikipedia.org/wiki/Levenshtein_distance">edit distance</a> is within the specified slop. For example, "Barack Obama" with a slop factor of 1 will also match a document containing "Barack Hussein Obama" or "Barack H. Obama". It looks like this graph:
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2KdgorJY-m7BgfgIWJ_5PhS9P1Or7L9oJPRgAZRsvnLmi4tfJ1AP6eMBZdpDCik8G1JsS1h1U9Ler_VVG_QQIrGD5f-81YcXl3sGoIH0HixcgpWclQLb_mRHHaYCKpmFYe-g_Cd13GQ51/s1600/barack_obama.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2KdgorJY-m7BgfgIWJ_5PhS9P1Or7L9oJPRgAZRsvnLmi4tfJ1AP6eMBZdpDCik8G1JsS1h1U9Ler_VVG_QQIrGD5f-81YcXl3sGoIH0HixcgpWclQLb_mRHHaYCKpmFYe-g_Cd13GQ51/s1600/barack_obama.png" /></a>
<br><br>
Now there are multiple paths through the graph, including an <b>any</b> (<code>*</code>) transition to match an arbitrary token. (Note: while the graph cannot properly express it, this query would also match a document that had the tokens Barack and Obama on top of one another, at the same position, which is a little bit strange!)
<br><br>
In general, proximity queries are more costly on both CPU and IO resources, since they must load, decode and visit another dimension (positions) for each potential document hit. That said, for exact (no slop) matches, using common-grams, shingles and ngrams to index additional "proximity terms" in the index can provide enormous
performance improvements in some cases, at the expense of an increase in index size.
<br><br>
<code>MultiPhraseQuery</code> is another proximity query. It generalizes <code>PhraseQuery</code> by allowing more than one token at each position, for example:
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZod2TsCOjkeuyVxfdZKyCBb9vWULQWtxMuER-XlcBLUn2qvFevR69fspDMXvMTejlrukAShyphenhyphenFBOl2teWBrcu71gzLeBp4sZ0io_vHDwoIDAbqOBLWJ9hN3N0QbQXVYRT4akMSnrGsltux/s1600/dns.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZod2TsCOjkeuyVxfdZKyCBb9vWULQWtxMuER-XlcBLUn2qvFevR69fspDMXvMTejlrukAShyphenhyphenFBOl2teWBrcu71gzLeBp4sZ0io_vHDwoIDAbqOBLWJ9hN3N0QbQXVYRT4akMSnrGsltux/s1600/dns.png" /></a>
<br><br>
This matches any document containing either <code>domain name system</code> or <code>domain name service</code>. <code>MultiPhraseQuery</code> also accepts a slop factor to allow for non-precise matches.
<br><br>
Finally, span queries (e.g. <code>SpanNearQuery</code>, <code>SpanFirstQuery</code>) go even further, allowing you to build up a complex compound query based on positions where each clause matched. What makes them unique is that you can arbitrarily nest them. For example, you could first build a <code>SpanNearQuery</code> matching Barack Obama with slop=1, then another one matching George Bush, and then make another <code>SpanNearQuery</code>, containing both of those as sub-clauses, matching if they appear within 10 terms of one another.
<br><br>
<b>Introducing TermAutomatonQuery</b>
<br><br>
As of Lucene 4.10 there will be a <a href="https://issues.apache.org/jira/browse/LUCENE-5815">new proximity query</a> to further generalize on <code>MultiPhraseQuery</code> and the span queries: it allows you to directly build an arbitrary automaton expressing how the terms must occur in sequence, including <b>any</b> transitions to handle slop. Here's an example:
<br><br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8n8HQsE1zQwfu19PHYtqjTmJSFtNAayzeNEUtJtSeyUvsRwQt9sjjAe5IxAxmwhSfJj8cRvgSsYvYg4uDFQAbBomBci9e1C7E6gAy1yVe6D33nPJqrTQ1miXe0pGXNNrhujRJwlwuI76/s1600/barack_obama3.png" imageanchor="1" ><img width=600 border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8n8HQsE1zQwfu19PHYtqjTmJSFtNAayzeNEUtJtSeyUvsRwQt9sjjAe5IxAxmwhSfJj8cRvgSsYvYg4uDFQAbBomBci9e1C7E6gAy1yVe6D33nPJqrTQ1miXe0pGXNNrhujRJwlwuI76/s1600/barack_obama3.png" /></a>
<br><br>
This is a very expert query, allowing you fine control over exactly what sequence of tokens constitutes a match. You build the automaton state-by-state and transition-by-transition, including explicitly adding <b>any</b> transitions (sorry, no <code>QueryParser</code> support yet, patches welcome!). Once that's done, the query determinizes the automaton and then uses the same infrastructure (e.g. <code>CompiledAutomaton</code>) that queries like <a href="http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html">FuzzyQuery</a> use for fast term matching, but applied to term positions instead of term bytes. The query is naively scored like a phrase query, which may not be ideal in some cases.
<br><br>
In addition to this new query there is also a simple utility class, <code>TokenStreamToTermAutomatonQuery</code>, that provides loss-less translation of any graph <code>TokenStream</code> into the equivalent <code>TermAutomatonQuery</code>. This is powerful because it means even <a href="http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html">arbitrary token stream graphs</a> will be correctly represented at search time, preserving the <code>PositionLengthAttribute</code> that some tokenizers now set.
<br><br>
While this means you can finally correctly apply arbitrary token stream graph synonyms at query-time, because the index still does not store <code>PositionLengthAttribute</code>, index-time synonyms are <a href="http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html">still not fully correct</a>. That said, it would be simple to build a <code>TokenFilter</code> that writes the position length into a payload, and then to extend the new <code>TermAutomatonQuery</code> to read from the payload and apply that length during matching (patches welcome!).
<br><br>
The query is likely quite slow, because it assumes every term is optional; in many cases it would be easy to determine required terms (e.g. Obama in the above example) and optimize such cases. In the case where the query was derived from a token stream, so that it has no cycles and does not use <b>any</b> transitions, it may be faster to enumerate all phrases accepted by the automaton (Lucene already has the <code>getFiniteStrings</code> API to do this for any automaton) and construct a boolean query from those phrase queries. This would match the same set of documents, also correctly preserving <code>PositionLengthAttribute</code>, but would assign different scores.
<br><br>
The code is very new and there are surely some exciting bugs! But it should be a nice start for any application that needs precise control over where terms occur inside documents.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com7tag:blogger.com,1999:blog-8623074010562846957.post-43880092823565189382014-05-12T07:30:00.000-04:002014-05-12T07:30:18.634-04:00Choosing a fast unique identifier (UUID) for Lucene<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("visualization", "1", {packages:["corechart"]});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = google.visualization.arrayToDataTable(
[['ID Source', 'K lookups/sec, 1 thread'], ['Zero-pad sequential', 593.4], ['UUID v1 [binary]', 509.6], ['Nanotime', 461.8], ['UUID v1', 430.3], ['Sequential', 415.6], ['Flake [binary]', 338.5], ['Flake', 231.3], ['UUID v4 [binary]', 157.8], ['UUID v4', 149.4]]
);
var options = {
title: 'UUID K lookups/sec, 1 thread',
titleTextStyle: {fontSize: 20},
// vAxis: {title: 'Algorithm', titleTextStyle: {fontSize: 20}},
// hAxis: {title: 'K lookups/sec', titleTextStyle: {fontSize: 20}},
legend: {position: 'none'}
};
var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
chart.draw(data, options);
}
</script>
Most search applications using <a href="http://lucene.apache.org">Apache Lucene</a> assign a unique id, or primary key, to each indexed document. While Lucene itself does not require this (it could care less!), the application usually needs it to later replace, delete or retrieve that one document by its external id. Most servers built on top of Lucene, such as <a href="http://elasticsearch.org">Elasticsearch</a> and <a href="http://lucene.apache.org/solr">Solr</a>, require a unique id and can auto-generate one if you do not provide it.
<br><br>
Sometimes your id values are already pre-defined, for example if an external database or content management system assigned one, or if you must use a <a href="http://en.wikipedia.org/wiki/Uniform_resource_identifier">URI</a>, but if you are free to assign your own ids then what works best for Lucene?
<br><br>
One obvious choice is Java's <a href="http://docs.oracle.com/javase/7/docs/api/java/util/UUID.html">UUID</a> class, which generates <a href="http://en.wikipedia.org/wiki/Universally_unique_identifier">version 4 universally unique identifiers</a>, but it turns out this is the worst choice for performance: it is 4X slower than the fastest. To understand why requires some understanding of how Lucene finds terms.
<br><br>
<b>BlockTree terms dictionary</b>
<br><br>
The purpose of the terms dictionary is to store all unique terms seen during indexing, and map each term to its metadata (<a href="http://blog.mikemccandless.com/2012/03/new-index-statistics-in-lucene-40.html"><code>docFreq</code>, <code>totalTermFreq</code>, etc.</a>), as well as the postings (documents, offsets, postings and payloads). When a term is requested, the terms dictionary must locate it in the on-disk index and return its metadata.
<br><br>
The default codec uses the <a href="http://lucene.apache.org/core/4_8_0/core/org/apache/lucene/codecs/BlockTreeTermsWriter.html">BlockTree terms dictionary</a>, which stores all terms for each field in sorted binary order, and assigns the terms into blocks sharing a common prefix. Each block contains between 25 and 48 terms by default. It uses an in-memory prefix-trie index structure (an <a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">FST</a>) to quickly map each prefix to the corresponding on-disk block, and on lookup it first checks the index based on the requested term's prefix, and then seeks to the appropriate on-disk block and scans to find the term.
<br><br>
In certain cases, when the terms in a segment have a predictable pattern, the terms index can know that the requested term cannot exist on-disk. This fast-match test can be a sizable performance gain especially when the index is cold (the pages are not cached by the the OS's IO cache) since it avoids a costly disk-seek. As Lucene is segment-based, a single id lookup must visit each segment until it finds a match, so quickly ruling out one or more segments can be a big win. It is also vital to keep your segment counts as low as possible!
<br><br>
Given this, fully random ids (like <a href="http://en.wikipedia.org/wiki/Universally_unique_identifier">UUID V4</a>) should perform worst, because they defeat the terms index fast-match test and require a disk seek for every segment. Ids with a predictable per-segment pattern, such as sequentially assigned values, or a timestamp, should perform best as they will maximize the gains from the terms index fast-match test.
<br><br>
<b>Testing Performance</b>
<br><br>
I created a simple performance tester to verify this; the full source code is <a href="https://code.google.com/a/apache-extras.org/p/luceneutil/source/browse/src/main/perf/IDPerfTest.java">here</a>. The test first indexes 100 million ids into an index with 7/7/8 segment structure (7 big segments, 7 medium segments, 8 small segments), and then searches for a random subset of 2 million of the IDs, recording the best time of 5 runs. I used Java 1.7.0_55, on Ubuntu 14.04, with a 3.5 GHz Ivy Bridge Core i7 3770K.
<br><br>
Since Lucene's terms are now <a href="http://blog.mikemccandless.com/2012/07/lucene-400-alpha-at-long-last.html">fully binary as of 4.0</a>, the most compact way to store any value is in binary form where all 256 values of every byte are used. A 128-bit id value then requires 16 bytes.
<br><br>
I tested the following identifier sources:
<ul>
<li> Sequential IDs (0, 1, 2, ...), binary encoded.
<br><br>
<li> Zero-padded sequential IDs (00000000, 00000001, ...), binary encoded.
<br><br>
<li> Nanotime, binary encoded. But remember that <a href="http://shipilev.net/blog/2014/nanotrusting-nanotime">nanotime is tricky</a>.
<br><br>
<li> <a href="http://en.wikipedia.org/wiki/Universally_unique_identifier">UUID V1</a>, derived from a timestamp, nodeID and sequence counter, using <a href="http://johannburkard.de/software/uuid">this implementation</a>.
<br><br>
<li> <a href="http://en.wikipedia.org/wiki/Universally_unique_identifier">UUID V4</a>, randomly generated using Java's <code>UUID.randomUUID()</code>.
<br><br>
<li> <a href="http://boundary.com/blog/2012/01/12/flake-a-decentralized-k-ordered-unique-id-generator-in-erlang">Flake IDs</a>, using <a href="https://github.com/mumrah/flake-java">this implementation</a>.
<br><br>
</ul>
For the UUIDs and Flake IDs I also tested binary encoding in addition to their standard (base 16 or 36) encoding. Note that I only tested lookup speed using one thread, but the results should scale linearly (on sufficiently concurrent hardware) as you add threads.
<div id="chart_div" style="width: 800px; height: 500px;"></div>
Zero-padded sequential ids, encoded in binary are fastest, quite a bit faster than non-zero-padded sequential ids. UUID V4 (using Java's <code>UUID.randomUUID()</code>) is ~4X slower.
<br><br>
But for most applications, sequential ids are not practical. The 2nd fastest is <a href="http://en.wikipedia.org/wiki/Universally_unique_identifier">UUID V1</a>, encoded in binary. I was surprised this is so much faster than <a href="http://boundary.com/blog/2012/01/12/flake-a-decentralized-k-ordered-unique-id-generator-in-erlang">Flake IDs</a> since Flake IDs use the same raw sources of information (time, node id, sequence) but shuffle the bits differently to preserve total ordering. I suspect the problem is the number of common leading digits that must be traversed in a Flake ID before you get to digits that differ across documents, since the high order bits of the 64-bit timestamp come first, whereas UUID V1 places the low order bits of the 64-bit timestamp first. Perhaps the terms index should optimize the case when all terms in one field share a common prefix.
<br><br>
I also separately tested varying the base from 10, 16, 36, 64, 256 and in general for the non-random ids, higher bases are faster. I was pleasantly surprised by this because I expected a base matching the BlockTree block size (25 to 48) would be best.
<br><br>
There are some important caveats to this test (patches welcome)! A real application would obviously be doing much more work than simply looking up ids, and the results may be different as hotspot must compile much more active code. The index is fully hot in my test (plenty of RAM to hold the entire index); for a cold index I would expect the results to be even more stark since avoiding a disk-seek becomes so much more important. In a real application, the ids using timestamps would be more spread apart in time; I could "simulate" this myself by faking the timestamps over a wider range. Perhaps this would close the gap between UUID V1 and Flake IDs? I used only one thread during indexing, but a real application with multiple indexing threads would spread out the ids across multiple segments at once.
<br><br>
I used Lucene's default <a href="http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">TieredMergePolicy</a>, but it is possible a smarter merge policy that favored merging segments whose ids were more "similar" might give better results. The test does not do any deletes/updates, which would require more work during lookup since a given id may be in more than one segment if it had been updated (just deleted in all but one of them).
<br><br>
Finally, I used using Lucene's default Codec, but we have nice postings formats optimized for primary-key lookups when you are willing to trade RAM for faster lookups, such as <a href="http://blog.mikemccandless.com/2013/09/lucene-now-has-in-memory-terms.html">this Google summer-of-code project from last year</a> and <a href="http://blog.mikemccandless.com/2011/06/primary-key-lookups-are-28x-faster-with.html">MemoryPostingsFormat</a>. Likely these would provide sizable performance gains!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com21tag:blogger.com,1999:blog-8623074010562846957.post-38282158815970151842014-04-11T11:47:00.001-04:002014-04-11T16:41:39.545-04:00Testing Lucene's index durability after crash or power lossOne of Lucene's useful <a href="http://blog.mikemccandless.com/2012/03/transactional-lucene.html">transactional features</a> is <em>index durability</em> which ensures that, once you successfully call <code>IndexWriter.commit</code>, even if the OS or JVM crashes or power is lost, or you <a href="http://en.wikipedia.org/wiki/Kill_(command)">kill -KILL</a> your JVM process, after rebooting, the index will be intact (not corrupt) and will reflect the last successful commit before the crash.
<br><br>
Of course, this only works if your hardware is healthy and your IO devices implement fsync properly (flush their write caches when asked by the OS). If you have data-loss issues, such as a silent bit-flipper in your memory, IO or CPU paths, thanks to the <a href="https://plus.google.com/112759599082866346694/posts/9ZahCzSWyfY">new end-to-end checksum feature</a> (<a href="https://issues.apache.org/jira/browse/LUCENE-2446">LUCENE-2446</a>), available as of Lucene 4.8.0, Lucene will now detect that as well during indexing or <code>CheckIndex</code>. This is similar to the <a href="http://en.wikipedia.org/wiki/ZFS">ZFS file system</a>'s block-level checksums, but not everyone uses ZFS yet (heh), and so Lucene now does its own checksum verification on top of the file system.
<br><br>
Be sure to enable checksum verification during merge by calling <code>IndexWriterConfig.setCheckIntegrityAtMerge</code>. In the future we'd like to remove that option and always validate checksums on merge, and we've already done so for the default stored fields format in <a href="https://issues.apache.org/jira/browse/LUCENE-5580">LUCENE-5580</a> and (soon) term vectors format in <a href="https://issues.apache.org/jira/browse/LUCENE-5602">LUCENE-5602</a>, as well as set up the low-level IO APIs so other codec components can do so as well, with <a href="https://issues.apache.org/jira/browse/LUCENE-5583">LUCENE-5583</a>, for Lucene 4.8.0.
<br><br>
<b>FileDescriptor.sync and fsync</b>
<br><br>
Under the hood, when you call <code>IndexWriter.commit</code>, Lucene gathers up all newly written filenames since the last commit, and invokes <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FileDescriptor.html#sync()">FileDescriptor.sync</a> on each one to ensure all changes are moved to stable storage.
<br><br>
At its heart, <a href="http://en.wikipedia.org/wiki/Sync_(Unix)">fsync</a> is a complex operation, as the OS must flush any dirty pages associated with the specified file from its IO buffer cache, work with the underlying IO device(s) to ensure their write caches are also flushed, and also work with the file system to ensure its integrity is preserved. You can separately fsync the bytes or metadata for a file, and also the directory(ies) containing the file. <a href="http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/">This blog post</a> is a good description of the challenges.
<br><br>
Recently we've been scrutinizing these parts of Lucene, and all this attention has uncovered some exciting issues!
<br><br>
In <a href="https://issues.apache.org/jira/browse/LUCENE-5570">LUCENE-5570</a>, to be fixed in Lucene 4.7.2, we discovered that the fsync implementation in our <code>FSDirectory</code> implementations is able to bring new 0-byte files into existence. This normally isn't a problem by itself, because <code>IndexWriter</code> shouldn't fsync a file that it didn't create. However, it exacerbates debugging when there is a bug in <code>IndexWriter</code> or in the application using Lucene (e.g., directly deleting index files that it shouldn't). In these cases it's confusing to discover these 0-byte files so much later, versus hitting a <code>FileNotFoundException</code> at the point when <code>IndexWriter</code> tried to fsync them.
<br><br>
In <a href="https://issues.apache.org/jira/browse/LUCENE-5588">LUCENE-5588</a>, to be fixed in Lucene 4.8.0, we realized we must also fsync the directory holding the index, otherwise it's possible on an OS crash or power loss that the directory won't link to the newly created files or that you won't be able to find your file by its name. This is clearly important because Lucene lists the directory to locate all the commit points (<code>segments_N</code> files), and of course also opens files by their names.
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuunCfkXoimsuJ6Ko2TLwwF9Ve6Z_chvHewf9zlbuTSc-K1KOCYwAVJ3yzCeT_plyOf6SOuibVVxVKGrxBSfqvarOzvp_qbT7avyE-PIRdOuxWgcJLbtXrlkU4tLA9T56nmnJtfJ7PwYlP/s1600/hardDrive.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuunCfkXoimsuJ6Ko2TLwwF9Ve6Z_chvHewf9zlbuTSc-K1KOCYwAVJ3yzCeT_plyOf6SOuibVVxVKGrxBSfqvarOzvp_qbT7avyE-PIRdOuxWgcJLbtXrlkU4tLA9T56nmnJtfJ7PwYlP/s320/hardDrive.jpg" /></a></div>
<br><br>
Since Lucene does not rely on file metadata like access time and modify time, it is tempting to use <a href="http://man7.org/linux/man-pages/man2/fsync.2.html">fdatasync</a> (or <a href="http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#force(boolean)">FileChannel.force(false)</a> from java) to fsync just the file's bytes. However, this is an optimization and at this point we're focusing on bugs. Furthermore, it's likely this won't be any faster since the metadata must still be sync'd by <code>fdatasync</code> if the file length has changed, which is always the case in Lucene since we only append to files when writing (we removed <code>Indexoutput.seek</code> in <a href="https://issues.apache.org/jira/browse/LUCENE-4399">LUCENE-4399</a>).
<br><br>
In <a href="https://issues.apache.org/jira/browse/LUCENE-5574">LUCENE-5574</a>, to be fixed as of Lucene 4.7.2, we found that a near-real-time reader, on closing, could delete files even if the writer it was opened from has been closed. This is normally not a problem by itself, because Lucene is write-once (never writes to the same file name more than once), as long as you use Lucene's APIs and don't modify the index files yourself. However, if you implement your own index replication by copying files into the index, and if you don't first close your near-real-time readers, then it is possible closing them would remove the files you had just copied.
<br><br>
During any given indexing session, Lucene writes many files and closes them, many files are deleted after being merged, etc., and only later, when the application finally calls <code>IndexWriter.commit</code>, will <code>IndexWriter</code> then re-open the newly created files in order to obtain a <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FileDescriptor.html">FileDescriptor</a> so we can <code>fsync</code> them.
<br><br>
This approach (closing the original file, and then opening it again later in order to sync), versus never closing the original file and syncing that same file handle you used for writing, is perhaps risky: the javadocs for <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FileDescriptor.html#sync()">FileDescriptor.sync</a> are somewhat vague as to whether this approach is safe. However, when we check the documentation for <a href="http://linux.die.net/man/2/fsync">fsync</a> on Unix/Posix and <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa364439(v=vs.85).aspx">FlushFileBuffers</a> on Windows, they make it clear that this practice is fine, in that the open file descriptor is really only necessary to identify which file's buffers need to be sync'd. It's also hard to imagine an OS that would separately track which open file descriptors had made which changes to the file. Nevertheless, out of paranoia or an abundance of caution, we are also exploring a possible patch on <a href="https://issues.apache.org/jira/browse/LUCENE-3237">LUCENE-3237</a> to fsync only the originally opened files.
<br><br>
<b>Testing that fsync really works</b>
<br><br>
With all these complex layers in between your application's call to <code>IndexWriter.commit</code> and the laws of physics ensuring <a href="http://en.wikipedia.org/wiki/Hard_disk_drive">little magnets were flipped</a> or a <a href="http://en.wikipedia.org/wiki/Flash_memory">few electrons were moved into a tiny floating gate in a NAND cell</a>, how can we reliably test that the whole series of abstractions is actually working?
<br><br>
In Lucene's <a href="http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html">randomized testing framework</a> we have a nice evil <code>Directory</code> implementation called <code>MockDirectoryWrapper</code>. It can do all sorts of nasty things like throw random exceptions, sometimes slow down opening, closing and writing of some files, refuse to delete still-open files (like Windows), refuse to close when there are still open files, etc. This has helped us find all sorts of fun bugs over time.
<br><br>
Another thing it does on close is to simulate an OS crash or power loss by randomly corrupting any un-sycn'd files and then confirming the index is not corrupt. This is useful for catching Lucene bugs where we are failing to call fsync when we should, but it won't catch bugs in our implementation of sync in our <code>FSDirectory</code> classes, such as the frustrating <a href="https://issues.apache.org/jira/browse/LUCENE-3418">LUCENE-3418</a> (first appeared in Lucene 3.1 and finally fixed in Lucene 3.4).
<div class="separator" style="clear: both; text-align: center;"><a href="http://cache.smarthome.com/images/2456s3.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://cache.smarthome.com/images/2456s3.jpg" /></a></div>
<br><br>
So, to catch such bugs, I've created a basic test setup, making use of a simple <a href="https://www.insteon.com/">Insteon</a> <a href="https://www.insteon.com/2456s3-appliancelinc.html">on/off</a> device, along with custom Python bindings I created long ago to interact with Insteon devices. I already use these devices all over my home for controlling lights and appliances, so also using this for Lucene is a nice intersection of two of my passions!
<br><br>
The script loops forever, first updating the sources, compiling, checking the index for corruption, then kicking off an indexing run with some randomization in the settings, and finally, waiting a few minutes and then cutting power to the box. Then, it restores power, waits for the machine to be responsive again, and starts again.
<br><br>
So far it's done 80 power cycles and no corruption yet. Good news!
<br><br>
To "test the tester", I tried temporarily changing fsync to do nothing, and indeed after a couple iterations, the index became corrupt. So indeed the test setup seems to "work".
<br><br>
Currently the test uses Linux on a spinning magnets hard drive with the <a href="http://en.wikipedia.org/wiki/Ext4">ext4 file system</a>. This is just a start, but it's better than no proper testing for Lucene's fsync. Over time I hope to test different combinations of OS's, file systems, IO hardware, etc.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com2tag:blogger.com,1999:blog-8623074010562846957.post-37221629972999957932014-03-05T11:36:00.000-05:002014-03-05T11:36:35.663-05:00Using Lucene's search server to search Jira issuesYou may remember
my <a href="http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html">first
blog post</a> describing how the Lucene developers eat our own dog
food by using a Lucene search application
to <a href="http://jirasearch.mikemccandless.com">find our Jira
issues</a>.
<br><br>
That application has become a powerful showcase of a number of modern Lucene
features such
as <a href="http://blog.mikemccandless.com/2013/02/drill-sideways-faceting-with-lucene.html">drill
sideways</a>
and <a href="http://blog.mikemccandless.com/2013/05/dynamic-faceting-with-lucene.html">dynamic
range</a>
faceting, <a href="http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html">a
new suggester based on infix
matches</a>, <a href="http://blog.mikemccandless.com/2012/12/a-new-lucene-highlighter-is-born.html">postings
highlighter</a>, block-join queries so you can jump to a specific
issue comment that matched your search, near-real-time indexing and
searching, etc. Whenever new users ask me about Lucene's capabilities, I point them to this application so they can see for themselves.
<br><br>
Recently, I've made some further progress so I want to give an update.
<br><br>
The source code for the
simple <a href="http://netty.io/">Netty</a>-based Lucene server is now
available
on <a href="https://svn.apache.org/repos/asf/lucene/dev/branches/lucene5376">this
subversion branch</a>
(see <a href="https://issues.apache.org/jira/browse/LUCENE-5376">LUCENE-5376</a>
for details). I've been gradually adding coverage for additional
Lucene modules, including facets, suggesters, analysis, queryparsers, highlighting, grouping,
joins and expressions. And of course normal indexing and searching!
Much remains to be done (there are plenty of nocommits), and the goal
here is not to build a feature rich search server but rather to
demonstrate how to use Lucene's current modules in a server context
with minimal "thin server" additional source code.
<br><br>
Separately, to test this new Lucene based server, and to complete the
"dog food," I built a simple Jira search application plugin, to help
us find Jira
issues, <a href="http://jirasearch.mikemccandless.com">here</a>. This
application has various Python tools to extract and index Jira issues
using <a href="https://docs.atlassian.com/jira/REST/latest/">Jira's
REST API</a> and a user-interface layer running
as a <a href="http://wsgi.readthedocs.org/en/latest/">Python WSGI
app</a>, to send requests to the server and render responses back to
the user. The goal of this Jira search application is to make it
simple to point it at any Jira instance / project and enable full
searching over all issues.
<br><br>
I just pushed some further changes to
the <a href="http://jirasearch.mikemccandless.com">production
site</a>:
<ul>
<li> I upgraded the Jira search application to the current server
branch (previously it was running on my private fork).
<br><br>
<li> I switched all analysis components to Lucene's analysis factories;
these factories
use <a href="http://en.wikipedia.org/wiki/Service_provider_interface">Java's
SPI (Service Provider Interface)</a> so that the server has access
to any char filters, tokenizers and token filters in the classpath.
This is very helpful when building a server because it means you
don't need any special code to handle the great many analysis
components that Lucene provides these days. Everything simply
passes through the factories (which know how to parse their own
arguments).
<br><br>
<li> I've added the <a href="http://tika.apache.org">Tika</a> project,
so you can now find Tika issues as well. This was very simple to
add, and seems be working!
<br><br>
<li> I inserted <code>WordDelimiterFilter</code> so that
CamelCaseTokens are split. For example,
try <a href="http://jirasearch.mikemccandless.com/search.py?index=jira&chg=new&text=infix">searching
on infix</a> and note the highlights. As Rober Muir reminded
me, <code>WordDelimiterFilter</code> corrupts offsets, which will
mess up highlighting in some cases, so I'm going to try to set
up <code>ICUTokenizer</code>, which I'm already using, to do this
splitting instead.
<br><br>
<li> I switched to Lucene's new <a href="http://blog.mikemccandless.com/2013/09/three-exciting-lucene-features-in-one.html">expressions module</a> to do blended relevance +
recency sort by default when you do a text search, which is helpful
because most of the time we are looking for recently touched issues.
Previously I used a custom <code>FieldComparator</code> to achieve
the same functionality, but expressions is more compact and powerful
and lets me remove that custom <code>FieldComparator</code>.
<br><br>
<li> I switched to near-real-time building of the suggestions, using
<a href="http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html">AnalyzingInfixSuggester</a>.
Previously I was fully rebuilding the suggester every five minutes,
so this saves a lot of CPU since now I just add new Jira issues as
they come, and refresh the suggester. It also means a much shorter delay from when an index is added to when it can be suggested.
See <a href="https://issues.apache.org/jira/browse/LUCENE-5477">LUCENE-5477</a>
for details.
<br><br>
<li> I now <code>commit</code> once per day. Previously I never
committed, and simply relied on near-real-time searching. This
works just fine, except when I need to bring the server down
(e.g. to push new changes out), it required full reindexing, which
was very fast but a poor user experience for those users who
happened to do a search while it was happening. Now, when I bounce
the server it comes back to the last commit and then the
near-real-time indexing quickly catches up on any changed issues
since that last commit.
<br><br>
<li> Various small issues, such as proper handling when a Jira issue
is renamed (the Jira REST API does not make it so easy to discover
this!); better production push automation; upgraded to a newer
version of <a href="http://getbootstrap.com">bootstrap</a> UI
library.
<br><br>
</ul>
There are still plenty of improvements to make to this Jira search application. For fields with many
possible drill-down values, I'd like to have a simple suggester so the
user can quickly drill down. I'd like to fix the suggester to filter
suggestions according to the project. For example, if you've drilled
down into Tika issues, then when you type a new search you should see
only Tika issues suggested. For that we need
to <a href="https://issues.apache.org/jira/browse/LUCENE-5350">make
AnalzyingInfixSuggester context aware</a>. I'd also like a more
compact UI for all of the facet fields; maybe I need to hide the less
commonly used facet fields under a "More"...
<br><br>
Please send me any feedback / problems when you're searching for
issues!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com2tag:blogger.com,1999:blog-8623074010562846957.post-54681646086384501052014-01-23T09:54:00.000-05:002014-01-24T06:33:03.419-05:00Finding long tail suggestions using Lucene's new FreeTextSuggesterLucene's <a href="http://lucene.apache.org/core/4_6_0/suggest/index.html">suggest module</a> offers a number of fun auto-suggest implementations to give a user live search suggestions as they type each character into a search box.
<br><br>
For example, <code>WFSTCompletionLookup</code> compiles all suggestions and their weights into a compact <a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">Finite State Transducer</a>, enabling fast prefix lookup for basic suggestions.
<br><br>
<a href="http://blog.mikemccandless.com/2012/09/lucenes-new-analyzing-suggester.html"><code>AnalyzingSuggester</code></a> improves on this by using an <code>Analyzer</code> to normalize both the suggestions and the user's query so that trivial differences in whitespace, casing, stop-words, synonyms, as determined by the analyzer, do not prevent a suggestion from matching.
<br><br>
Finally, <code>AnalyzingInfixSuggester</code> goes further by allowing <em>infix</em> matches so that words inside each suggestion (not just the prefix) can trigger a match. You can see this one action at <a href="http://jirasearch.mikemccandless.com">the Lucene/Solr Jira search application</a> (e.g., try "python") that I recently created to <a href="http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html">eat our own dog food</a>. It is also the only suggester implementation so far that supports highlighting (this has <a href="https://issues.apache.org/jira/browse/LUCENE-4518">proven challenging</a> for the other suggesters).
<br><br>
Yet, a common limitation to all of these suggesters is that they can only suggest from a finite set of previously built suggestions. This may not be a problem if your suggestions are past user queries and you have tons and tons of them (e.g., you are Google). Alternatively, if your universe of suggestions is inherently closed, such as the movie and show titles that Netflix's search will suggest, or all product names on an e-commerce site, then a closed set of suggestions is appropriate.
<br><br>
<b>N-Gram language models</b>
<br><br>
For everyone else, where a high percentage of the incoming queries fall into the never-seen-before <a href="http://en.wikipedia.org/wiki/Long_tail">long tail</a>, Lucene's newest suggester, <code>FreeTextSuggester</code>, can help! It uses the approach described in this <a href="http://googleblog.blogspot.com/2011/04/more-predictions-in-autocomplete.html">Google blog post</a>.
<br><br>
Rather than precisely matching a previous suggestion, it builds up a simple <a href="http://en.wikipedia.org/wiki/Language_model">statistical n-gram language model</a> from all suggestions and looks at the last tokens (plus the prefix of whatever final token the user is typing, if present) to predict the most likely next token.
<br><br>
For example, perhaps the user's query so far is: "flashforge 3d p", and because flashforge is an uncommon brand of 3D printer, this particular suggestion prefix was never added to the suggester. Yet, "3d printer" was a frequently seen phrase in other contexts (different brands). In this case, <code>FreeTextSuggester</code> will see "3d" and the "p" prefix for the next token and predict printer, even though "flashforge 3d printer" was never explicitly added as a suggestion.
<br><br>
You specify the order (N) of the model when you create the suggester: larger values of N require more data to train properly but can make more accurate predictions. All lower order models are also built, so if you specify N=3, you will get trigrams, bigrams and unigrams, all compiled into a single weighted FST for maximum sharing of the text tokens. Of course, larger N will create much larger FSTs. In practice N=3 is the highest you should go, unless you have tons of both suggestions to train and RAM to hold the resulting FST.
<br><br>
To handle sparse data, where a given context (the N-1 prior words) was not seen frequently enough to make accurate predictions, the suggester uses the <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.76.1126">stupid backoff language model</a> (yes, this is really its name, and yes, it performs well!).
<br><br>
I expect the best way to use this new <code>FreeTextSuggester</code> will be as a fallback: you would first use one of the existing exact match suggesters, but when those suggesters fail to find any suggestions for a given query, because it's "unusual" and has crossed over into the <a href="http://en.wikipedia.org/wiki/Long_tail">long tail</a>, you then fall back to <code>FreeTextSuggester</code>.
<br><br>
<a href="http://www.google.com">Google</a> seems to use such a modal approach to suggestions as well: if you type "flashforge 3d p" you should see something like this, where each suggestion covers your entire query so far (indeed, Google <b>has</b> heard of the flashforge brand of 3d printer!):
<br><br>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2K8s7-YeGq43PbZQ8qBMHXvWvz2YCeZwR8eZ2oeQwgsoA4c3vGzQb2Y_atm3HGIPtj-DV3ZW2D_pFLGodpDK5rRfgAl9cT19jh7P-ViQ6y6ZWNnKd_c3Rl4zky1KZiRMiagTA_A0bchD2/s1600/suggest1.png" /></div>
<br><br>
But then if you keep typing and enter "flashforge 3d printer power u", the suggestions change: instead of suggesting an entire query, matching
everything I have typed, Google instead suggests the last word or two:
<br><br>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcsPvW4IXoez9-FXHrk3A95UFXPYBFvK3wPfBIjuZd7yXY_VoniP1tC6Fiq1j7-rNIy45TYoBp8ZqlXhcNGLBzEm8mWyoB9oFZlohh8X_gapQUVbMAnx8tPcg4T6reL8aigwR1Tl6FNgJx/s1600/suggest2.png" /></div>
<br><br>
As usual, this feature is very new and likely to contain exciting bugs! See the Jira issue, <a href="https://issues.apache.org/jira/browse/LUCENE-5214">LUCENE-5214</a>, for details. If you play with this new suggester please start a discussion on the <a href="mailto:java-user@lucene.apache.org">Lucene's user list</a>!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com7tag:blogger.com,1999:blog-8623074010562846957.post-34001991528167720982014-01-08T17:56:00.001-05:002014-01-08T18:12:17.443-05:00Geospatial (distance) faceting using Lucene's dynamic range facetsThere have been several recent, quiet improvements to Lucene that, taken together, have made it surprisingly simple to add geospatial distance faceting to any Lucene search application, for example:
<pre>
< 1 km (147)
< 2 km (579)
< 5 km (2775)
</pre>
Such distance facets, which allow the user to quickly filter their search results to those that are close to their location, has become especially important lately since most searches are now from mobile smartphones.
<br><br>
In the past, this has been challenging to implement because it's so dynamic and so costly: the facet counts depend on each user's location, and so cannot be cached and shared across users, and the underlying math for spatial distance is complex.
<br><br>
But several recent Lucene improvements now make this surprisingly simple!
<br><br>
First, Lucene's <a href="http://blog.mikemccandless.com/2013/05/dynamic-faceting-with-lucene.html">dynamic range faceting</a> has <a href="https://issues.apache.org/jira/browse/LUCENE-5297">been generalized</a> to accept any <code>ValueSource</code>, not just a numeric doc values field from the index. Thanks to the recently added <a href="http://blog.mikemccandless.com/2013/09/three-exciting-lucene-features-in-one.html">expressions module</a>, this means you can offer dynamic range facets computed from an arbitrary <a href="http://en.wikipedia.org/wiki/JavaScript">JavaScript</a> expression, since the expression is compiled on-the-fly to a <code>ValueSource</code> using custom generated Java bytecodes with <a href="http://asm.ow2.org">ASM</a>. Lucene's range faceting is also faster now, <a href="http://blog.mikemccandless.com/2013/12/fast-range-faceting-using-segment-trees.html">using segment trees to quickly assign each value to the matching ranges</a>.
<br><br>
Second, the <a href="http://en.wikipedia.org/wiki/Haversine_formula">Haversine distance function</a> <a href="https://issues.apache.org/jira/browse/LUCENE-5258">was added</a> to the expressions module. The implementation uses impressively fast approximations to the normally costly trigonometric functions, poached in part from the <a href="https://code.google.com/p/jodk/">Java Optimized Development Kit</a> project, without sacrificing too much accuracy. It's unlikely the approximations will ever matter in practice, and there is <a href="https://issues.apache.org/jira/browse/LUCENE-5271">an open issue</a> to further improve the approximation.
<br><br>
Suddenly, armed with these improvements, if you index latitude and longitude as <code>DoubleDocValuesField</code>s in each document, and you know the user's latitude/longitude location for each request, you can easily compute facet counts and offer drill-downs by any set of chosen distances.
<br><br>
First, index your documents with latitude/longitude fields:
<br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%">1
2
3
4</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%">Document doc <span style="color: #666666">=</span> <span style="color: #AA22FF; font-weight: bold">new</span> Document<span style="color: #666666">();</span>
doc<span style="color: #666666">.</span><span style="color: #BB4444">add</span><span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">new</span> DoubleField<span style="color: #666666">(</span><span style="color: #BB4444">"latitude"</span><span style="color: #666666">,</span> <span style="color: #666666">40.759011,</span> Field<span style="color: #666666">.</span><span style="color: #BB4444">Store</span><span style="color: #666666">.</span><span style="color: #BB4444">NO</span><span style="color: #666666">));</span>
doc<span style="color: #666666">.</span><span style="color: #BB4444">add</span><span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">new</span> DoubleField<span style="color: #666666">(</span><span style="color: #BB4444">"longitude"</span><span style="color: #666666">,</span> <span style="color: #666666">-73.9844722,</span> Field<span style="color: #666666">.</span><span style="color: #BB4444">Store</span><span style="color: #666666">.</span><span style="color: #BB4444">NO</span><span style="color: #666666">));</span>
writer<span style="color: #666666">.</span><span style="color: #BB4444">addDocument</span><span style="color: #666666">(</span>doc<span style="color: #666666">);</span>
</pre></div>
</td></tr></table>
At search time, obtain the <code>ValueSource</code> by building a dynamic expression that invokes the Haversine function:
<br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span style="color: #AA22FF; font-weight: bold">private</span> ValueSource getDistanceValueSource<span style="color: #666666">()</span> <span style="color: #666666">{</span>
Expression distance<span style="color: #666666">;</span>
<span style="color: #AA22FF; font-weight: bold">try</span> <span style="color: #666666">{</span>
distance <span style="color: #666666">=</span> JavascriptCompiler<span style="color: #666666">.</span><span style="color: #BB4444">compile</span><span style="color: #666666">(</span>
<span style="color: #BB4444">"haversin(40.7143528,-74.0059731,latitude,longitude)"</span><span style="color: #666666">);</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">catch</span> <span style="color: #666666">(</span>ParseException pe<span style="color: #666666">)</span> <span style="color: #666666">{</span>
<span style="color: #008800; font-style: italic">// Should not happen</span>
<span style="color: #AA22FF; font-weight: bold">throw</span> <span style="color: #AA22FF; font-weight: bold">new</span> RuntimeException<span style="color: #666666">(</span>pe<span style="color: #666666">);</span>
<span style="color: #666666">}</span>
SimpleBindings bindings <span style="color: #666666">=</span> <span style="color: #AA22FF; font-weight: bold">new</span> SimpleBindings<span style="color: #666666">();</span>
bindings<span style="color: #666666">.</span><span style="color: #BB4444">add</span><span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">new</span> SortField<span style="color: #666666">(</span><span style="color: #BB4444">"latitude"</span><span style="color: #666666">,</span> SortField<span style="color: #666666">.</span><span style="color: #BB4444">Type</span><span style="color: #666666">.</span><span style="color: #BB4444">DOUBLE</span><span style="color: #666666">));</span>
bindings<span style="color: #666666">.</span><span style="color: #BB4444">add</span><span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">new</span> SortField<span style="color: #666666">(</span><span style="color: #BB4444">"longitude"</span><span style="color: #666666">,</span> SortField<span style="color: #666666">.</span><span style="color: #BB4444">Type</span><span style="color: #666666">.</span><span style="color: #BB4444">DOUBLE</span><span style="color: #666666">));</span>
<span style="color: #AA22FF; font-weight: bold">return</span> distance<span style="color: #666666">.</span><span style="color: #BB4444">getValueSource</span><span style="color: #666666">(</span>bindings<span style="color: #666666">);</span>
<span style="color: #666666">}</span>
</pre></div>
</td></tr></table>
Instead of the hardwired latitude/longitude above, you should fill in the user's location.
<br><br>
Using that <code>ValueSource</code>, compute the dynamic facet counts like this:
<br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%">FacetsCollector fc <span style="color: #666666">=</span> <span style="color: #AA22FF; font-weight: bold">new</span> FacetsCollector<span style="color: #666666">();</span>
searcher<span style="color: #666666">.</span><span style="color: #BB4444">search</span><span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">new</span> MatchAllDocsQuery<span style="color: #666666">(),</span> fc<span style="color: #666666">);</span>
Facets facets <span style="color: #666666">=</span> <span style="color: #AA22FF; font-weight: bold">new</span> DoubleRangeFacetCounts<span style="color: #666666">(</span>
<span style="color: #BB4444">"field"</span><span style="color: #666666">,</span>
getDistanceValueSource<span style="color: #666666">(),</span> fc<span style="color: #666666">,</span>
ONE_KM<span style="color: #666666">,</span>
TWO_KM<span style="color: #666666">,</span>
FIVE_KM<span style="color: #666666">,</span>
TEN_KM<span style="color: #666666">);</span>
<span style="color: #AA22FF; font-weight: bold">return</span> facets<span style="color: #666666">.</span><span style="color: #BB4444">getTopChildren</span><span style="color: #666666">(10,</span> <span style="color: #BB4444">"field"</span><span style="color: #666666">);</span>
</pre></div>
</td></tr></table>
Normally you'd use a "real" query instead of the top-level-browse <code>MatchAllDocsQuery</code>.
Finally, once the user picks a distance for drill-down, use the <code>Range.getFilter</code> method and add that to a <code>DrillDownQuery</code> using <code>ConstantScoreQuery</code>:
<br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span style="color: #AA22FF; font-weight: bold">public</span> TopDocs drillDown<span style="color: #666666">(</span>DoubleRange range<span style="color: #666666">)</span> <span style="color: #AA22FF; font-weight: bold">throws</span> IOException <span style="color: #666666">{</span>
<span style="color: #008800; font-style: italic">// Passing no baseQuery means we drill down on all</span>
<span style="color: #008800; font-style: italic">// documents ("browse only"):</span>
DrillDownQuery q <span style="color: #666666">=</span> <span style="color: #AA22FF; font-weight: bold">new</span> DrillDownQuery<span style="color: #666666">(</span><span style="color: #AA22FF; font-weight: bold">null</span><span style="color: #666666">);</span>
q<span style="color: #666666">.</span><span style="color: #BB4444">add</span><span style="color: #666666">(</span><span style="color: #BB4444">"field"</span><span style="color: #666666">,</span> <span style="color: #AA22FF; font-weight: bold">new</span> ConstantScoreQuery<span style="color: #666666">(</span>
range<span style="color: #666666">.</span><span style="color: #BB4444">getFilter</span><span style="color: #666666">(</span>getDistanceValueSource<span style="color: #666666">())));</span>
<span style="color: #AA22FF; font-weight: bold">return</span> searcher<span style="color: #666666">.</span><span style="color: #BB4444">search</span><span style="color: #666666">(</span>q<span style="color: #666666">,</span> <span style="color: #666666">10);</span>
<span style="color: #666666">}</span>
</pre></div>
</td></tr></table>
See the <a href="https://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x/lucene/demo/src/java/org/apache/lucene/demo/facet/DistanceFacetsExample.java">full source code here</a>, from the <code>lucene/demo</code> module.
<br><br>
When I first tested this example, there was <a href="https://issues.apache.org/jira/browse/LUCENE-5345">a fun bug</a>, and then later <a href="https://issues.apache.org/jira/browse/LUCENE-5339">the facet APIs were overhauled</a>, so you'll need to wait for the Lucene 4.7 release, or just use the current <a href="https://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x">the 4.x sources</a>, to get this example working.
<br><br>
While this example is simple, and works correctly, there are some clear performance improvements that are possible, such as using a bounding box as a fast match to avoid computing Haversine for hits that are clearly outside of the range of possible drill-downs (patches welcome!). Even so, this is a nice step forward for Lucene's faceting and it's amazing that geospatial distance faceting with Lucene can be so simple.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com16tag:blogger.com,1999:blog-8623074010562846957.post-29936809529070828012013-12-12T17:12:00.001-05:002013-12-12T17:12:01.436-05:00Fast range faceting using segment trees and the Java ASM libraryIn Lucene's facet module we recently added support
for <a href="http://blog.mikemccandless.com/2013/05/dynamic-faceting-with-lucene.html">dynamic
range faceting</a>, to show how many hits match each of a dynamic set
of ranges. For example, the <code>Updated</code> drill-down in the
<a href="http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html">Lucene/Solr
issue search application</a> uses range facets. Another
example is distance facets (< 1 km, < 2 km, etc.), where the distance
is dynamically computed based on the user's current location. Price
faceting might also use range facets, if the ranges cannot be
established during indexing.
<br><br>
To implement range faceting, for each hit, we first calculate the
value (the distance, the age, the price) to be aggregated, and then
lookup which ranges match that value and increment its counts. Today
we use a simple linear search through all ranges, which
has <code>O(N)</code> cost, where <code>N</code> is the number of ranges.
<br><br>
But this is inefficient!
<br><br>
<b>Segment trees</b>
<br><br>
There are fun data structures
like <a href="http://en.wikipedia.org/wiki/Segment_tree">segment
trees</a> and
<a href="http://en.wikipedia.org/wiki/Interval_tree">interval
trees</a> with <code>O(log(N) + M)</code> cost per lookup, where
<code>M</code> is the number of ranges that match the given value. I
chose to explore segment trees, as Lucene only requires looking up by
a single <em>value</em> (interval trees can also efficiently look up
all ranges overlapping a provided <em>range</em>) and also because all
the ranges are known up front (interval trees also support dynamically
adding or removing ranges).
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhykxAIE8URt-KE0Di2-YO6S-Qt_mWytj_jhCg3U2JY1BXiWkENr0dZK85740oBe6n2xdTSJod6QK-WctNTQpJKJS94AFbJe-fBBJLWo0xFMlgbO5dYJr69d2OQ1i-TpDEBOEGaXoGbmypK/s1600/updated.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhykxAIE8URt-KE0Di2-YO6S-Qt_mWytj_jhCg3U2JY1BXiWkENr0dZK85740oBe6n2xdTSJod6QK-WctNTQpJKJS94AFbJe-fBBJLWo0xFMlgbO5dYJr69d2OQ1i-TpDEBOEGaXoGbmypK/s320/updated.png" /></a></div>
<br><br>
If the ranges will never overlap, you can use a simple binary search;
<a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/ImmutableRangeSet.html">Guava's
<code>ImmutableRangeSet</code></a> takes this approach. However,
Lucene's range faceting allows overlapping ranges so we can't do
that.
<br><br>
Segment trees are simple to visualize: you "project" all ranges down on
top of one another, creating a one-dimensional
<a href="http://en.wikipedia.org/wiki/Venn_diagram">Venn diagram</a>,
to define the <em>elementary intervals</em>. This classifies the
entire range of numbers into a minimal number of distinct ranges, each
elementary interval, such that all points in each elementary interval
always match the same set of input ranges. The lookup process is then
a binary search to determine which elementary interval a point belongs
to, recording the matched ranges as you recurse down the tree.
<br><br>
Consider these ranges; the lower number is inclusive and the upper
number is exclusive:
<pre>
0: 0 – 10
1: 0 – 20
2: 10 – 30
3: 15 – 50
4: 40 – 70
</pre>
The elementary intervals (think Venn diagram!) are:
<pre>
-∞ – 0
0 – 10
10 – 15
15 – 20
20 – 30
30 – 40
40 – 50
50 – 70
70 – ∞
</pre>
Finally, you build a binary tree on top of the elementary ranges, and
then add output range indices to both internal nodes and the leaves of
that tree, necessary to prevent adversarial cases that would require
too much (<code>O(N^2)</code>) space. During lookup, as you walk down
the tree, you gather up the output ranges (indices) you encounter; for
our example, each elementary range is assigned the follow range
indices as outputs:
<pre>
-∞ – 0 →
0 – 10 → 0
10 – 15 → 1, 2
15 – 20 → 1, 2, 3
20 – 30 → 2, 3
30 – 40 → 3
40 – 50 → 3, 4
50 – 70 → 4
70 – ∞ →
</pre>
Some ranges correspond to 1 elementary interval, while other ranges
correspond to 2 or 3 or more, in general. Some, 2 in this example,
may have no matching input ranges.
<br><br>
<b>Looking up matched ranges</b>
<br><br>
I've pushed all sources described below
to <a href="https://code.google.com/p/segment-trees">new Google code
project</a>; the code is still somewhat rough and exploratory, so
there are likely exciting bugs lurking, but it does seem to work: it
includes (passing!) tests and simple micro-benchmarks.
<br><br>
I started with a basic segment tree implementation as described on
the <a href="http://en.wikipedia.org/wiki/Segment_tree">Wikipedia
page</a>, for long values,
called <a href="https://code.google.com/p/segment-trees/source/browse/src/java/com/changingbits/SimpleLongRangeMultiSet.java"><code>SimpleLongRangeMultiSet</code></a>;
here's the recursive <code>lookup</code> method:
<br><br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"> <span style="color: #AA22FF; font-weight: bold">private</span> <span style="color: #00BB00; font-weight: bold">int</span> lookup<span style="color: #666666">(</span>Node node<span style="color: #666666">,</span> <span style="color: #00BB00; font-weight: bold">long</span> v<span style="color: #666666">,</span> <span style="color: #00BB00; font-weight: bold">int</span><span style="color: #666666">[]</span> answers<span style="color: #666666">,</span> <span style="color: #00BB00; font-weight: bold">int</span> upto<span style="color: #666666">)</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>node<span style="color: #666666">.</span><span style="color: #BB4444">outputs</span> <span style="color: #666666">!=</span> <span style="color: #AA22FF; font-weight: bold">null</span><span style="color: #666666">)</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">for</span><span style="color: #666666">(</span><span style="color: #00BB00; font-weight: bold">int</span> range <span style="color: #666666">:</span> node<span style="color: #666666">.</span><span style="color: #BB4444">outputs</span><span style="color: #666666">)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> range<span style="color: #666666">;</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>node<span style="color: #666666">.</span><span style="color: #BB4444">left</span> <span style="color: #666666">!=</span> <span style="color: #AA22FF; font-weight: bold">null</span><span style="color: #666666">)</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> node<span style="color: #666666">.</span><span style="color: #BB4444">left</span><span style="color: #666666">.</span><span style="color: #BB4444">end</span><span style="color: #666666">)</span> <span style="color: #666666">{</span>
upto <span style="color: #666666">=</span> lookup<span style="color: #666666">(</span>node<span style="color: #666666">.</span><span style="color: #BB4444">left</span><span style="color: #666666">,</span> v<span style="color: #666666">,</span> answers<span style="color: #666666">,</span> upto<span style="color: #666666">);</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">else</span> <span style="color: #666666">{</span>
upto <span style="color: #666666">=</span> lookup<span style="color: #666666">(</span>node<span style="color: #666666">.</span><span style="color: #BB4444">right</span><span style="color: #666666">,</span> v<span style="color: #666666">,</span> answers<span style="color: #666666">,</span> upto<span style="color: #666666">);</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #AA22FF; font-weight: bold">return</span> upto<span style="color: #666666">;</span>
<span style="color: #666666">}</span>
</pre></div>
</td></tr></table>
<br><br>
This worked correctly, but I realized there must be non-trivial
overhead for the recursion, checking for nulls, the for loop over the
output values, etc. Next, I tried switching to parallel arrays to
hold the binary tree
(<a href="https://code.google.com/p/segment-trees/source/browse/src/java/com/changingbits/ArrayLongRangeMultiSet.java">ArrayLongRangeMultiSet</a>),
where the left child of node N is at 2*N and the right child is at
2*N+1, but this turned out to be slower.
<br><br>
After that I tested a code specializing implementation, first by
creating dynamic Java source code from the binary tree. This eliminates the
recursion and creates a single simple method that uses a series of
<code>if</code> statements, specialized to the specific ranges, to do the binary
search and record the range indices. Here's the resulting specialized
code, compiled from the above ranges:
<br><br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"> <span style="color: #00BB00; font-weight: bold">void</span> lookup<span style="color: #666666">(</span><span style="color: #00BB00; font-weight: bold">long</span> v<span style="color: #666666">,</span> <span style="color: #00BB00; font-weight: bold">int</span><span style="color: #666666">[]</span> answers<span style="color: #666666">)</span> <span style="color: #666666">{</span>
<span style="color: #00BB00; font-weight: bold">int</span> upto <span style="color: #666666">=</span> <span style="color: #666666">0;</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">19)</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">9)</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666">>=</span> <span style="color: #666666">0)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">0;</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">1;</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">else</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">1;</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">2;</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666">>=</span> <span style="color: #666666">15)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">3;</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">else</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">39)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">3;</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">29)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">2;</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">else</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">49)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">3;</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">4;</span>
<span style="color: #666666">}</span> <span style="color: #AA22FF; font-weight: bold">else</span> <span style="color: #666666">{</span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #666666">(</span>v <span style="color: #666666"><=</span> <span style="color: #666666">69)</span> <span style="color: #666666">{</span>
answers<span style="color: #666666">[</span>upto<span style="color: #666666">++]</span> <span style="color: #666666">=</span> <span style="color: #666666">4;</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
<span style="color: #666666">}</span>
</pre></div>
</td></tr></table>
<br><br>
Finally, using the <a href="http://asm.ow2.org/">ASM</a> library, I
compiled the tree directly to
specialized <a href="http://en.wikipedia.org/wiki/Java_bytecode">Java
bytecode</a>, and this proved to be fastest (up to 2.5X faster in some
cases).
<br><br>
As a baseline, I also added the simple linear
search
method, <a href="https://code.google.com/p/segment-trees/source/browse/src/test/com/changingbits/LinearLongRangeMultiSet.java"><code>LinearLongRangeMultiSet</code></a>;
as long as you don't have too many ranges (say 10 or less), its
performance is often better than the Java segment tree.
<br><br>
The implementation also allows you to specify the allowed range of
input values (for example, maybe all values are >=0 in your usage),
which can save an if statement or two in the lookup method.
<br><br>
<b>Counting all matched ranges</b>
<br><br>
While the segment tree allows you to quickly look up all matching
ranges for a given value, after a nice tip from fellow Lucene
committee Robert Muir, we realized Lucene's range faceting does not
need to know the ranges for each value; instead, it only requires the
aggregated counts for each range in the end, after seeing many values.
<br><br>
This leads to an optimization: compute counts for each elementary
interval and then in the end, roll up those counts to get the count
for each range. This will only work for single-valued fields, since
for a multi-valued field you'd need to carefully never increment the
same range more than once per hit.
<br><br>
So based on that approach, I created a
new <a href="https://code.google.com/p/segment-trees/source/browse/src/test/com/changingbits/LongRangeCounter.java">LongRangeCounter</a>
abstract base class, and
the <a href="https://code.google.com/p/segment-trees/source/browse/src/test/com/changingbits/SimpleLongRangeCounter.java">SimpleLongRangeCounter</a>
Java implementation, and also the ASM specialized version, and the
results are indeed faster (~20 to 50%) than using the lookup method to
count; I'll use this approach with Lucene.
<br><br>
Segment trees are normally always "perfectly" balanced but one final
twist I explored was to use a <em>training set</em> of values to bias
the order of the if statements. For example, if your ranges cover a
tiny portion of the search space, as is the case for
the <a href="http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html">
<code>Updated</code> drill-down</a>, then it should be faster to use a
slightly unbalanced tree, by first checking if the value is less
than the maximum range. However, in testing, while there are some
cases where this "training" is a bit faster, often it's slower; I'm
not sure why.
<br><br>
<b>Lucene</b>
<br><br>
I haven't folded this into Lucene yet, but I plan to; all the
exploratory code lives in
the <a href="https://code.google.com/p/segment-trees">segment-trees</a>
Google code project for now.
<br><br>
Results on the micro-benchmarks can be entirely different once the
implementations are folded into a "real" search application.
While <a href="http://asm.ow2.org/">ASM</a> is a powerful way to
generate specialized code, and it gives sizable performance gains at
least in the micro-benchmarks, it is an added dependency and
complexity for ongoing development and many more developers know Java than
<a href="http://asm.ow2.org/">ASM</a>. It may also confuse hotspot,
causing deoptimizations when there are multiple implementations for an
abstract base class. Furthermore, if there are many ranges, the
resulting specialized bytecode can be become quite large (but,
still <code>O(N*log(N))</code> in size), which may cause other
problems. On balance I'm not sure the sizable performance gains
(on a micro-benchmark) warrant using ASM in Lucene's range faceting.Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com8tag:blogger.com,1999:blog-8623074010562846957.post-62188877679131877242013-11-29T12:43:00.000-05:002013-11-29T12:43:16.674-05:00Pulling H264 video from an IP camera using PythonIP cameras have come a long ways, and recently I upgraded some old
cameras
to <a href="http://www.lorextechnology.com/hd-security-cameras/High-Definition-IP-security-camera-for-NVR/prod330045.p">these
new Lorex cameras (model LNB2151/LNB2153)</a> and I'm very impressed.
<br><br>
These cameras record 1080p wide-angle video at 30 frames per second,
use <a href="http://en.wikipedia.org/wiki/Power_over_Ethernet">power
over ethernet (PoE)</a>, can see when it's dark using builtin
infrared <a href="http://en.wikipedia.org/wiki/Light-emitting_diode">LED</a>s
and are weather-proof. The video quality is impressive and they are
surprisingly inexpensive. The camera can deliver two streams at once,
so you can pull a lower resolution stream for preview, motion
detection, etc., and simultaneously pull the higher resolution stream
to simply record it for later scrutinizing.
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.lorextechnology.com/hd-security-cameras/High-Definition-IP-security-camera-for-NVR/prod330045.p" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpawmAbbNLNUyb2EYnRy8Obtk_PGFjQVOCvB4PJKMHBmOts-qEx3J2PP3dL-5GH7w2-Zp_IJpfbss1m3U2ZXZHIEwv1znnEVUQ1wxTnNyqngUGGaV_xqshv8Ii4xij-78dfsoJ2HhxDQHo/s320/lnb2153.png" /></a></div>
<br>
After buying a few of these cameras I needed a simple way to pull the
<a href="http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC">raw H264
video</a> from them, and with some digging I discovered the cameras
speak <a href="http://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol">RTSP</a>
and <a href="http://en.wikipedia.org/wiki/Real-time_Transport_Protocol">RTP</a>
which are standard protocols for streaming video and audio from IP
cameras. Many IP cameras have adopted these standards.
<br><br>
Both <a href="http://www.videolan.org/vlc/index.html">VLC</a>
and <a href="http://www.mplayerhq.hu/design7/news.html">MPlayer</a>
can play RTSP/RTP video streams; for the Lorex cameras the default URL
is:
<br><br>
<code>rtsp://admin:000000@<hostname>/PSIA/Streaming/channels/1</code>.
<br><br>
After more digging I found the nice open-source
(<a href="http://www.gnu.org/copyleft/lesser.html">LGPL
license</a>) <a href="http://www.live555.com/liveMedia/">Live555
project</a>, which is a C++ library for all sorts of media related
protocols, including RTSP, RTP and
RTCP. <a href="http://www.videolan.org/vlc/index.html">VLC</a>
and <a href="http://www.mplayerhq.hu/design7/news.html">MPlayer</a>
use this library for their RTSP support. Perfect!
<br><br>
My C++ is a bit rusty, and I really don't understand all of Live555's
numerous APIs, but I managed to cobble together a simple Python
extension module, derived from
Live555's <code>testRTSPClient.cpp</code> example, that seems to work
well.
<br><br>
I've <a href="https://code.google.com/p/pylive555">posted my
current source code</a> in a new Google code project
named <a href="https://code.google.com/p/pylive555">pylive555</a>. It
provides a very simple API (only 3 functions!) to pull frames from a
remote camera via
RTSP/RTP; <a href="http://www.live555.com/liveMedia/">Live555</a> has
many, many other APIs that I haven't exposed.
<br><br>
The code is thread-friendly (releases
the <a href="https://wiki.python.org/moin/GlobalInterpreterLock">global
interpreter lock</a> when invoking the Live555 APIs).
<br><br>
I've included a
simple <a href="https://code.google.com/p/pylive555/source/browse/example.py"><code>example.py</code></a>
Python program, that shows how to load H264 video frames from the
camera and save them to a local file. You could start from this
example and modify it to do other things, for example use the
<a href="http://www.ffmpeg.org/">ffmpeg H264 codec</a> to decode
individual frames, use a motion detection library to trigger recording,
parse each frame's metadata to find the keyframes, etc. Here's the current <code>example.py</code>:
<br><br>
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">time</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sys</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">live555</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">threading</span>
<span style="color: #008800; font-style: italic"># Shows how to use live555 module to pull frames from an RTSP/RTP</span>
<span style="color: #008800; font-style: italic"># source. Run this (likely first customizing the URL below:</span>
<span style="color: #008800; font-style: italic"># Example: python3 example.py 10.17.4.118 1 10 out.264 </span>
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #AA22FF">len</span>(sys<span style="color: #666666">.</span>argv) <span style="color: #666666">!=</span> <span style="color: #666666">5</span>:
<span style="color: #AA22FF; font-weight: bold">print</span>()
<span style="color: #AA22FF; font-weight: bold">print</span>(<span style="color: #BB4444">'Usage: python3 example.py cameraIP channel seconds fileOut'</span>)
<span style="color: #AA22FF; font-weight: bold">print</span>()
sys<span style="color: #666666">.</span>exit(<span style="color: #666666">1</span>)
cameraIP <span style="color: #666666">=</span> sys<span style="color: #666666">.</span>argv[<span style="color: #666666">1</span>]
channel <span style="color: #666666">=</span> sys<span style="color: #666666">.</span>argv[<span style="color: #666666">2</span>]
seconds <span style="color: #666666">=</span> <span style="color: #AA22FF">float</span>(sys<span style="color: #666666">.</span>argv[<span style="color: #666666">3</span>])
fileOut <span style="color: #666666">=</span> sys<span style="color: #666666">.</span>argv[<span style="color: #666666">4</span>]
<span style="color: #008800; font-style: italic"># NOTE: the username & password, and the URL path, will vary from one</span>
<span style="color: #008800; font-style: italic"># camera to another! This URL path works with the Lorex LNB2153:</span>
url <span style="color: #666666">=</span> <span style="color: #BB4444">'rtsp://admin:000000@</span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BB4444">/PSIA/Streaming/channels/</span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BB4444">'</span> <span style="color: #666666">%</span> (cameraIP, channel)
fOut <span style="color: #666666">=</span> <span style="color: #AA22FF">open</span>(fileOut, <span style="color: #BB4444">'wb'</span>)
<span style="color: #AA22FF; font-weight: bold">def</span> <span style="color: #00A000">oneFrame</span>(codecName, <span style="color: #AA22FF">bytes</span>, sec, usec, durUSec):
<span style="color: #AA22FF; font-weight: bold">print</span>(<span style="color: #BB4444">'frame for </span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BB4444">: </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BB4444"> bytes'</span> <span style="color: #666666">%</span> (codecName, <span style="color: #AA22FF">len</span>(<span style="color: #AA22FF">bytes</span>)))
fOut<span style="color: #666666">.</span>write(b<span style="color: #BB4444">'</span><span style="color: #BB6622; font-weight: bold">\0\0\0\1</span><span style="color: #BB4444">'</span> <span style="color: #666666">+</span> <span style="color: #AA22FF">bytes</span>)
<span style="color: #008800; font-style: italic"># Starts pulling frames from the URL, with the provided callback:</span>
useTCP <span style="color: #666666">=</span> <span style="color: #AA22FF">False</span>
live555<span style="color: #666666">.</span>startRTSP(url, oneFrame, useTCP)
<span style="color: #008800; font-style: italic"># Run Live555's event loop in a background thread:</span>
t <span style="color: #666666">=</span> threading<span style="color: #666666">.</span>Thread(target<span style="color: #666666">=</span>live555<span style="color: #666666">.</span>runEventLoop, args<span style="color: #666666">=</span>())
t<span style="color: #666666">.</span>setDaemon(<span style="color: #AA22FF">True</span>)
t<span style="color: #666666">.</span>start()
endTime <span style="color: #666666">=</span> time<span style="color: #666666">.</span>time() <span style="color: #666666">+</span> seconds
<span style="color: #AA22FF; font-weight: bold">while</span> time<span style="color: #666666">.</span>time() <span style="color: #666666"><</span> endTime:
time<span style="color: #666666">.</span>sleep(<span style="color: #666666">0.1</span>)
<span style="color: #008800; font-style: italic"># Tell Live555's event loop to stop:</span>
live555<span style="color: #666666">.</span>stopEventLoop()
<span style="color: #008800; font-style: italic"># Wait for the background thread to finish:</span>
t<span style="color: #666666">.</span>join()
</pre></div>
</td></tr></table>
<br><br>
Installation is very easy; see
the <a href="https://code.google.com/p/pylive555/source/browse/README.txt"><code>README.txt</code></a>.
I've only tested on Linux with Python3.2 and with the Lorex LNB2151
cameras.
<br><br>
I'm planning on installing one of these Lorex cameras inside
a <a href="http://www.batmanagement.com/Batcentral/batcentral.html">bat
house</a> that I'll build with the kids this winter. If we're lucky
we'll be able to view live bats in the summer!Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com74tag:blogger.com,1999:blog-8623074010562846957.post-28351935406346699982013-11-12T16:51:00.000-05:002013-11-12T16:51:16.264-05:00Playing a sound (AIFF) file from Python using PySDL2Sometimes you need to play sounds or music (digitized samples) from
Python, which really ought to be a simple task. Yet it took me a
little while to work out, and the resulting source code is quite
simple, so I figured I'd share it here in case anybody else is
struggling with it.
<br><br>
The Python wiki
lists <a href="https://wiki.python.org/moin/Audio/">quite a few
packages</a> for working with audio, but most of them are overkill for
basic audio recording and playback.
<br><br>
For quite some time I had been
using <a href="http://people.csail.mit.edu/hubert/pyaudio/">PyAudio</a>,
which adds Python bindings to
the <a href="http://www.portaudio.com">PortAudio</a> project. I
really like it because it focuses entirely on recording and playing
audio. But, for some reason, when I recently upgraded to Mavericks,
it stutters whenever I try to play samples at a sample rate lower than
44.1 KHz. I've emailed the author to try to get to the bottom of it.
<br><br>
In the meantime, I tried a new
package, <a href="https://pypi.python.org/pypi/PySDL2/0.2.0">PySDL2</a>,
which adds Python bindings to
the <a href="http://www.libsdl.org/download-2.0.php">SDL2 (Simple
Directmedia Layer)</a> project.
<br><br>
SDL2 does quite a bit more than basic
audio, and I didn't dig into any of that yet. I
hit <a href="https://bitbucket.org/marcusva/py-sdl2/issue/26/sdl_audiospeccallback-should-be-type">one
small issue</a> with PySDL2, but the one-line change in the issue
fixes it. Here's the resulting code:
<table class="highlighttable"><tr><td><div class="linenodiv" style="background-color: #f0f0f0; padding-right: 10px"><pre style="line-height: 125%"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55</pre></div></td><td class="code"><div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sdl2</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sys</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">aifc</span>
<span style="color: #AA22FF; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">threading</span>
<span style="color: #AA22FF; font-weight: bold">class</span> <span style="color: #0000FF">ReadAIFF</span>:
<span style="color: #AA22FF; font-weight: bold">def</span> <span style="color: #00A000">__init__</span>(<span style="color: #AA22FF">self</span>, fileName):
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>a <span style="color: #666666">=</span> aifc<span style="color: #666666">.</span>open(fileName)
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>frameUpto <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>bytesPerFrame <span style="color: #666666">=</span> <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>a<span style="color: #666666">.</span>getnchannels() <span style="color: #666666">*</span> <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>a<span style="color: #666666">.</span>getsampwidth()
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>numFrames <span style="color: #666666">=</span> <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>a<span style="color: #666666">.</span>getnframes()
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>done <span style="color: #666666">=</span> threading<span style="color: #666666">.</span>Event()
<span style="color: #AA22FF; font-weight: bold">def</span> <span style="color: #00A000">playNextChunk</span>(<span style="color: #AA22FF">self</span>, unused, buf, bufSize):
framesInBuffer <span style="color: #666666">=</span> bufSize<span style="color: #666666">/</span><span style="color: #AA22FF">self</span><span style="color: #666666">.</span>bytesPerFrame
framesToRead <span style="color: #666666">=</span> <span style="color: #AA22FF">min</span>(framesInBuffer, <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>numFrames<span style="color: #666666">-</span><span style="color: #AA22FF">self</span><span style="color: #666666">.</span>frameUpto)
<span style="color: #AA22FF; font-weight: bold">if</span> <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>frameUpto <span style="color: #666666">==</span> <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>numFrames:
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>done<span style="color: #666666">.</span>set()
<span style="color: #008800; font-style: italic"># TODO: is there a faster way to copy the string into the ctypes</span>
<span style="color: #008800; font-style: italic"># pointer/array?</span>
<span style="color: #AA22FF; font-weight: bold">for</span> i, b <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #AA22FF">enumerate</span>(<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>a<span style="color: #666666">.</span>readframes(framesToRead)):
buf[i] <span style="color: #666666">=</span> <span style="color: #AA22FF">ord</span>(b)
<span style="color: #008800; font-style: italic"># Play silence after:</span>
<span style="color: #008800; font-style: italic"># TODO: is there a faster way to zero out the array?</span>
<span style="color: #AA22FF; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #AA22FF">range</span>(<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>bytesPerFrame<span style="color: #666666">*</span>framesToRead, <span style="color: #AA22FF">self</span><span style="color: #666666">.</span>bytesPerFrame<span style="color: #666666">*</span>framesInBuffer):
buf[i] <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #AA22FF">self</span><span style="color: #666666">.</span>frameUpto <span style="color: #666666">+=</span> framesToRead
<span style="color: #AA22FF; font-weight: bold">if</span> sdl2<span style="color: #666666">.</span>SDL_Init(sdl2<span style="color: #666666">.</span>SDL_INIT_AUDIO) <span style="color: #666666">!=</span> <span style="color: #666666">0</span>:
<span style="color: #AA22FF; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">RuntimeError</span>(<span style="color: #BB4444">'failed to init audio'</span>)
p <span style="color: #666666">=</span> ReadAIFF(sys<span style="color: #666666">.</span>argv[<span style="color: #666666">1</span>])
spec <span style="color: #666666">=</span> sdl2<span style="color: #666666">.</span>SDL_AudioSpec(p<span style="color: #666666">.</span>a<span style="color: #666666">.</span>getframerate(),
sdl2<span style="color: #666666">.</span>AUDIO_S16MSB,
p<span style="color: #666666">.</span>a<span style="color: #666666">.</span>getnchannels(),
<span style="color: #666666">512</span>,
sdl2<span style="color: #666666">.</span>SDL_AudioCallback(p<span style="color: #666666">.</span>playNextChunk))
<span style="color: #008800; font-style: italic"># TODO: instead of passing None for the 4th arg, I really should pass</span>
<span style="color: #008800; font-style: italic"># another AudioSpec and then confirm it matched what I asked for:</span>
devID <span style="color: #666666">=</span> sdl2<span style="color: #666666">.</span>SDL_OpenAudioDevice(<span style="color: #AA22FF">None</span>, <span style="color: #666666">0</span>, spec, <span style="color: #AA22FF">None</span>, <span style="color: #666666">0</span>)
<span style="color: #AA22FF; font-weight: bold">if</span> devID <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
<span style="color: #AA22FF; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">RuntimeError</span>(<span style="color: #BB4444">'failed to open audio device'</span>)
<span style="color: #008800; font-style: italic"># Tell audio device to start playing:</span>
sdl2<span style="color: #666666">.</span>SDL_PauseAudioDevice(devID, <span style="color: #666666">0</span>)
<span style="color: #008800; font-style: italic"># Wait until all samples are done playing</span>
p<span style="color: #666666">.</span>done<span style="color: #666666">.</span>wait()
sdl2<span style="color: #666666">.</span>SDL_CloseAudioDevice(devID)
</pre></div>
</td></tr></table>
<br><br>
The code is straightforward: it loads an AIFF file, using Python's
builtin <code>aifc</code> module, and then creates a
callback, <code>playNextChunk</code> which is invoked
by <code>PySDL2</code> when it needs more samples to play. So far it
seems to work very well!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com2tag:blogger.com,1999:blog-8623074010562846957.post-15596554533025605082013-09-28T17:32:00.000-04:002013-09-28T17:32:55.984-04:00Lucene now has an in-memory terms dictionary, thanks to Google Summer of CodeLast
year, <a href="http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/billybob/1">Han
Jiang's Google Summer of Code project</a>
was <a href="http://blog.mikemccandless.com/2012/08/lucenes-new-blockpostingsformat-thanks.html">a
big success</a>: he created a new (now, default) postings format
for substantially faster searches, along with smaller indices.
<br><br>
This summer, Han was at it again, with
a <a href="http://www.google-melange.com/gsoc/project/google/gsoc2013/billybob/42001">
new Google Summer of Code project</a> with Lucene: he created a new
terms dictionary holding all terms and their metadata in memory as an
FST.
<br><br>
In fact, he created two new terms dictionary implementations. The
first, <code>FSTTermsWriter/Reader</code>, hold all terms and metadata
in a single in-memory FST, while the second,
<code>FSTOrdTermsWriter/Reader</code>, does the same but also supports
retrieving the ordinal for a term (<code>TermsEnum.ord()</code>) and
looking up a term given its ordinal (<code>TermsEnum.seekExact(long
ord)</code>). The second one also uses this <code>ord</code>
internally so that the FST is more compact, while all metadata is
stored outside of the FST, referenced by <code>ord</code>.
<br><br>
Like the default <code>BlockTree</code> terms dictionary, these new
terms dictionaries accept any <code>PostingsBaseFormat</code> so you
can separately plug in whichever format you want to encode/decode the
postings.
<br><br>
Han also improved the <code>PostingsBaseFormat</code> API so that
there is now a cleaner separation of how terms and their metadata are
encoded vs. how postings are encoded;
<code>PostingsWriterBase.encodeTerm</code>
and <code>PostingsReaderBase.decodeTerm</code> now handle encoding and
decoding any term metadata required by the postings format,
abstracting away how the long[]/byte[] were persisted by the terms
dictionary. Previously this line was annoyingly blurry.
<br><br>
Unfortunately, while the performance for primary key lookups is
substantially faster, other queries
e.g. <code>WildcardQuery</code> are slower; see
<a href="https://issues.apache.org/jira/browse/LUCENE-3069">LUCENE-3069</a>
for details. Fortunately, using <code>PerFieldPostingsFormat</code>,
you are free to pick and choose which fields (e.g. your "id" field)
should use the new terms dictionary.
<br><br>
For now this feature is trunk-only (eventually Lucene 5.0).
<br><br>
Thank you Han and thank you Google!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com11tag:blogger.com,1999:blog-8623074010562846957.post-86085672437258607762013-09-16T07:25:00.000-04:002013-09-16T07:25:29.609-04:00Three exciting Lucene features in one day<h1>Three exciting Lucene features in one day</h1>
Yesterday was a productive day: suddenly, there are three exciting new
features coming to Lucene.
<br><br>
<b>Expressions module</b>
<br><br>
The first feature, committed yesterday, is the
new <a href="https://issues.apache.org/jira/browse/LUCENE-5207">expressions
module</a>. This allows you to define a dynamic field for sorting,
using an arbitrary <code>String</code> expression. There is builtin support for parsing
<a href="http://en.wikipedia.org/wiki/JavaScript">JavaScript</a>, but
the parser is pluggable if you want to create your own syntax.
<br><br>
For example, you could define a sort field using the
expression
<pre>
sqrt(_score) + ln(popularity)
</pre>
if you want to offer a blended sort primarily by relevance and
boosting by a popularity field.
<br><br>
The code is very easy to use; there are some nice examples in
the <a href="https://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x/lucene/expressions/src/test/org/apache/lucene/expressions/TestDemoExpressions.java"><code>TestDemoExpressions.java</code></a>
unit test case, and this will be available in Lucene's next stable
release (4.6).
<br><br>
<b>Updateable numeric doc-values fields</b>
<br><br>
The second feature, also committed yesterday,
is <a href="https://issues.apache.org/jira/browse/LUCENE-5189">updateable
numeric doc-values fields</a>, letting you change previously indexed
numeric values using the new <code>updateNumericDocValue</code> method
on <code>IndexWriter</code>. It works fine with near-real-time
readers, so you can update the numeric values for a few documents and
then re-open a new near-real-time reader to see the changes.
<br><br>
The feature is currently trunk only as we work out a few remaining
issues involving an particularly controversial boolean. It also
currently does not work on sparse fields, i.e. you can only update a
document's value if that document had already indexed that field in
the first place.
<br><br>
Combined, these two features enable powerful use-cases where you want
to sort by a blended field that is changing over time. For example,
perhaps you measure how often your users click through each document
in the search results, and then use that to update
the <code>popularity</code> field, which is then used for a blended
sort. This way the rankings of the search results change over time as
you learn from users which documents are popular and which are not.
<br><br>
Of course such a feature was always possible before, using custom
external code, but with both expressions and updateable doc-values now
available it becomes trivial to implement!
<br><br>
<b>Free text suggestions</b>
<br><br>
Finally, the third feature is a new suggester
implementation, <a href="https://issues.apache.org/jira/browse/LUCENE-5214"><code>FreeTextSuggester</code></a>.
It is a very different suggester than the existing ones: rather than
suggest from a finite universe of pre-built suggestions, it uses a
simple <a href="http://en.wikipedia.org/wiki/Language_model#N-gram_models">ngram
language model</a> to predict the "long tail" of possible suggestions
based on the 1 or 2 previous tokens.
<br><br>
Under the hood, it uses
<code>ShingleFilter</code> to create the ngrams, and an FST to store
and lookup the resulting ngram models. While multiple ngram models
are stored compactly in a single FST, the FST can still get quite
large; the 3-gram, 2-gram and 1-gram model built on
the <a href="http://en.wikipedia.org/wiki/AOL_search_data_leak">AOL
query logs</a> is 19.4 MB (the queries themselves are 25.4 MB). This
was inspired
by <a href="http://googleblog.blogspot.com/2011/04/more-predictions-in-autocomplete.html">Google's
approach</a>.
<br><br>
Likely this suggester would not be used by itself, but rather as a
fallback when your primary suggester failed to find any suggestions;
you can see this behavior with <a href="http://google.com">Google</a>.
Try searching for "the fast and the ", and you will see the suggestions
are still full queries. But if the next word you type is "burning"
then suddenly google (so far!) does not have a full suggestion and
falls back to their free text approach.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com4tag:blogger.com,1999:blog-8623074010562846957.post-29540895451881116622013-08-14T11:52:00.002-04:002013-08-14T11:52:47.642-04:00SuggestStopFilter carefully removes stop words for suggestersLucene now has a nice set of suggesters that use
an analyzer to tokenize the suggestions: <a href="http://blog.mikemccandless.com/2012/09/lucenes-new-analyzing-suggester.html"><code>AnalyzingSuggester</code></a>,
<code>FuzzySuggester</code>
and <a href="http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html"><code>AnalyzingInfixSuggester</code></a>.
Using an analyzer is powerful because it lets you customize exactly
how suggestions are matched: you can normalize case, apply stemming, match across
different synonym forms, etc.
<br><br>
One of the most common things you'll do with your analyzer is to
remove stop-words using <code>StopFilter</code>. Unfortunately, if
you try this, you'll quickly notice that the stop filter is too
aggressive because it happily removes the last token even if the user
isn't done typing it yet. For example if the user has typed "a",
you'd expect suggestions like apple, aardvark, etc., but you won't get
that because <code>StopFilter</code> removed the "a" token.
<br><br>
You could try using <code>StopFilter</code> only while indexing, which
was my first attempt with the suggestions
at <a href="http://jirasearch.mikemccandless.com">jirasearch.mikemccandless.com</a>,
but then, at least
for <a href="http://blog.mikemccandless.com/2013/06/a-new-lucene-suggester-based-on-infix.html"><code>AnalyzingInfixSuggester</code></a>,
you'll fail to get matches when you
pass <code>allTermsRequired=true</code> because the suggester then requires
that even stop words find matches.
<br><br>
Finally, you could use the new <a href="https://issues.apache.org/jira/browse/LUCENE-5165"><code>StopSuggestFilter</code></a>
at lookup time: this filter is just like <code>StopFilter</code> except
when the token is the very last token, it checks the offset for that
token and if the offset indicates that the token has ended without any
further non-token characters, then the token is preserved. The token
is also marked as a keyword, so that any later stem filters won't change
it. This way a query "a" can find "apple", but a query "a " (with a
trailing space) will find nothing because the "a" will be removed.
<br><br>
I've pushed <code>StopSuggestFilter</code> to
<a href="http://jirasearch.mikemccandless.com">jirasearch.mikemccandless.com</a>
and it seems to be working well so far!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com10tag:blogger.com,1999:blog-8623074010562846957.post-13363614015189915782013-08-02T13:27:00.000-04:002013-08-02T13:27:38.310-04:00A new version of the Compact Language DetectorIt's been almost two years since I
originally <a href="http://blog.mikemccandless.com/2011/10/language-detection-with-googles-compact.html">factored
out</a>
the <a href="http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html">fast
and
accurate</a> <a href="https://code.google.com/p/chromium-compact-language-detector/">Compact
Language Detector</a> from
the <a href="http://www.chromium.org/">Chromium project</a>, and the
effort was clearly worthwhile: the project is popular and others have
created additional bindings for languages including at least Perl,
Ruby, R, JavaScript, PHP and C#/.NET.
<br><br>
Eric Fischer used CLD to create the
colorful <a href="http://www.flickr.com/photos/walkingsf/6276642489/">Twitter
language map</a>, and since then further language maps have appeared,
e.g. for <a href="http://www.businessinsider.com/multilingual-twitter-map-new-york-city-2013-2">New
York</a>
and <a href="http://spatial.ly/2012/10/londons-twitter-languages/">London</a>.
What a multi-lingual world we live in!
<br><br>
Suddenly, just a few weeks ago, I received an out-of-the-blue email
from Dick Sites, creator of CLD, with great news: he was finishing up
version 2.0 of CLD and had already posted the source code on
a <a href="https://code.google.com/p/cld2/">new project</a>.
<br><br>
So I've now reworked the Python bindings and ported the unit tests to
Python (they pass!) to take advantage of the new features. It was
much easier this time around since the CLD2 sources were already
pulled out into their own project (thank you Dick and Google!).
<br><br>
There are a number of improvements over the previous version of CLD:
<ul>
<li> Improved accuracy.
<br><br>
<li> Upgraded to Unicode 6.2 characters.
<br><br>
<li> More languages detected: 83 languages, up from 64 previously.
<br><br>
<li> A new "full language table" detector, available in Python as a
separate cld2full module, that detects 161 languages. This
increases the C library size from 1.8 MB (for 83 languages) to
5.5 MB (for 161
languages). <a href="https://code.google.com/p/cld2/wiki/CLD2FullVersion">Details
are here</a>.
<br><br>
<li> An option to identify which parts (byte ranges) of the text
contain which language, in case the application needs to do
further language-specific processing. From Python, pass the
optional <code>returnVectors=True</code> argument to get the byte
ranges, but note that this requires additional non-trivial CPU
cost. <a href="https://code.google.com/p/cld2/wiki/LanguageWebShare">This
wiki page</a> shows very interesting statistics on how frequently
different languages appear in one page, across top web sites,
showing the importance of handling multiple languages in a single
text input.
<br><br>
<li> A new <code>hintLanguageHTTPHeaders</code> parameter, which you
can pass from the <code>Content-Language</code> HTTP header.
Also, CLD2 will spot any lang=X attribute inside
the <code><html></code> tag itself (if you pass it HTML).
</ul>
In the new Python bindings, I've exposed CLD2's debug* flags, to add
verbosity to CLD2's detection
process. <a href="https://code.google.com/p/cld2/wiki/CLD2UnitTestOutput">This
document</a> describes how to interpret the resulting output.
<br><br>
The <code>detect</code> function returns up to 3 top detected
languages. Each detected language includes the percent of the text
that was detected as the language, and a confidence score. The
function no longer returns a single "picked" summary language, and the
<code>pickSummaryLanguage</code> option has been removed: this option
was apparently present for internal backwards compatibility reasons
and did not improve accuracy.
<br><br>
Remember that the provided input must be valid UTF-8 bytes, otherwise
all sorts of things could go wrong (wrong results, segmentation
fault).
<br><br>
To see the list of detected languages, just run this <code>python -c
"import cld2; print cld2.DETECTED_LANGUAGES"</code>, or <code>python -c
"import cld2full; print cld2full.DETECTED_LANGUAGES"</code> to see the
full set of languages.
<br><br>
The <a href="https://code.google.com/p/chromium-compact-language-detector/source/browse/README">README</a>
gives details on how to build and install CLD2.
<br><br>
Once again, thank you Google, and thank you Dick Sites for making this
very useful library available to the world as open-source.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com19tag:blogger.com,1999:blog-8623074010562846957.post-10335154762365981912013-06-22T17:15:00.000-04:002013-06-22T17:15:59.012-04:002X faster PhraseQuery with Lucene using C++ via JNII <a href="http://blog.mikemccandless.com/2013/06/screaming-fast-lucene-searches-using-c.html">recently
described</a> the
new <a href="https://github.com/mikemccand/lucene-c-boost">lucene-c-boost
github project</a>, which provides amazing speedups (up to 7.8X
faster) for common Lucene query types using specialized C++ implementations via
JNI.
<br><br>
The code works with a stock Lucene 4.3.0 JAR and default codec, and
has a trivial API: just call <code>NativeSearch.search</code> instead
of <code>IndexSearcher.search</code>.
<br><br>
Now, a quick update: I've optimized <code>PhraseQuery</code> now as
well:
<br><br>
<table><tr><th>Task</th><th>QPS base </th><th>StdDev base </th><th>QPS opt </th><th>StdDev opt </th><th>% change</th></tr>
<tr><td>HighPhrase</td><td>3.5</td><td>(2.7%)</td><td>6.5</td><td>(0.4%)</td><td><font color="green">1.9 X</font></td>
<tr><td>MedPhrase</td><td>27.1</td><td>(1.4%)</td><td>51.9</td><td>(0.3%)</td><td><font color="green">1.9 X</font></td>
<tr><td>LowPhrase</td><td>7.6</td><td>(1.7%)</td><td>16.4</td><td>(0.3%)</td><td><font color="green">2.2 X</font></td>
</table>
<br><br>
~2X speedup (~90% - ~119%) is nice!
<br><br>
Again, it's great to see a
reduced variance on the runtimes since hotspot is mostly not an
issue. It's odd that <code>LowPhrase</code> gets slower QPS
than <code>MedPhrase</code>: these queries look mis-labelled (I
see the <code>LowPhrase</code> queries getting more hits than <code>MedPhrase</code>!).
<br><br>
All changes have been pushed
to <a href="https://github.com/mikemccand/lucene-c-boost">lucene-c-boost</a>;
next I'd like to figure out how to get facets working.
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com4tag:blogger.com,1999:blog-8623074010562846957.post-82396006692303862452013-06-22T16:37:00.000-04:002013-06-22T16:37:32.578-04:00A new Lucene suggester based on infix matchesSuggest, sometimes called auto-suggest, type-ahead search or
auto-complete, is now an essential search feature ever since Google
added it
almost <a href="http://googleblog.blogspot.com/2008/08/at-loss-for-words.html">5
years ago</a>.
<br><br>
Lucene has a number of implementations;
I <a href="http://blog.mikemccandless.com/2012/09/lucenes-new-analyzing-suggester.html">previously
described <code>AnalyzingSuggester</code></a>. Since
then, <code>FuzzySuggester</code> was also added, which extends
<code>AnalyzingSuggester</code> by also accepting mis-spelled inputs.
<br><br>
Here I describe our newest
suggester: <code>AnalyzingInfixSuggester</code>, now going through
iterations on
the <a href="https://issues.apache.org/jira/browse/LUCENE-4845">LUCENE-4845</a>
Jira issue.
<br><br>
Unlike the existing suggesters, which generally find suggestions whose
whole prefix matches the current user input, this suggester will find
matches of tokens anywhere in the user input and in the suggestion;
this is why it has <em>Infix</em> in its name.
<br><br>
You can see it in action
at <a href="http://jirasearch.mikemccandless.com">the example Jira
search application</a>
that <a href="http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html">I
built to showcase various Lucene features</a>.
<br><br>
For example, if you
enter <code>japan</code> you should see various issues suggested,
including:
<ul>
<li> SOLR-4945: <font color=red>Japan</font>ese Autocomplete and Highlighter broken
<li> LUCENE-3922: Add <font color=red>Japan</font>ese Kanji number normalization to Kuromoji
<li> LUCENE-3921: Add decompose compound <font color=red>Japan</font>ese Katakana
token capability to Kuromoji
</ul>
As you can see, the incoming characters can match not just the prefix
of each suggestion but also the prefix of any token within.
<br><br>
Unlike the existing suggesters, this new suggester does not use a
specialized data-structure such
as <a href="http://blog.mikemccandless.com/2010/12/using-finite-state-transducers-in.html">FSTs</a>.
Instead, it's an "ordinary" Lucene index under-the-hood, making use
of <code>EdgeNGramTokenFilter</code> to index the short prefixes of
each token, up to length 3 by default, for fast prefix querying.
<br><br>
It also uses the
new <a href="https://issues.apache.org/jira/browse/LUCENE-3918">index
sorter</a> APIs to pre-sort all postings by suggested weight at index
time, and at lookup time uses a
custom <code>Collector</code> to stop after finding the first N
matching hits since these hits are the best matches when sorting by
weight. The lookup method lets you specify whether all terms must be
found, or any of the terms
(<a href="http://jirasearch.mikemccandless.com">Jira search</a>
requires all terms).
<br><br>
Since the suggestions are sorted solely by weight, and no other
relevance criteria, this suggester is a good fit for applications that
have a strong a-priori weighting for each suggestion, such as a movie
search engine ranking suggestions by popularity, recency or a blend, for
each movie.
In <a href="http://jirasearch.mikemccandless.com">Jira search</a> I
rank each suggestion (Jira issue) by how recently it was updated.
<br><br>
Specifically, there is no penalty for suggestions with matching tokens
far from the beginning, which could mean the relevance is poor in some
cases; an alternative approach (patch is on the issue) uses FSTs
instead, which can require that the matched tokens are within the
first three tokens, for example. This would also be possible
with <code>AnalyzingInfixSuggester</code> using an index-time analyzer
that dropped all but the first three tokens.
<br><br>
One nice benefit of an index-based approach
is <code>AnalyzingInfixSuggester</code> handles highlighting of the
matched tokens (red color, above),
which <a href="https://issues.apache.org/jira/browse/LUCENE-4518">has
unfortunately proven difficult to provide</a> with the FST-based
suggesters. Another benefit is, in theory, the suggester could
support near-real-time indexing, but I haven't exposed that in the
current patch and probably won't for some time (patches welcome!).
<br><br>
Performance is reasonable: somewhere
between <code>AnalyzingSuggester</code>
and <code>FuzzySuggester</code>, between 58 - 100 kQPS (details
on <a href="https://issues.apache.org/jira/browse/LUCENE-4845">the issue</a>).
<br><br>
<b>Analysis fun</b>
<br><br>
As
with <code>AnalyzingSuggester</code>, <code>AnalyzingInfixSuggester</code>
let's you separately configure the index-time vs. search-time
analyzers.
With <a href="http://jirasearch.mikemccandless.com">Jira search</a>, I
enabled stop-word removal at index time, but not at
search time, so that a query like <code>or</code> would still
successfully find any suggestions containing words starting
with <code>or</code>, rather than dropping the term entirely.
<br><br>
Which suggester should you use for your application? Impossible to
say! You'll have to test each of Lucene's offerings and pick one.
Auto-suggest is an area where one-size-does-not-fit-all, so it's great
that Lucene is picking up a number of competing implementations.
Whichever you use,
please <a href="mailto:java-user@lucene.apache.org">give us
feedback</a> so we can further iterate and improve!
Michael McCandlesshttp://www.blogger.com/profile/04277432937861334672noreply@blogger.com51