It turns out, the Jira issue tracking system, which we make heavy use of here at Apache, uses Lucene under the hood for searching and browsing issues. This is wonderful since it means Lucene developers are eating their own dog food whenever they use Jira.
Atlassian has opened up some doozy bugs over time, including one of the earliest bug numbers I've ever worked on, LUCENE-140. They sent me a t-shirt for fixing that one (thank you!).
Now, imagine this: what if there were a sneaky bug in Lucene, say a certain text fragment that causes an exception during indexing. A user opens an issue to report this, including the problematic text fragment, yet, because Jira uses Lucene, it hits an exception while indexing that fragment and causes this one bug to be un-searchable and un-viewable when browsing! An invisible bug fixed point.
It's somewhat mind bending to think about, Lucene recursing on itself through Jira, yet it's theoretically possible! Maybe we have a few of invisible bug fixed points lurking already and nobody knows...
While we haven't seen one with Lucene in particular, the overall pattern has definitely been seen a few times. There was one very recently where we specifically had a reported case on our production server and somehow the bug regressed, bringing down our production server! Someone who didn't write a unit-test got found out that time…
ReplyDeleteAlso, within JIRA, not all views of an issue are sourced from the index (some are sourced from the DB). So if this theoretical situation did occur, you'd still be able to see the issue in some views.
OK thanks Jed.
ReplyDeleteReminiscent of Ken Thompson's C compiler backdoor described in 'Reflections on Trusting Trust' http://cm.bell-labs.com/who/ken/trust.html (but not on purpose in this case...).
ReplyDelete