Harmony is stuck in my head!

I often get songs stuck in my head, but I've noticed a curious thing: I'm especially proned to getting songs with a harmony stuck in my head. It's a puzzle and my brain won't stoup until it's teased apart the primary melody and harmony. Literally it just keeps playing over and over, even as I type this.

This is the song stuck in my head right now. It's a great rendition of Michael Jackson's beat it. I especially love the Xylophone -- it's partially syncopated (plays, sometimes, on the off beat). The harmony starts at 1:14. Warning: it's addictive -- your brain, too, may not stop until it's figured out the harmony!

Direct Democracy

I think this is a great idea, to allow the US population to vote, directly, on whether a bill should become law, instead of the indirect process we now use, trusting our congresspeople to vote on our behalf.

Everyone in the US knows how lobbyists, hired by corporations or other groups with lots of money, sway how our congresspeople vote by simply bribing them. It's a disgusting situation, yet, somehow we all complacently accept it as normal. I'm sure our founding fathers had no idea this would happen nor how much technology would advance.

Congress does other things, of course, like holding public hearings on important topics, etc., so I don't think it'll really be as simple as outright abolishing it.

Unfortunately it seems very unlikely that the US will ever switch to a direct democracy. But I for one would sure love to see it.

Discrimination against kids

A few weeks back we went camping with the kids; it was great fun! I did tons of camping as a kid and I've been eagerly looking forward to our kids being old enough to go. They finally are.

My wife found a rather luxurious campground: it had indoor pools, jacuzzi, outdoor pools, places to buy food, drinks, firewood, etc. It was generally very kid friendly -- lots of activities for kids, play room, etc. We were quite spoiled and we loved it.

However, there was a strict rule that kids are not allowed in the jacuzzi nor the sauna. When we pulled out inflatable pool toys, we were politely informed that they, too, are not allowed.

Now, don't get me wrong: these rules are common these days. Many places with pools and jacuzzis and saunas will have the same rules. These rules didn't exist when I was a kid, but now they are commonplace and accepted as normal. My kids just accepted them, in stride, as kids will do.

But I am bothered: this is quite simply discrimination against kids.

I understand why the rules are there. Thanks to our overly litigious society, if kids get hurt in the pool or overheat in the jacuzzi, the parents sue the campground rather than accept their own negligence. And unfortunately, they often win, or, settle out of court. This is not unlike McDonalds being sued for making coffee that's too hot.

At the same time, we also have become a "safety above all" society, for better or worse. We must wear helmets when we bike, elbow pads and knee pads when we get on our roller blades, seat belts and car seats when we go driving, life preservers if we get anywhere near water. Kids barely venture out into their front yard, let alone out around the neighborhood, for the [overblown] fear that something bad could happen. Examples abound.

As a parent I fully appreciate the dangers of water. In fact water is a terrifying combination: it is at once incredibly fun yet also deadly. To a child who can't yet swim, a pool is literally a potential death trap, and it's the parent's job to keep their kids safe.

Thus, the insurance costs go up, likely by quite a bit, if the campground allows such "unsafe" activities, and so they choose to discriminate against kids and save some money.

There was a time when slavery was common place, but that certainly didn't make it right. Just because kid discrimination has now become widely accepted does not make it right. If you encounter discrimination against your kids when you travel, please write up a review making this clear so future families can plan accordingly.

A better grass?

I came across this article about a newly developed grass that does not require the normal intense life-support we all have come to assume is "normal".

You don't have to water it (after the initial seeding), nor apply pesticides nor fertilizer. And it only requires mowing once per month instead of the typical once per week schedule for life-support grass. It was designed to simply survive, naturally, in our challenging northeast climate.

I've always felt that such a grass must exist, but that the existing grass seed companies would not be interested in pursuing it. See, if the grass simply takes care of itself, we all will buy much less grass seed over time. The lawn care service industry will see much less business, mowing our lawns monthly instead of weekly. Manufacturers of pesticides and fertilizers and lawn care equipment will see less demand, etc. It's quite clearly not in the interest of the lawn care industry to pursue nor allow such innovation.

I sure hope this grass is successful, but the pessimist in me expects that in a few years time, either this company will have been sued out of existence, or the rights to this grass will have been purchased for a princely sum, and then promptly shelved, by one of the big established players in the grass seed industry.

For better or worse, capitalism favors waste in mature markets.

Today, on my morning run, I saw a student walking, late for the bus. The bus saw her walking, way down the road and so stopped and waited for probably two minutes or so for her to catch up and get on.

This might seem like only reasonable behavior, on the bus driver's part. S/he was being nice, right?

As crazy as it sounds, while it was a nice thing to do, I don't think the bus should have stopped. Here's why.

It sends the message that one student's inability to be on time is allowed to cut into the time the rest of the students get at school. The needs of the one outweigh the needs of the many (thank you Spock). It's only two minutes, but if this happens a few times on the route, day in and day out, that adds up to net/net less time at school for all the kids.

The rest of the students, who made the bus on time, probably having rushed through their morning at home to do so, pay the price for those students who can't make the bus on time. They will conclude that they, too, can be a bit late and the bus will wait. Why bother rushing to be on time? Rather than being taught that they should try hard to make the bus on time, to take responsibility for not making others wait, they are taught the reverse.

Finally, seeing the bigger picture, this teaches kids that the world will stop and wait for them. Make up for their faults. Be forgiving. That you need not try very hard for things because the rest of the world will compensate. You need not take responsibility. It ties right into the dangerous sense of entitlement that many kids seem to have now. For better or worse, the world simply is not like that once you grow up.

She should have simply missed the bus and learned a good lesson.

Fun questions

Here are two fun questions I've [temporarily] stumped my kids on:

• How can gravity make something go up?
• How can the moon get you wet?

Spell correction

Spell correction is a challenging feature for search engines. Unfortunately, it's also crucial: mis-spelling is rampant when users run searches. In part this is because we all can't remember how to spell, and that's no wonder: the number of English words today is 5X what it was in Shakespeare's time! But it's also because we are simply in a hurry, or, lazy, and make many typos.

I rely on aspell when I'm using emacs. Modern web browsers and word processors check the spelling of all text you enter. Web-side search engines have excellent spell correction; in fact, I no longer bother to correct my typos when entering a search. I've often wondered whether such "crutches" of our modern world are in fact weakening our minds and perhaps causing our language to further evolve? For example, I wonder how Microsoft Word's often wrong (in my experience) grammar checker has crimped "modern" writing.

My Chemistry teacher in high school refused to allow us to use calculators during our tests, for fear that we would lose our ability to do math with only basic tools (paper, pencil, brain, hands). My Physics teacher did the opposite, for the reverse fear that the distraction of doing basic math would take precious time and thought away from actually thinking about how to solve the problems. Who's right?

Google clearly sets the gold standard for respelling, that any search engine is now required to live up to. If you don't match that high bar, users are automatically disappointed. And you really don't want to disappoint your users: it's nearly impossible to get them to try out your new application, and, they often don't give second chances.

For most approaches to spell correction, the more data you throw at them the better they perform. If you have lots of queries coming in, you can use that as your sole source. Google of course has tons of queries to tap into. If you are less fortunate, you can use your index/documents as your source. Both of these approaches assume most people know how to spell well! The assumption seems to hold, for now, but I have to wonder, as we all lean on this crutch and become worse at spelling with time, won't this eventually undermine Google's approach? No worries; Google will adapt. This is not unlike investing in index funds: that approach only works well if relatively few people do it.

Lucene's basic spellchecker package, under contrib, which requires you to provide a Dictionary of "known words", allows you to derive these words from your search index. It has some limitations: it can only do context-free correction (one word at once, independent of all other words in the query); it doesn't take word frequency in the index into account when deriving the index (so if a typo gets into your index, which can easily happen, you could end up suggesting that typo!); etc. But it does provide a pluggable distance measure for picking the best candidate. It's a good start.

One particularly sneaky feature to get right is spell correction in the context of entitlements; my post this morning on Lucene's user list raises this problem in a real use case (single index to search multiple user's emails). Entitlements means restricting access for certain users to certain documents. For example, you could have a large search index containing all documents from your large intranet, but because of security on the intranet, only certain users are allowed to access certain documents.

Lucene makes it easy to implement entitlements during searching, by using static (based solely on what's indexed) or dynamic (based on some "live" external source at search time) filtering.

However, properly doing spell correction in the presence of entitlements is dangerous. If you build a global lexicon based on your index, that lexicon can easily "bleed" entitlements when there are terms that only occur in documents from one entitlement class. This might be acceptable for context-free spell correction, but if your spell correction has context (can suggest whole phrases at a time) you could easily bleed a very dangerous phrase (eg, "Bob was fired") by accident.

So, you might choose to splinter your spell correction dictionary by user class, but that could result in far too little data per user class. I'm not sure how to solve it, well; it's a challenging problem.

I hope I haven't mis-spelled any words here!

The best way to learn is to teach

They say the best way to learn something is to teach it. Well, I say writing a book sure counts as teaching because writing the 2nd edition of Lucene in Action sure has taught me all sorts of juicy details about Lucene, far more than I would have learned on my own.

I can also say that writing a book about an active open-source project is very demanding. I try to keep the manuscript current, as changes are happening to Lucene, but then more than once I've been burned by keeping it just a little too current, only to see that the community up and changed its mind on something I had already folded into the book's manuscript and source code!

Finally, as Lucene is getting very close to releasing 2.9 I'm now scrambling to fix the loooong tail of little things all throughout the book. Even once I finish that, it's several more months for a deep technical review, Manning's production process, etc. I'm looking forward to finishing!

A kid's mind

I just love how kids think. It's so carefree and unrestrained by all the silly "limitations" we adults have learned with time.

Here's an example: we just finished a vacation with cousins (my kids' cousins). They were all in the car yesterday, eagerly discussing Halloween and who's going to wear what costume. But then they all realized and lamented that in fact they would not be together for Halloween. So, immediately, my son said "Dad, can we fly to California for Halloween?".

It just tickled me pink! See, we live in Boston, MA, so flying to California is easily a 9 hr affair, one way, "door to door". Not to mention, expensive! But my son's thinking of course wasn't restrained by such silly things.

As adults the idea would never be allowed to cross our mind. Somewhere, deep in our brains, is a group of neurons that quickly and mercilessly kills off such thoughts before we can even think them. But there's no such limitation, yet, with a kid's mind.

If you spend even a small amount of time with any child, you'll see many examples of this unrestrained thinking, and it's delightfully refreshing. We all should strive not to grow up. It's easily the best thing you could do for yourself!

On bottled water

The bottled water industry is truly silly.

It's an enormous money-maker, a $20B industry in the US alone. You're buying a product that's hundreds of times more expensive than tap water. Yet, often the bottled water simply comes from a municipal source anyway. Furthermore, it's easily less safe than your tap water. See, the EPA has tougher regulations for tap water than the FDA has for bottled water. For example, the FDA allows some contamination of E Coli, and does not require testing for known parasites such as Cryptosporidium or Giardia Lamblia. Likely your bottled water does not contain Fluoride as well.

Not to mention the insane consumption of oil required to schlepp around all this bottled water and then again to discard the empty plastic bottles. You should of course recycle them, but precious few of us actually do and so they fill up landfill, a "gift" from us to our future generations. Or perhaps your empty bottles end up in the Great Pacific Garbage Patch.

And as if all of this weren't already reason enough to avoid bottled water, there is the curious problem of the chemicals in plastic, such as Bisphenol A (BPA), leaching into the water over time. Previously, it was believed that these chemicals didn't easily leach unless the plastic was hot (this is why you're not suposed to put plastic containers in the microwave or dishwasher).

However, this delightful study showed that simply drinking bottled water increased BPA in urine by 2/3rds. Perhaps bottled water should include a clear "bottled on" date so you can at least roughly gauge how much BPA you're about to drink.

Plastic is clearly an incredibly useful material, and I'm sure we'll eventually sort out the problems of the various chemicals that leach from it. In the meantime, I simply play it safe by avoiding plastic touching our food/drink, when practical. For example, we only put glass ware in the microwave, and when we need to carry water on-the-go, we always use a Klean Kanteen. In fact we now have many Klean Kanteens: in the car, in the stroller, next to the kid's beds and in our home offices, on the dining room table, etc.

Hurricane Bill is coming!

I find myself, this quiet Saturday morning, in Falmouth MA, staring down the barrel of Hurricane Bill! Seriously, it's headed straight for us, having gained strength last night.

Yet, we're planning to go happily to the beach this morning, anyway. Why play chicken with a hurricane? Because the computer models at the National Weather Service insist that Bill will take a turn northward, sometime very soon, and not in fact touch Cape Cod at all (though it is projected to make landfall in Nova Scotia Sunday PM).

We place alot of confidence in our computer models these days, and I sure hope they're right.

I was in Falmouth for Hurricane Bob in 1991 and it was stunning. Have you ever tried to stand up when 100 mph sustained winds are blowing at you? It's quite an experience, and the resulting damage was unreal.

Anticipation is half the fun

I've found that for most people, but especially kids, the anticipation leading up to something is a sizable part of the fun. If you have an exciting vacation coming up, take every chance to remind your kids that it's coming, what the plans are, etc. If you don't, they've missed out on half the fun!

I'm also convinced that this is why so many people can go to the Disney parks. The lines are amazingly long, and the actual rides amazingly short. It's the anticipation of going on a ride that keeps you happy.

Life support grass

Seriously, could we possibly have picked a worse plant for our yards than "grass", even if we tried?

I mean the stuff is so sickly, it has no prayer of living on its own in our local climate. It requires massive amounts of life support just to barely eek out an existence.

I'll admit, with all the life support, grass can be truly beautiful. But the price we pay to reach that beauty is ridiculous.

We dump hundreds of gallons of water on the stuff, every night, using sprinkler systems now built standard into the ground for new homes. This of course leaches all nutrients from the soil. Worse, grass is nitrogen-leaching: it removes nitrogen from the soil. So we must dump on loads of fertilizer to put it back.

Being on such luxurious life support, the grass grows like it's in a jungle, and so we are forced to mow it, at least once per week. While leaving those clippings in place would make great natural fertilizer (after all, this is where all the fertilizer went!), it's not pretty so we truck the clippings away and dump them somewhere else.

Of course, lots of other plants thrive, too, even better than grass. We like to call them "weeds", since they are not grass. And so we must dump toxins over the yard to kill them. We dump separate toxins to kill all sorts of bugs. Sometimes, from too much water, a fungus develops, so we dump something else on to kill that.

All this stuff we dump on the yard likely endangers our kids, us, and our pets, but somehow we don't seem to care. It also kills off the worms that'd naturally aerate the soil, and so we must do our own forced mechanical aeration. It messes up the pH balance, so we dump yet more stuff on (lime, sulphur) to fix that.

In fall, when the leaves drop on the weak grass, we must quickly rake or blow them off, because the grass dies off quickly if it's left covered by leaves.

After winter, which the grass barely survives, some of it has died off and turned brown. This is fully natural, and that dead grass would normally serve as nature's fertilizer, yet we don't like the color, so we dethatch and reseed.

These practices don't end with grass, of course. We truck in loads of mulch, the lipstick of modern lawn care, each year. We spray all sorts of toxins on the trees, the bushes, etc. Terminex shows up, spraying all sorts of other toxins. People with blowers show up and blow every last little thing off the asphault of your driveway.

This whole ritual is now commonplace. It's assumed, accepted and expected practice. If you don't subscribe to the life-support grass movement, people think something is wrong with you. How did we get ourselves into such a mess?

Can't we, instead, find a plant that has no trouble surviving in our natural climate, with zero life support? Why did we all fall in love with this sickly life support grass, anyway?

For example, crabgrass thrives. It's very hardy, grows with no additional watering, takes care of seeding by itself while grass never succeeds in seeding itself (presumably it's been selected and bred not to). Clover is another example, and has the advantage of being nitrogen fixing (the exact opposite of grass): it extracts nitrogren from the air and puts it back into the soil. This is why it's such a dark green even without fertilizer.

Surely we can do better.

ZFS and ECC RAM

I started this thread over on the ZFS discuss list.

My question dug into why the ZFS bigwigs alway so strongly recommend ECC RAM. Was it simply for the added security of preventing a few corrupted files (because non-ECC RAM will likely flip a few bits in the lifetime of your computer)?

Or... was there something more spooky going on such that something catastrophic (losing an entire RAID-Z pool) is possible if your RAM has a bit error at a particularly bad time?

This is important to know, because when spec'ing out a new server, you need the facts in order to make proper cost/benefit tradeoffs. This decision should be no different from whether I should get the latest and greatest hard drive (risky, since it has no track record), or an known-good older generation drive that has less capacity and performance but has a good record. If non-ECC RAM means I risk losing the pool then I'll fork out the extra $$$!

It's great to see that ZFS has such a vibrant community that my simple question received so many answers. In this day and age the health of the community behind the software you are using is more important than the health of the software itself! Lucene also has a great community, though, I'm biased!

My thread also indicates one of the challenges with open-source: sometimes you can't get a "definitive" answer to questions like this. Many people chimed in with "opinions", on both sides, but if I tally up the votes, and take into account the number of posts (rough measure of "authority") behind each vote, more people say "a bit error will just corrupt files/directories" than "a bit error can lose the pool".

The dicussion also pointed out this very-important issue, which is to create a way to rollback ZFS to a prior known good state. It's the closest ZFS will get to providing something like fsck, I think. Sort of spooky ZFS doesn't already have that. I hope by the time I need it, if ever, this issue is done!

WDTLER and WDIDLE3

Western Digital states that the Caviar GP drives are not recommended for RAID arrays, and that instead you should get their enterprise RE-4 drive. But there's a $100 price difference between the two right now! ($230 vs $330 at Newegg). So I decided to risk it and build my RAIDZ array with the GP drives. Check back in a couple of years to see if I have any regrets!

In building the array I discovered two very important fixes I needed to make to the drives, in order to make them behave more like the RE-4 drives.

First was to enable Time-Limited Error Recovery. This tells the drive to NOT make ridiculous efforts to recover a sector that it's having trouble reading, and to instead quickly report back an error that the sector could not be read. See, if the drive takes too long to answer a read request, the RAID level will assume it has gone kaput and boot it from the array. By enabling TLER, you prevent this from happening, thus letting the RAID level handle the error. Use the WDTLER utility to do this.

Second, the GP drives have a feature called Intellipark, which parks the drives heads (moves them off the platters) so as to reduce air resistance drag on the motor that spins the platter (every little power saving counts!). You can hear it clearly when it kicks in: it makes a slight clicking sound when parking. When you need to use the drive again, there's a clear delay and new clicking sound while the disk head unparks.

While nice in theory, it's unfortunately rather frustrating in practice. See, modern OS's use write caching to gather up a bunch of writes in RAM, and only actually write to the hard drives in bulk, every 10-30 seconds. The GP's idle timer is 8 seconds by default (a rather poorly chosen default). As a result the drive incessantly parks and unparks as random services write a few bytes here and there. Eventually, too many such cycles (I've read in forums that 300,000 is the spec'd limit) will cause wear & tear and increase the chance of failure. This thread on the Linux Kernel mailing list gives some details. While this is a problem even in non-RAID settings, it's exacerbated by RAID because now you have N drives that park/unpark, in sequence.

Fortunately, there's another utility called WDIDLE3 that lets you increase the time (to a max of 25.5 seconds, which I don't think is enough), or to disable the timer entirely, which is what I did.

If you don't run Windows and thus cannot directly run these EXEs, one simple workaround is to slipstream them into the Ultimate Boot CD as described here. Those instructions are for WDTLER specifically, but simply slip in WDIDLE3 at the same time. Keep the resulting CD accessible since you'll likely need to run it again if you have to replace any drives in your array!

As best I can tell, Western Digital does not officially support these utilities, so use them at your own risk. They both worked fine for me, on OpenSolaris, but your mileage may vary!

Newegg vs Amazon

I'm using 6 of the 2 TB Western Digital Caviar GP drives in my new build, in a RAID-Z array. Despite reading horror stories online, eg the many users seeing drives die quickly in the customer reviews at Newegg, mine are working great despite the sizable stress tests I've been running.

Except: one of my drives keeps reallocating sectors. I see this in its SMART diagnostics (5 sectors as of 2 weeks ago, 14 reallocated as of yesterday). This isn't normal (eg, the other 5 drives have 0 reallocated sectors), so I'll be keeping an eye on it and at some point might ask WD for a warranty replacement. I wonder if there's an accepted "policy" on how many reallocated sectors is too many? This reminds of the numerous "how many dead pixels are too many" discussions for new LCD monitors.

Of course, I don't lose any data because of this; ZFS's RAIDZ simply corrects the error for me.

I bought 3 of the drives from Newegg and 3 from Amazon. If I were more patient I would have spread them out over time as well. In general you should buy your drives across space and time, to minimize the chance of "correlated failures". If you buy all your drives from the same place, it's likely they were manufactured in the same "batch" which means any manufacturing defect in the production of that batch would make it more likely that you'd lose 2 or more drives at once, thus destroying all your data.

Newegg, it turns out, does a poor job shipping hard drives. They simply wrap them in bubble wrap and tape it up, sometimes packing 2 drives together inside the bubble wrap. What they don't realize is, because of rough handling from UPS, those bubbles pop, one by one I imagine, during transit, such that by the time I receive it, there is zero protection (no bubbles left) along at least one edge of the hard drives. It's rather shocking because Newegg is otherwise excellent. I've read several posts in the user comments noting exactly what I just said, yet Newegg hasn't improved. It's a bad sign when a company stops listening to its customers.

In contrast, Amazon (whose price matched Newegg's) packed each drive into it's own dedicated foam packing and box. Fabulous!

[EDIT Jul 28, 2009: I just received one more drive from Amazon, and they unfortunately have taken a turn for the worse! They now ship in a similar fashion to Newegg, wrapping the drive in minimal bubble wrap which pops during transit. They also take the wasteful step of "box within a box", which I don't think adds much protection to the drive. This drive will be my "hot spare", so if/once it get swapped into the array, I'll try to remember to watch for reallocated sectors and any other problems. Sigh.]

The one drive I see failing was in fact one from Newegg (I kept track of the serial numbers); it's entirely possible Newegg's poor shipping and the rough handling from UPS led to this drive's failure.

Losing one drive in a RAID array is quite terrifying because until you get the new drive resilvered, you're running with no safety margin! If you lose another drive, you've lost all your data. RAIDZ is not a replacement for good backups. It's best to have a spare drive on hand; you can even install it and notify ZFS to keep the drive as a hot spare, meaning if any drive drops out of the array, ZFS will immediately start the resilvering process to bring the new drive in. Or you could create a RAIDZ2 array, which has two drives worth of redundancy, but then you've "lost" 2 drives worth of storage!

OpenSolaris challenges

Whenever I encounter someone who's overly ecstatic about some new technology or gizmo or something, I quickly say "tell me what's wrong with it". If they can't think of anything, then I can't trust their opinion.

Nothing is perfect. There are always tradeoffs to be made. Only once you are properly informed with the facts, clearly seeing the goods and the bads, minus all the hype, can you finally make a good decision.

If you are passionate about something, and you use it day in and day out, then you ought to have a big list of the things that bother you most about it. Next time you see someone loving their iPhone, try asking them what's wrong with it.

Unfortunately, hype, "popular opinion", "conventional wisdom", "everybody's doing it", etc. drive so many decisions these days. Not long ago, when you bought a house, everyone pushed you to choose these newfangled mortgages like ARMs, interest only loans, etc., instead of the boring old-fashioned 30 year fixed rate mortgage. Alan Greenspan was giving speech after speech praising the "innovation" in the financial services industry. Look where that got us!

I came across this quote recently: "If you find yourself in the majority then it's time to switch sides". I've been realizing lately how true that is.

So in this spirit of presenting a balanced picture, here are some of the challenges I've hit with OpenSolaris:

• It took practically an Act of God to switch from a dynamically (DHCP assigned) IP address to a static one. I ran the nice GUI administration tool, made the change, and at first all seemed good. But then on my next reboot, appparently a bunch of services failed to start. After much futzing, it was only when I uninstalled VirtualBox that things finally worked (I think VirtualBox's virtual adapter somehow conflicted). I now have a static IP!

• There is apparently no SMART support for SATA drives, which is stunning. These days, as drives become more and more complex, we need access to their diagnostics. I rely on SMART to monitor the health, temperatures, remapped sectors, etc. of my drives.

• The 1-wire File System has not been ported to OpenSolaris. I have a network of 1-wire devices in my house to monitor temperatures, eg, outdoors, in the kid's bedroom, the attic, etc. I'm still working on this one... there seem to be some problems talking to libusb. I may end up simply running a tiny Linux PC (the Fit PC 2 looks cute) instead, for such random services.

Sun's ZFS filesystem

I've been test driving Sun's (now Oracle's!) OpenSolaris (2009.06) and ZFS filesystem as my home filer and general development machine.

I'm impressed!

ZFS provides some incredible features. For example, taking a snapshot of your entire filesystem is wicked fast. This gives you a "point in time" copy of all files that you can keep around for as long as you want. It's very space efficient because only when a file is changed does the snapshot actually consume disk space (preserving the old copy).

From the snapshot, which is read-only, you can then make a clone that's read-write. This effectively lets you fork your filesystem, which is amazing. Sun builds on this by providing "boot environments", which let you clone your world, boot to it, do all kinds of reckless things, and if you don't like the results, switch back to your current safe world again, no harm done. I used to leave my home filers pretty much untouched once I started using them for fear of screwing something up. Now with boot environments I can freely experiment away.

I have a great many Lucene source code checkouts, to try out ideas, apply patches, etc., and by using ZFS's cloning I can now create a new checkout and apply a patch in only a few seconds. And it's very space efficient because only the changed files in the new checkout consume disk space. Since I'm using an Intel X25 SSD as my primary storage, space efficiency is important. The machine uses Intel's Core i7 920 CPU, which has fabulous concurrency and can run the Lucene unit tests 3X faster than my old machine. This all nets out to wonderful productivity gains.

ZFS also nicely decouples the raw storage device (the "pool"), from filesystems that pull from that storage. For the secondary storage I set up a RAID-Z pool (like raid5, but fixes the "write hole" problem) using 6 of the Western Digital Green Caviar 2TB drives. Be sure to use the WDTLER utility if you use these drives in a RAID array. This gives me 9TB usable space to play with; from here I've created many filesystems that all share the pool.

Performance is excellent: copying a 1TB directory on the RAID-Z pool to another directory on the same pool averages 100 MB/sec.

I also just read this morning that ZFS will add de-duping at the block level, thus making it even more space efficient.

ZFS can provide these features because it has a write-once core: no block is ever overwritten (unless it was already freed). Lucene has the same core approach: no file is ever overwritten in the index. Lucene's transactional semantics derive directly from this as well (though Lucene can't "fork" an index... maybe someday!).

Bye bye Linux, hello Solaris! I only hope this innovation continues now that Oracle has acquired Sun.