I just bought a couple of toys. I guess they're an early father's day present to myself :-) I got a maxtor 160GB external harddrive from etech4sale for about $80. It's unbelievable that a gig of storage costs 50 cents. A terabyte is now around $500. And the even more amazing thing is that it's not insane that I'd actually use a terabyte of storage. The sum total of my data (mostly media of course) is probably around 150GB right now. It's not insane that folks who have a huge movie collection (as big as my CD collection) would be able to use up a TB. Filesystems have to be designed differently for that kind of storage requirement - first, the typical block size (4-8K) seems a little small for what you're actually doing with the data; second, the searching/organization capabilities of my OS/filesystem are more or less useless when you think about what I really want to do - e.g. search for a particular sentence or phrase in my audio or video file. I used to think that the biggest issue with big hard drives would be an explosion in the number of files I would have - and that with that small block size I would have to have a deeper B+tree, I wouldn't be able to keep enough levels in memory, and so each attempt to access a disk block would result in an increasing number of seeks (as the depth of the tree grows). Not to mention all the files and all those directories I would have to maintain and navigate... Instead, my biggest issue is not "where all my stuff is" - it's "how do I find stuff inside my stuff". The next frontier for search? (of course none of these are new ruminations - my friend William has talked about this for years now).
Read More...