We are at an immense turning point in IT. Almost overnight, storage has gone through the looking glass, changing from something rather limited and expensive to something remarkably cheap and virtually without limit.
We've been waiting for IT to turn this corner since 1980 or so. In 1983, a 20 MB hard disk cost as much as a used Volkswagen, and PCs generally had around 16 MBs of RAM at most. Storage was scarce. Storage space on the chip and on disk had to be husbanded, squeezed, pruned, and reallocated immediately for maximum performance.
For 25 years, software designers faced the same challenge when it came to storage: More memory meant faster performance, but space for code in memory was always extremely limited by the hardware's physical constraints.
Designers usually had to settle on a middle position between maximum performance and optimal code size. Applications whose code was too large to fit on the free space on the chip tended to run too slowly because of the massive paging the size of the code triggered.
The first disk drives I encountered in the late '60s were the size of large washing machines. The drives weren't as noisy as washing machines, but a roomful of them (you needed a roomful to get enough storage for your IT operations) could just about drown out a conversation. IBM rented (never sold) the drives to customers, and so the size helped justify the hefty price. The disks were in a stack inside a kind of semi-transparent cylindrical cake carrier almost 20 inches in diameter.
Apple brought out the Mac in 1984, with an operating system at least 10 years ahead of Microsoft's MS/DOS, but a hard disk (which was optional) for the minuscule Mac cost as much as a new Volkswagen. Storage was priced beyond the consumer's reach, so most Macs never had a hard disk; this means the Macs ran at a tenth of the speed for which they were built.
Then a few years ago, drive technology turned a corner and took us through the looking glass to a place that's just the opposite, where there's almost infinite space to profit from in ways that will speed up your code. The new challenge to the entire IT industry is: Figure out how our software can be improved to take advantage of this suddenly unlimited disk space.
With unlimited disk storage and massive RAM, a totally new architecture is necessary. The basic structure of software and database has to be reconsidered, for example. For more than 40 years, almost all database systems have been using some form of data compression forced on the designer by the limited space available to him on hard disk and in memory. Now, suddenly, no form of data compression is needed for any operation on a computer, making that compression obsolete.
The new storage technology doesn't just bring freedom for the programmer, it also brings a new level of difficulty to the job of backup. No big change is without its price. As someone once pointed out, "Believing in progress is believing in getting something for nothing."
Twenty years ago, many IT pros had most of their hard disk backed up at any time. Today, with these gigadrives packed with monster applications, you'll have a hard time finding anyone who has most of his hard drive space backed up in a recent copy somewhere.
The extra drive space can also be used to increase the frequency of backups; have you done that yet? Perhaps you used to keep seven generations of backup on tape. With megadrives, you can keep 100 generations just as easily or 700 for that matter.
The extra drive space can also be used to build historical depth into databases and data warehouses. Records can be flagged as deleted instead of being physically deleted, and historic copies retained when records are updated, meaning the user obtains the power to look at the history of the values in any field of the record.
With that architecture in place, "Show me every address where this person has ever lived" becomes a matter of a single click. Most corporate databases currently can't even answer this kind of query because of lack of retention of past versions of records due to perennial storage scarcity.
This challenge of using the new wealth of space available to us is one we'll be looking at in subsequent posts.