Journalspace Gets Creamed

If you can't be a good example, then you'll just have to serve as a horrible warning. — Catherine Aird

By now, I'm sure just about everyone will have heard about the disaster that has fallen upon the poor SOBs at journalspace.com. The short story is that the server that hosted all of the data for the blog site got hosed, and lost all of the data. Some of the high points:

• The data were stored on a RAID1 array - a pair of mirrored drives
• There were no bakcups, or any backup system in place at all
• The drives did not fail, but were both completely overwritten on every block
• No conclusive root cause was found, but a recently departed sysadmin had already been caught doing "a slash-and-burn" on other systems

So in the end, it looks extremely likely that an incompetent sysadmin set the system up with no meaningful backups, and then progressed to a malicious sysadmin by performing a thorough wipe of the only copy of the system data as he was shown out the door. What a wonderful cornucopia of lessons that can be gleaned from this one example! This is the kind of thing that you expect to see as a hypothetical scenario in security textbooks, not on the front page of Slashdot.

So let's take a quick rundown of lessons learned from our hapless friends.

Backups, backups, backups.

The lack of external backups is what catapulted this from an outage and a headache for the remaining sysadmins into a practically worst case scenario. In short, mirroring is not the same as backing up.

Trust, but verify.

Just because you implicitly trust your sysadmins (otherwise they can't do their jobs) doesn't mean you shouldn't keep an eye on them. Use sudo to log commands, monitor configurations via tools like RANCID, and Puppet or Bcfg2.

Watch the watchers.

Along the same lines, don't let one person exclusively handle any important project. One bad apple working in isolation will have a much, much easier time planting logic bombs than one who has one or two others working side by side.

Don't give them a chance to pull the trigger

Going to fire a sysadmin? Any hint of a possibility of a chance it might get ugly? Be prepared to make sure that any and all rights that admin has are completely gone by the time they know they're getting fired. And please note that most sysadmins will take sudden revocation of their rights as a hint they're getting fired, so the chat with HR should probably happen simultaneously with at least two other trusted admins pulling rights and locking accounts.

Cleanup after their messes.

Dislike a sysadmin enough to get rid of them? Then that same dislike and mistrust should extend to all of the work they've done for you. As soon as they're out the door, it's time to audit what they did. Make sure the work you didn't know they did is up to standards, and make sure to look for backdoors and time bombs.

It's too late for those poor souls at journalspace, but hopefully they'll at least serve to inspire others to fix something.

Priorities

A few years ago, when I stayed in some hotels, an Internet access was just becoming really standard, there was a per day fee to use it. On the other hand, the hotels also served a decent complimentary continental breakfast for all guests.

Now, when I recently stayed at a few hotels, Internet access was completely free, but now the free breakfasts are gone.

My, how priorities have shifted.

DoD Computer Security Decides to Pull Pants Up

As I'm sure everyone has heard the Department of Defense has decided to ban all removable media from their computer systems, mostly due to viruses running rampant throughout the (presumably) otherwise secure networks.

Now, people have pointed out that this is a pretty drastic step. After all, there are other ways of handling things that could have theoretically prevented this particular problem without inconveniencing users quite so much. Up to date virus scanners, security policies disabling autorun, restricted privileges on user accounts - all of these things would have helped reduce the ability of such a virus to spread. They should all be considered pretty basic measures in any reasonably high security environment, and it's quite possible that they were at least partially in place.

But there's an elephant in the room that I haven't seen anyone else mention, and would like to point out. Microsoft declared its security Initiative in 2002. In the six years since, we've had two major service packs, and a whole new OS.

So will someone please, please, please tell me why, in this day and age where security breaches make the news weekly, the default behavior for Windows is still to take any newly inserted media and automatically try as hard as possible to run whatever it happens to find on it? It was simply annoying on Windows 95, but it's downright dangerous now.

Come on, Microsoft. I would expect that any operating system that calls itself "Professional" would show a little more restraint than a two year old trying to eat a piece of gum it just peeled off a New York sidewalk. Time for Windows to grow up a little and break this dirty habit.

Money Makes the World Go Round

It's no big secret that the financial world is going through what can be kindly described as a catastrophic disaster. Stock markets, profit margins, layoffs - all of the meters are currently pointing somewhere between bad and worse.

Likewise, there are plenty of people out there expounding on how we got into this situation, mostly pointing at the various shell games that Wall Street has been playing with mortgages. I don't really have anything to add on the twenty plus year saga of how we've made a bubble big enough to take out neighboring markets when it popped.

Instead, I just have a very simple observation to make. One that the entire financial industry has not simply forgotten, but must continually and actively ignore in order to continue to exist.

To put it bluntly: money has no intrinsic value.

Now, before you just laugh at me, think for a moment about this idea of value, or utility. While the utility of something can vary widely from person to person, and place to place, some things are more universal. For example, no matter who you are, food has some value. Everybody eats. The value of food can be influenced by the skill with which it is prepared, or the ration between its supply and demand, but its base value is directly created by its intrinsic properties. A pound of rice is always a pound of rice, and can always be made into a meal.

So the question, then, is where does the value from money come from? Or to put it a little more viscerally, why is having a pocket full of cash better than nothing but an empty wallet?

The answer, obviously, is because you can buy stuff with it. But what if you took that option away? What good would that money do you in everybody's famous hypothetical scenario, stranded on a desert island? Quite simply, none! A hundred bucks worth of military rations would be a thousand times more valuable than a hundred dollar bill. Moneys value springs purely from our collective agreement to pretend it has value. When you take away the ability to convert money into something else with immediate value, you remove the indirect value of money, revealing its utter lack of intrinsic value.

If you're still not convinced, then ponder this riddle. If the carefully crafted metallic sculpture that we call a "coin" and mass produce at US mints has value, then why doesn't an exact replica that came from someone's basement also have the same value?

What we call the financial trading world, though, is built upon a willful ignorance of this fact. The industry is built upon layer after layer of abstraction, and at each one, the intrinsic value that is abstracted into money is further diluted.

Consider day trading. Throughout the day, any given stock will have some degree of fluctuation. Even if it ends the day at exactly the same price it started at, there will be points where the price is up, even if only a few cents, and other points where it is down. With modern computers, it is possible for even a casual investor at home to have automatic orders rapidly buy and sell the same stock over and over again. Buy the stock at $1.00, sell it at $1.05. Wait for the stock to fall back to $1.00, and do it again. In the days of conducting business over the phone, the cost of the phone calls alone could easily have swamped any profits made. In the days of computers, though, anyone can cheaply run the switch a thousand times a day, with tremendous cumulative effects.

But where did this value behind the money come from? No work was done. No commodity was created. No service was performed, or even promised. Nothing was proffered for this creation or transfer of wealth, not even a kind word. The whole stock market system was intended to be, like currency, an abstract representation of underlying value. A share of stock in a company is a voucher for a fraction of the total intrinsic value of that company.

With the introduction of computers and near instantaneous trading, though, the rules changed. The speed upped the pressure behind this loophole, and money suddenly started gushing through with disregard for the rules. Why bother with all of the tedious research, hoping that the stock will go up a substantial amount, when you can make money off of random noise? As long as the stock doesn't completely tank, you're fine!

Day trading was by no means the first means of exploiting a loophole. But in the last few decades, as regulations have simultaneously become more byzantine and less restrictive, the opportunities for making money by creatively shuffling money around have become more potentially lucrative and tempting. Why go through all the effort of actually creating intrinsic value, when you can not only carefully stack up your bills to make one plus one equal three, but do it a thousand times over?

Yahoo! Music Store

Okay, so the Yahoo! music store is the latest one to shut down, taking with it the DRM authorization servers required to use the "purchased" music. (Since I've never touched Yahoo! music, I have no idea if the DRM servers are required to play music, every 90 days, when you want to move computers, or what. The relevant bit is that you'll run into a problem sooner or later with the servers gone.)

This has been covered before, so let's just quickly recap:

1. You can't buy DRM encumbered media, only lease with an option to get screwed.
2. The option to get screwed is exercised at the discretion of whoever owns the DRM infrastructure.
3. Do you think that the company actually wants to keep a whole collection of servers up and running for the last three people using purchases from a music store that was discontinued 4 years ago in favor of a new, more profitable one?
4. Revoking DRM is a brutally effective method of forcing consumers to leave an old platform, in hopes they'll all sign up for its successor. The fact that customers were happy with the old platform isn't perceived as a downside; it's the reason why the company is doing it in the first place.
5. Strong DRM means that companies can use technical means to enforce policies, regardless of their legality. Existing code doesn't automatically update to reflect new court rulings, and your only appeal is with the companies helpdesk.

All those of you who have been writing about the dangers of DRM may now proceed to jump up and down while shouting "I told you so!"

Lies, Damned Lies, and Marketing: A Plea to Netflix

I work in IT. Not surprisingly, this means I get to do a fair amount of support for broken computers. In my case, it's mostly for a handful servers from a particular well-known vendor. Since we pay a premium for the top-level support, I tend to be pretty satisfied when calling in for failed components. The calls basically tend to consist of "What's broken?", "Let's run a quick diagnostic to make sure", "Do you want a technician or just parts?", and "Do you want it there tomorrow, or this afternoon?".

Then one day, I had to make a call in for a desktop. Same vendor, still had one of the higher level support contracts, and still quite obviously a hardware failure.

Unfortunately, this meant that instead of getting routed to a bunch of IT-savvy techs determined to keep my downtime to a minimum, I got to deal with the general home user support group.

Now, I do enough end user support to be able to sympathize with quite a bit of what these guys go through. I really don't mind them asking me really basic questions like "Is the computer on fire?"; I've had users who would neglect to mention this when asking why we turned off their Internet. I completely understand them strongly wanting to get an error code back before they'd start shipping replacement parts; I wouldn't be surprised if they've had users who didn't understand that you need to put a blank CD in before they can make a mix CD of their pirated MP3s. I won't pretend to like these things, but I understand they're necessary and don't hold it against the poor people at the other end of the line.

No, what bugs the hell out of me when they keep claiming they're "sorry". Yes, that's right, every time I talk to a new person, and every time I mention something that's a problem, they rattle off, all in one quick, unconvincing, insincere, scripted breath, "Oh-I'm-terribly-sorry-sir-I-feel-really-bad-about-that-I-hope-that-we-can-fix-the-problem-and-I'm-sorry-for-the-inconvenience" .

Oh, really? You feel personally bad about every annoying user with a broken coffee cup holder who can't tell you if it's plugged in because the power's out? Bull. After the fourth or fifth time, I'm actually far more annoyed than if you just said "Okay" and punted me off to the tech in line, because it's quite obvious that you're lying to me. I'm paying the extra support money for tech support on the product. If I wanted someone to talk to and empathize with me, I'll to find a qualified therapist and talk about my childhood, thank you very much.

Where was I going with this? Oh yes, Netflix.

As I'm sure that anyone who has a Netflix account, reads techie news sites, has an Internet connection, or uses electricity has heard by now, Netflix is removing the profiles feature, which lets you split up a single account into separate queues and preferences. This lets multiple people share a single account, rather than each buying their own - perfect for households with more than one person.

Now, the canceling of this feature is bad enough. My wife and I use this, and let me tell you, it's a lot easier than trying to come up with ratings that accommodate chick flicks, romantic comedies, sci-fi, and anime. I mean, seriously, how many people really like all of those categories?

As if that weren't bad enough, though, Netflix had to take it one more step. They decided to just give all their profile users a father-knows-best pat on the head, and tell 'em "It's for you own good." Like the tech who personally fells the pain of each and every of the thousand customers per day, Netflix has spun a falsehood that is insultingly transparent:

As a Netflix product manager I'm tasked with the wonderful job of helping members find movies they'll love. But today my job is more challenging as we've decided to terminate the profiles feature on September 1. Please know that the motivation is solely driven by keeping our service as simple and as easy to use as possible. Too many members found the feature difficult to understand and cumbersome, having to consistently log in and out of the website.

Let me get this straight. You have a feature that, while perhaps not wildly popular, is strongly loved by those who do use it. "Some" people allegedly find it "confusing" (we'll assume for the moment that Netflix has legitimate data to back this claim up), so rather than, oh, I don't know, fixing the problem, you just decide to nuke it completely. How does that "help" users?

Now, where Dad could give 5 stars to Goldfinger, Mom could give 5 stars to Pretty Woman, and Junior could give 5 stars to Shrek, Netflix will be trying to analyze a single person that would give 5 stars to all three movies. I can only imagine the bizarre recommendations for such split personality victim! How does that "help" users?

Before, each member of the household would have their own queue, and would get their own next movie for each one sent back. Now they'll have to carefully shuffle the queue each time one goes back to make sure that the right next movie goes back, or else Junior sending the cartoon he just watched back will land Julia Robert's latest movie in the mailbox. How does that "help" users?

If you're going to pull out some backend code that implements this feature, fine - but I doubt there's a software engineer on the planet who thinks it's a good idea to pull a feature away before you have something more compelling to convince your customers to give you money.

If maintaining the feature is taking up too much time, or is getting you stuck in some expensive patent war, then tell us you can't afford the feature and we'll probably understand and get over it.

But please, please, please - don't just rip the feature out of our hands and tell us it's for our own good. It's a blatant lie of the worst kind - a marketing lie - and once your customers think that you're lying to them, they're quite liable to take their money off to one of your competitors in a hurry.

You can trust me on that.

What Time Is It Anyway?

I can't believe that I feel I need to write this post. Really. But, I do, so here it goes.

Start by asking yourself a question: what time is it? A simple enough question, with a simple enough answer.

Now pretend for a moment that you had 100 people scattered around the globe on a conference call. Now ask them all, at precisely same moment, what time it is.

(Hands down, all you physics majors out there. We're ignoring relativity, since this is all make believe anyway, so everyone agrees it happens at the same instant in time.)

Now all of a sudden the answer to your question isn't quite so straightforward anymore, is it? You have to worry about dealing with multiple timezones, the international date line, and daylight savings time. The only way to deal with this is to use dates that explicitly include the timezone. Trying to deal with dates and times missing timezones is like trying to use latitudes without longitudes, or an email address without a domain. As soon as the scope expands beyond a very tiny size, it breaks down quickly.

Now, the fact that just about any standard formatted timestamp includes this information seems like it would make this pretty obvious. Email, HTTP, filesystems - they all either include a timestamp, or are universally defined as relative to a fixed timezone that you can easily base off of.

So will someone tell me why, in this day and age, the derby database chose to define timestamp columns that are missing the timezone? You wouldn't forget to make numerical types with floating point support, would you? Or strings that didn't support storing lower case? Or... well, you get the general idea.

So come on, guys. It's a big world. Databases are all about sharing information, and these days even a modest open source project can easily be sharing between half a dozen timezones across three continents. At this point, you're just making yourselves look silly.