Indiscriminate Prose

Working around MySQL dain bramange

2011-02-28T18:59:00.006-05:00

I'm currently in the process of porting a perl app that was written for MySQL over to a PostgreSQL backend. Originally it was targeted for MySQL 3.x (yes, the app is that old), long before any kind of transactions or constraints were available, so data updates use this general idiom (those of you used to full-featured databases, you may want to avert your eyes):

High level function locks tables
Perform queries on database to check for consistency (uniqueness, business rule checks, etc)
Pass parameters to lower level update function

Perform global permissions checking
Perform single update/insert/delete as appropriate

Call lower level functions additional as needed
Unlock tables

Later on, in the 4.1 days, rudimentary transactions were added, though constraints were still nonexistent to useless. Simple transaction support was shoehorned in by adding it to the lower level functions.

High level function locks tables
Perform queries on database to check for consistency (uniqueness, business rule checks, etc)
Pass parameters to lower level update function

Begin transaction
Perform global permissions checking
Perform update/insert/delete as appropriate
Commit or rollback transaction

Unlock tables

(I warned you not to keep reading, didn't I?)

So let's recap where we are now, thanks to sticking with what Selko describes as "not really a relational database, but does sequential processing behind a 'near-SQL dialect' language":

No constraints in the database, making direct edits (and let's face it, they always happen sooner or later) are quite dangerous.
Because there are no constraints, all of the checking has to be done in the application layer.
Because the checking is done in the application, all involved tables still have to be locked - MVCC would actually make concurrency issues worse, not better.
Because transactions are handled in the lower level functions, they can't encompass multiple statements. What's the point? (Okay, to be fair they do cover auditing records that the lower level functions silently create, but that's all.)

Ugh. The ideal path going forward would be move all of those constraints down into the database where they belong. This is far from a trivial task, however, and one that I probably don't have the resources for. The checks are scattered throughout the codebase, with odd conditions awkward to express in SQL. One common idiom is a column that references table A if the value is positive, table B if negative, or nothing at all if its zero. If I could wave my magic keyboard, that's the road I'd go down, but it's out for repair, and I need a working app sooner rather than later.

Instead, I'm going to have to spring begin/commits throughout the higher level functions, while still preserving the table locks everywhere. To make things even better, some of the higher level functions may call each other, so I need to simulate nested transactions, which PostgreSQL doesn't quite support (yeah, I know about savepoints.) So now I'm looking at something roughly like this:

If not already in a transaction, begin one.
Lock tables.
Perform checks.
Call low level functions, or other higher level functions as needed.
If we began our own transaction, commit, else pass control back up.

Yeah, it's still pretty fugly, but that's the design tax we're still paying for all of the features left out way back in MySQL 3. Thanks, guys.

Converting MySQL blob data to PostgreSQL bytea

2011-02-28T12:25:00.005-05:00

I just went through several annoying iterations of preparing to load a MySQL data set into a PostgreSQL server, with somewhat less than helpful error messages (why on earth would a bytea column complain about charset errors anyway?) Now that it's working, I though I'd share this little function. It's suitable for use in a script that's post-processing the txt files from running mysldump with the -T option. It takes one argument, a binary strong from a blob column, and returns the same data in an escaped form suitable for loading into a bytea column with the copy from function.

May it save you from a few of the headaches I have suffered.


sub encode_bytea {
   my ($blob) = @_;

   return join('', map { $_ = '\\\\' . sprintf("%.3o", $_) }
       unpack("C*", $blob));
}

How to Lose A Customer in 7 Seconds or WTF, Mozy?

2011-02-19T09:38:00.002-05:00

Ah, Mozy. I first signed up with you a few years ago, when it was basically between you and Carbonite. Almost immediately, you saved my bacon (or, more specifically, a couple dozen gigs of photos and music) from a hard drive crash. Not long after that, I laughed at the poor Carbonite users, their backups trashed by a combination of faulty hardware and lack of cross data center replication.

But that was then, and this is now. Hard drives are cheap, bandwidth is plentiful, and everyone is doing "cloud stuff". So how do you respond to the flocks of competitors? Lowering prices? Adding innovative new features? (Yeah, backing up to local hard drives is a good new feature, but hardly one I'd call innovative.) No, not you! Instead you not only drop the unlimited option, you also raise prices!

I'm currently backing up about 75G of data from a single PC (man, those high megapixel cameras can crank up the file size!) This means the price for me doubled from about $50 to $100 per year. And what do I get for that extra $50? Nothing. Nothing at all. I can't run it on Linux or back up to friends machines for free, live CrashPlan. I don't get to keep my costs down around $50, like Carbonite. I can't sync my backup set between multiple machines, like Livedrive. I can't play my music or view my photos online or on mobile devices, like SugarSync. I get nothing.

I've never been a big fan of paying something for nothing, so farewell Mozy. I've chosen to go with CrashPlan. The browser based file access, weekly status reports, and cross platform support hit all of my needs at the right cost. Although, to be fair, I never would have bothered to find CrashPlan if Mozy hadn't decided to shoot themselves in the foot. I'd still be giving Mozy my money, for fewer features. I guess that puts things in a different light.

Mozy, you may be doing your best to bleed customers, but your sacrifice shall not go completely unappreciated. As a formerly happy, but now happily former customer of yours, you have my gratitude.

DNS Sinkholes... That Work

2010-08-24T16:12:00.002-04:00

One idea that's been ramping up in the last few months is the DNS sinkhole. Essentially you configure your local client facing resolves to always return an IP address of your choice for a list of "known bad" domains. The list of domains will usually come from some trusted source, and the IP returned will either be a completely non-functional one (like a 127 address), or one that points at your local honeypot server. Assuming that you've got a good list of malicious domains, it can be a great way to block viruses and other malware without having to touch any of your clients.

The only problem, though, is that most of the examples I've found... well, kinda suck. They're either limited in scope, less efficient than they should be, or just don't work. So rather than complaining about them all individually, here's the set up that Works For Me®.

First, you need a minimal zone file. All of the malicious zones will get served out of this zone file, so you are restricted to records that will be common to all. This example uses 127.1.1.1 for the target IP address, substitute your own if you want to send the traffic somewhere else.

$TTL    600
@                       1D IN SOA       localhost root (
                                        42              ; serial
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum

        IN  NS  localhost.
        IN  A 127.1.1.1
*       IN  A 127.1.1.1

The SOA and NS records are both just set to 'localhost', avoiding hitting any real records, but still resolving.
Pick any serial you like - it should never change, only the list of zones pointing at it.
The first 127.1.1.1 record causes BIND to return a record for the zone itself. For example, if you pointed "badstuff.com" at this zone file, it would allow any queries for badstuff.com to return 127.1.1.1.
The second 127.1.1.1 record handles anything within badstuff.com - www.badstuff.com, download.badstuff.com, etc. You need both records to get full coverage.

Take that zone file, and stuff a copy of it on all of your servers. Here, rather than bothering to set up the usual master/slave relationship, I just made all of my servers master - much simpler, since none of the usual benefits of master/slave configurations apply.

Once it's in place, create a config block in your named.conf (or included file) for each zone that you want to filter (getting the list of zones is outside of the scope of this post):

zone "badstuff.com" {
    type master;
    notify no;
    file "Malware.zone";
};

This will cause your server to start answering all queries for badstuff.com and anything under it out of the file Malware.zone instead of issuing a recursive query for it. Make sure you include the 'notify no;' line - that will prevent your servers from trying to flood each other with useless notifies.

Here are a few other links you should follow if you're serious about doing real DNS sinkholes.

Wow.

2009-05-01T12:14:00.002-04:00

I hereby nominate this Schneier post for the Understatement Of The Year Award:

Nuclear war is not a suitable response to a cyberattack.

The very fact that such a response was needed is a depressing one indeed.

So long, and thanks for all the transactions...

2009-03-24T20:53:00.004-04:00

Well, it's been a long time coming, but it's finally happening. The breakup will take quite a while, and it may never be a complete break. But in the end, after hundreds of tables and countless rows, I begin the process of replacing MySQL with PostgreSQL as our primary database.

But why, you ask? Not for any of the reasons you're probably thinking of. It's not because of any of the limitations of any of the foreign key implementations. It's not because of the cases where MySQL's serial data processing nature conflicts with what set theory says you should be able to do, such as modifying a table that's referenced again in a subquery. It's not because PostgreSQL is substantially closer to fully standards compliant SQL, and therefore a lot of the commercial big names such as Oracle. And no, we never had any catastrophic data loss due to any less than ACID aspects or a MySQL crash.

No, overall MySQL has been quite good to us. We're switching for the simple and inescapable reason that, unlike PostgreSQL, MySQL can not store IPv6 addresses in a usable form.

Now before you come up with some bright idea, take my word for it - I've already looked at it, thought it through, and utterly rejected it. Once you strip away the slightly funky formatting conventions, an IPv6 address is at its core simply a 128 bit integer. Too bad that MySQL only supports up to 64 bit integers, meaning that the IPv4 method (store it in a 32 bit integer) doesn't work. You can stick it in a text string, but then you get formatting normalization issues, and you can't efficiently perform any bit masking operations, which are required for things like determining which subnet a given address belongs to. Cramming it into a decimal type is a little bit closer, but sadly the bitwise operators such as '|' and '&' will silently truncate the output to 64 bits, making it - and anything else - utterly useless.

Quite simply, as comfortable as we are with MySQL, IPv6 addresses just don't fit.

And then, like a ray of sunshine, there's PostgreSQL! It actually has a pair of native data types (inet and cidr) that can not only cleanly store either an IPv4 or IPv6 address and subnet mask or prefix length, but also work with a number of functions for some of the most common operations, such as extracting the network address or calculating the broadcast address. What is currently a flat out impossibility in MySQL is not only possible, but trivial in PostgreSQL.

My group is responsible for maintaining the network here. We live, breathe, and die by IP addresses. We register them, track them, and shuffle them around between databases, DNS, DHCP, and ACLs a hundred times a day. Imagine trying to sell a database to the phone company that can't store phone numbers, and you'll have a pretty decent feel for the position we're in with IPv6 addresses. IPv6 adoption may be slow now, but it's going to come sooner or later, and when it does come, it's probably going to hit critical mass and come fast and hard.

Sure, I'm not looking forward to the changeover. I need to worry about removing MySQL specific features from the schema, such as auto increment fields and set/enum data types. I have several gigs of absolutely mission critical data that has to be moved over without getting scrambled. And there are many, many lines of perl and SQL code that have to be tested thoroughly. But in the end, "difficult but possible" beats "not a chance" any day.

Patents Vs. Innovation

2009-03-19T10:24:00.002-04:00

If you follow the tech industry at all these days - and possibly even if you don't - you can't help but hear all about patents. Whether it's Microsoft suing TomTom over ridiculous FAT patents, or RedHat reluctantly assembling a defensive patent arsenal, the one thing that every agrees on is that patent law and practice has a huge impact on both the tech sector, and the economy at large.

One aspect that the guys over at Techdirt have continued to hammer at is the difference between invention, and innovation. Patents place a significant emphasis at protecting the "rights" of the inventor (rights which exist only because of patent law - how's that for circular reasoning for you?), to the point where innovation, the act of making the invention actually useful, is being harmed. I've come up with a simple analogy that I think helps clarify this argument.

Imagine that you're a manager with a couple dozen employees under you. (If you actually are a manager, this should be pretty easy.) Now, odds are that most of your employees are pretty decent. They're good at their jobs, you can rely on them to produce good quality work, but they don't often come up with game changing ideas.

Except for that one guy. You know, that one guy who, if you had to pick your replacement, you'd name in a heartbeat. He's the one who doesn't just come up with a way to make a work process faster, he shuffles things around and makes that entire process disappear, freeing up everyone's time to work more on things that make money.

Now picture he's come up with a new version of some form that everyone has to fill out fifty times a day. The new version is faster, more accurate, and will allow everyone to increase their client billable hours by 20%. Great! Everyone can start using it, the company makes more money, everyone gets raises, and everyone's happy.

Except that in this case, we're modeling our little company after the patent situation we're in. Now, the only person who can use the new form is your one go to guy. Everyone else is stuck with the old form. The company bottom line barely moves, no one gets the big raise, and everyone gets to just watch the one guy doing interesting work while they spend their day filling out the form they're not allowed to use because they didn't come up with it.

So I ask you this: as the manager, which set of rules would you rather try to run a company under?

Automatic Updates: Frying Pan vs Fire

2009-01-23T19:41:00.002-05:00

Ah, yet another IT related disaster. They make such great blog posts! This time, though, unlike my last one which was a clear cautionary tale about backups worth of a Grimm Brothers tale, the story of the poor boffins at Sheffield Teaching Hospitals Trust has no such simple answer.

Search around with any kind of security related checklist, column, book, blog, or chewing gum wrapper, and one of the constants will be to make sure that all of your security patches are applied. After all, you can be pretty sure that if the hole is sufficiently well known enough for the bad guys to be ready to exploit it. The simplest, most effective, and efficient way to do this is to simply enable whatever automated mechanism your OS has, whether it's up2date on RedHat, updatesd on Fedora, or Automatic Updates on Windows.

In the case of Sheffield, they opted to disable Automatic Updates, and were promptly rewarded with a hospital wide virus outbreak. While the hospital was at least wise enough to design their workflows such that they were able to maintain an acceptable level of patient care, at a minimum they're throwing money out the window on cleanup efforts, including virus removal, and secondary effects such as having to reschedule non-critical procedures.

At first glance - and in just about any other such tale - the moral would be a simple "Leave Automatic Updates on!" But there's a catch. Why were Automatic Updates disabled, you ask? As a matter of fact, until just a few days prior to the outbreak, they were not only enabled, but had a domain policy ensuring that it remained enabled, verified the patches got installed after an internal testing period, and forced a reboot to make the patches take effect, rather than leaving the machine running with the old vulnerable code still in memory.

In exchange for their diligence, Sheffield ended up with a PC deciding to reboot in the middle of a surgery. Can you just imagine being the poor front line helpdesk schmuck who has to explain to a surgeon why his computer decided to reboot all of a sudden? Trying to tell him or her that it's really for the best with a straight face?

Security patches are a critical part of ensuring security of any computer system. Not applying them entails risk; however, given that applying these patches will by definition change behavior somehow, applying them carries its own risk. For far too many computers (mostly, but by no means limited to, Windows), sprinting along on the patch treadmill is the only line of defense against any other machines on the same network. The OS itself is brittle, with nearly any intrusion easily leveraged into total control of the entire machine. Progress is being made, such as UAC on Windows, or SELinux, but that doesn't help with the millions of legacy machines already out there.

Patching these days is a nasty catch 22. With every patch release, you have to take a guess which will be worse - the fallout from applying the patch, or the fallout from not applying the patch. Admins with strict requirements for both availability and security are stuck walking a narrow path, without even any assurance that the patch even exists.

Journalspace Gets Creamed

2009-01-02T23:43:00.004-05:00

If you can't be a good example, then you'll just have to serve as a horrible warning. — Catherine Aird

By now, I'm sure just about everyone will have heard about the disaster that has fallen upon the poor SOBs at journalspace.com. The short story is that the server that hosted all of the data for the blog site got hosed, and lost all of the data. Some of the high points:

The data were stored on a RAID1 array - a pair of mirrored drives
There were no bakcups, or any backup system in place at all
The drives did not fail, but were both completely overwritten on every block
No conclusive root cause was found, but a recently departed sysadmin had already been caught doing "a slash-and-burn" on other systems

So in the end, it looks extremely likely that an incompetent sysadmin set the system up with no meaningful backups, and then progressed to a malicious sysadmin by performing a thorough wipe of the only copy of the system data as he was shown out the door. What a wonderful cornucopia of lessons that can be gleaned from this one example! This is the kind of thing that you expect to see as a hypothetical scenario in security textbooks, not on the front page of Slashdot.

So let's take a quick rundown of lessons learned from our hapless friends.

Backups, backups, backups.: The lack of external backups is what catapulted this from an outage and a headache for the remaining sysadmins into a practically worst case scenario. In short, mirroring is not the same as backing up.
Trust, but verify.: Just because you implicitly trust your sysadmins (otherwise they can't do their jobs) doesn't mean you shouldn't keep an eye on them. Use sudo to log commands, monitor configurations via tools like RANCID, and Puppet or Bcfg2.
Watch the watchers.: Along the same lines, don't let one person exclusively handle any important project. One bad apple working in isolation will have a much, much easier time planting logic bombs than one who has one or two others working side by side.
Don't give them a chance to pull the trigger: Going to fire a sysadmin? Any hint of a possibility of a chance it might get ugly? Be prepared to make sure that any and all rights that admin has are completely gone by the time they know they're getting fired. And please note that most sysadmins will take sudden revocation of their rights as a hint they're getting fired, so the chat with HR should probably happen simultaneously with at least two other trusted admins pulling rights and locking accounts.
Cleanup after their messes.: Dislike a sysadmin enough to get rid of them? Then that same dislike and mistrust should extend to all of the work they've done for you. As soon as they're out the door, it's time to audit what they did. Make sure the work you didn't know they did is up to standards, and make sure to look for backdoors and time bombs.

It's too late for those poor souls at journalspace, but hopefully they'll at least serve to inspire others to fix something.

Priorities

2008-12-26T13:26:00.002-05:00

A few years ago, when I stayed in some hotels, an Internet access was just becoming really standard, there was a per day fee to use it. On the other hand, the hotels also served a decent complimentary continental breakfast for all guests.

Now, when I recently stayed at a few hotels, Internet access was completely free, but now the free breakfasts are gone.

My, how priorities have shifted.

DoD Computer Security Decides to Pull Pants Up

2008-11-22T07:45:00.002-05:00

As I'm sure everyone has heard the Department of Defense has decided to ban all removable media from their computer systems, mostly due to viruses running rampant throughout the (presumably) otherwise secure networks.

Now, people have pointed out that this is a pretty drastic step. After all, there are other ways of handling things that could have theoretically prevented this particular problem without inconveniencing users quite so much. Up to date virus scanners, security policies disabling autorun, restricted privileges on user accounts - all of these things would have helped reduce the ability of such a virus to spread. They should all be considered pretty basic measures in any reasonably high security environment, and it's quite possible that they were at least partially in place.

But there's an elephant in the room that I haven't seen anyone else mention, and would like to point out. Microsoft declared its security Initiative in 2002. In the six years since, we've had two major service packs, and a whole new OS.

So will someone please, please, please tell me why, in this day and age where security breaches make the news weekly, the default behavior for Windows is still to take any newly inserted media and automatically try as hard as possible to run whatever it happens to find on it? It was simply annoying on Windows 95, but it's downright dangerous now.

Come on, Microsoft. I would expect that any operating system that calls itself "Professional" would show a little more restraint than a two year old trying to eat a piece of gum it just peeled off a New York sidewalk. Time for Windows to grow up a little and break this dirty habit.

Money Makes the World Go Round

2008-11-15T21:49:00.002-05:00

It's no big secret that the financial world is going through what can be kindly described as a catastrophic disaster. Stock markets, profit margins, layoffs - all of the meters are currently pointing somewhere between bad and worse.

Likewise, there are plenty of people out there expounding on how we got into this situation, mostly pointing at the various shell games that Wall Street has been playing with mortgages. I don't really have anything to add on the twenty plus year saga of how we've made a bubble big enough to take out neighboring markets when it popped.

Instead, I just have a very simple observation to make. One that the entire financial industry has not simply forgotten, but must continually and actively ignore in order to continue to exist.

To put it bluntly: money has no intrinsic value.

Now, before you just laugh at me, think for a moment about this idea of value, or utility. While the utility of something can vary widely from person to person, and place to place, some things are more universal. For example, no matter who you are, food has some value. Everybody eats. The value of food can be influenced by the skill with which it is prepared, or the ration between its supply and demand, but its base value is directly created by its intrinsic properties. A pound of rice is always a pound of rice, and can always be made into a meal.

So the question, then, is where does the value from money come from? Or to put it a little more viscerally, why is having a pocket full of cash better than nothing but an empty wallet?

The answer, obviously, is because you can buy stuff with it. But what if you took that option away? What good would that money do you in everybody's famous hypothetical scenario, stranded on a desert island? Quite simply, none! A hundred bucks worth of military rations would be a thousand times more valuable than a hundred dollar bill. Moneys value springs purely from our collective agreement to pretend it has value. When you take away the ability to convert money into something else with immediate value, you remove the indirect value of money, revealing its utter lack of intrinsic value.

If you're still not convinced, then ponder this riddle. If the carefully crafted metallic sculpture that we call a "coin" and mass produce at US mints has value, then why doesn't an exact replica that came from someone's basement also have the same value?

What we call the financial trading world, though, is built upon a willful ignorance of this fact. The industry is built upon layer after layer of abstraction, and at each one, the intrinsic value that is abstracted into money is further diluted.

Consider day trading. Throughout the day, any given stock will have some degree of fluctuation. Even if it ends the day at exactly the same price it started at, there will be points where the price is up, even if only a few cents, and other points where it is down. With modern computers, it is possible for even a casual investor at home to have automatic orders rapidly buy and sell the same stock over and over again. Buy the stock at $1.00, sell it at $1.05. Wait for the stock to fall back to $1.00, and do it again. In the days of conducting business over the phone, the cost of the phone calls alone could easily have swamped any profits made. In the days of computers, though, anyone can cheaply run the switch a thousand times a day, with tremendous cumulative effects.

But where did this value behind the money come from? No work was done. No commodity was created. No service was performed, or even promised. Nothing was proffered for this creation or transfer of wealth, not even a kind word. The whole stock market system was intended to be, like currency, an abstract representation of underlying value. A share of stock in a company is a voucher for a fraction of the total intrinsic value of that company.

With the introduction of computers and near instantaneous trading, though, the rules changed. The speed upped the pressure behind this loophole, and money suddenly started gushing through with disregard for the rules. Why bother with all of the tedious research, hoping that the stock will go up a substantial amount, when you can make money off of random noise? As long as the stock doesn't completely tank, you're fine!

Day trading was by no means the first means of exploiting a loophole. But in the last few decades, as regulations have simultaneously become more byzantine and less restrictive, the opportunities for making money by creatively shuffling money around have become more potentially lucrative and tempting. Why go through all the effort of actually creating intrinsic value, when you can not only carefully stack up your bills to make one plus one equal three, but do it a thousand times over?

Yahoo! Music Store

2008-07-25T07:22:00.003-04:00

Okay, so the Yahoo! music store is the latest one to shut down, taking with it the DRM authorization servers required to use the "purchased" music. (Since I've never touched Yahoo! music, I have no idea if the DRM servers are required to play music, every 90 days, when you want to move computers, or what. The relevant bit is that you'll run into a problem sooner or later with the servers gone.)

This has been covered before, so let's just quickly recap:

You can't buy DRM encumbered media, only lease with an option to get screwed.
The option to get screwed is exercised at the discretion of whoever owns the DRM infrastructure.
Do you think that the company actually wants to keep a whole collection of servers up and running for the last three people using purchases from a music store that was discontinued 4 years ago in favor of a new, more profitable one?
Revoking DRM is a brutally effective method of forcing consumers to leave an old platform, in hopes they'll all sign up for its successor. The fact that customers were happy with the old platform isn't perceived as a downside; it's the reason why the company is doing it in the first place.
Strong DRM means that companies can use technical means to enforce policies, regardless of their legality. Existing code doesn't automatically update to reflect new court rulings, and your only appeal is with the companies helpdesk.

All those of you who have been writing about the dangers of DRM may now proceed to jump up and down while shouting "I told you so!"

Lies, Damned Lies, and Marketing: A Plea to Netflix

2008-06-21T21:56:00.003-04:00

I work in IT. Not surprisingly, this means I get to do a fair amount of support for broken computers. In my case, it's mostly for a handful servers from a particular well-known vendor. Since we pay a premium for the top-level support, I tend to be pretty satisfied when calling in for failed components. The calls basically tend to consist of "What's broken?", "Let's run a quick diagnostic to make sure", "Do you want a technician or just parts?", and "Do you want it there tomorrow, or this afternoon?".

Then one day, I had to make a call in for a desktop. Same vendor, still had one of the higher level support contracts, and still quite obviously a hardware failure.

Unfortunately, this meant that instead of getting routed to a bunch of IT-savvy techs determined to keep my downtime to a minimum, I got to deal with the general home user support group.

Now, I do enough end user support to be able to sympathize with quite a bit of what these guys go through. I really don't mind them asking me really basic questions like "Is the computer on fire?"; I've had users who would neglect to mention this when asking why we turned off their Internet. I completely understand them strongly wanting to get an error code back before they'd start shipping replacement parts; I wouldn't be surprised if they've had users who didn't understand that you need to put a blank CD in before they can make a mix CD of their pirated MP3s. I won't pretend to like these things, but I understand they're necessary and don't hold it against the poor people at the other end of the line.

No, what bugs the hell out of me when they keep claiming they're "sorry". Yes, that's right, every time I talk to a new person, and every time I mention something that's a problem, they rattle off, all in one quick, unconvincing, insincere, scripted breath, "Oh-I'm-terribly-sorry-sir-I-feel-really-bad-about-that-I-hope-that-we-can-fix-the-problem-and-I'm-sorry-for-the-inconvenience" .

Oh, really? You feel personally bad about every annoying user with a broken coffee cup holder who can't tell you if it's plugged in because the power's out? Bull. After the fourth or fifth time, I'm actually far more annoyed than if you just said "Okay" and punted me off to the tech in line, because it's quite obvious that you're lying to me. I'm paying the extra support money for tech support on the product. If I wanted someone to talk to and empathize with me, I'll to find a qualified therapist and talk about my childhood, thank you very much.

Where was I going with this? Oh yes, Netflix.

As I'm sure that anyone who has a Netflix account, reads techie news sites, has an Internet connection, or uses electricity has heard by now, Netflix is removing the profiles feature, which lets you split up a single account into separate queues and preferences. This lets multiple people share a single account, rather than each buying their own - perfect for households with more than one person.

Now, the canceling of this feature is bad enough. My wife and I use this, and let me tell you, it's a lot easier than trying to come up with ratings that accommodate chick flicks, romantic comedies, sci-fi, and anime. I mean, seriously, how many people really like all of those categories?

As if that weren't bad enough, though, Netflix had to take it one more step. They decided to just give all their profile users a father-knows-best pat on the head, and tell 'em "It's for you own good." Like the tech who personally fells the pain of each and every of the thousand customers per day, Netflix has spun a falsehood that is insultingly transparent:

As a Netflix product manager I'm tasked with the wonderful job of helping members find movies they'll love. But today my job is more challenging as we've decided to terminate the profiles feature on September 1. Please know that the motivation is solely driven by keeping our service as simple and as easy to use as possible. Too many members found the feature difficult to understand and cumbersome, having to consistently log in and out of the website.

Let me get this straight. You have a feature that, while perhaps not wildly popular, is strongly loved by those who do use it. "Some" people allegedly find it "confusing" (we'll assume for the moment that Netflix has legitimate data to back this claim up), so rather than, oh, I don't know, fixing the problem, you just decide to nuke it completely. How does that "help" users?

Now, where Dad could give 5 stars to Goldfinger, Mom could give 5 stars to Pretty Woman, and Junior could give 5 stars to Shrek, Netflix will be trying to analyze a single person that would give 5 stars to all three movies. I can only imagine the bizarre recommendations for such split personality victim! How does that "help" users?

Before, each member of the household would have their own queue, and would get their own next movie for each one sent back. Now they'll have to carefully shuffle the queue each time one goes back to make sure that the right next movie goes back, or else Junior sending the cartoon he just watched back will land Julia Robert's latest movie in the mailbox. How does that "help" users?

If you're going to pull out some backend code that implements this feature, fine - but I doubt there's a software engineer on the planet who thinks it's a good idea to pull a feature away before you have something more compelling to convince your customers to give you money.

If maintaining the feature is taking up too much time, or is getting you stuck in some expensive patent war, then tell us you can't afford the feature and we'll probably understand and get over it.

But please, please, please - don't just rip the feature out of our hands and tell us it's for our own good. It's a blatant lie of the worst kind - a marketing lie - and once your customers think that you're lying to them, they're quite liable to take their money off to one of your competitors in a hurry.

You can trust me on that.

What Time Is It Anyway?

2008-06-04T20:49:00.002-04:00

I can't believe that I feel I need to write this post. Really. But, I do, so here it goes.

Start by asking yourself a question: what time is it? A simple enough question, with a simple enough answer.

Now pretend for a moment that you had 100 people scattered around the globe on a conference call. Now ask them all, at precisely same moment, what time it is.

(Hands down, all you physics majors out there. We're ignoring relativity, since this is all make believe anyway, so everyone agrees it happens at the same instant in time.)

Now all of a sudden the answer to your question isn't quite so straightforward anymore, is it? You have to worry about dealing with multiple timezones, the international date line, and daylight savings time. The only way to deal with this is to use dates that explicitly include the timezone. Trying to deal with dates and times missing timezones is like trying to use latitudes without longitudes, or an email address without a domain. As soon as the scope expands beyond a very tiny size, it breaks down quickly.

Now, the fact that just about any standard formatted timestamp includes this information seems like it would make this pretty obvious. Email, HTTP, filesystems - they all either include a timestamp, or are universally defined as relative to a fixed timezone that you can easily base off of.

So will someone tell me why, in this day and age, the derby database chose to define timestamp columns that are missing the timezone? You wouldn't forget to make numerical types with floating point support, would you? Or strings that didn't support storing lower case? Or... well, you get the general idea.

So come on, guys. It's a big world. Databases are all about sharing information, and these days even a modest open source project can easily be sharing between half a dozen timezones across three continents. At this point, you're just making yourselves look silly.

Successful Failures and Faulty Successes

2008-05-16T22:25:00.002-04:00

Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin. — John von Neumann

So unless your job has nothing to do with IT, or you've been living under a rock somewhere out of Blackberry range, you've no doubt heard about the utterly terrifying Debian OpenSSL vulnerability which left all those vast 4096 bit private keys into effectively 15 bit keys. (For those of you who don't speak crypto, this means that the bad guys can guess your private key in about 32 thousand guesses - pretty trivial for any modern computer.

The problem, ironically enough, came about when Debian developers attempted to fix some compiler warnings about a function in OpenSSL that was using uninitialized memory. While normally a horrible idea, OpenSSL was using this as an additional source of entropy, to make the private keys it generated more random. Unfortunately, the actual result was to remove nearly all entropy, leaving only 15 bits behind from the PID.

I think it's pretty safe to say that this was a catastrophic failure. Now Debian has done an admirable job of releasing a fix to the tool that generates the weak certificates (though you still have to go back and replace already existing ones), the question remains - how in the world did OpenSSL exist in this blatant failure mode, completely undetected, for two years?

This problem is a beautiful illustration of part of why cryptography, and security in general, is so devilishly difficult to get right. In normal software testing, successes are successes, and they're good, and failures are failures, and they're bad. Simple enough, right?

When you're testing security, though, things are different. In security, you also have to make sure test for what I like to call successful failures, and faulty successes.

Take a firewall system, such as iptables or pfw, for example. Without one, if a client attempts to make a TCP connection, it expects to succeed. If the connection succeeds, the test succeeds; if it fails, the test fails. Once the firewall is in place and configured to block that connection, though, that connection damn well better fail! That counts as a successful failure - a case where you succeeded in selectively making something like a TCP connection fail in a case where it would be undesirable. Likewise, when a file is encrypted, unauthorized attempts to read it (or at least, extract meaningful data from it) by anyone without the appropriate key is expected to fail.

Likewise, you also have to test for the inverse case. Back to our firewall example, let's say that the admin carelessly mistyped the mask, leaving our service unprotected from ranges that we don't want to have access. Connection attempts will all of a sudden start succeeding where we don't want them to. We now have a faulty success. Even worse, we won't notice unless we happen to test from the tiny sliver of IP addresses that were erroneously granted access. Back in the crypto realm, this is what happened to OpenSSL. It failed to prevent success, where success means a request that should have been prevented was not blocked.

These additional twists on defining success and failures help to make testing security software and configurations devilishly difficult. The OpenSSL bug didn't cause any visible changes in the test results. Everything still compiled; the output was still valid; data was encrypted and decrypted properly; no regression tests failed.

Hopefully someone someday will figure out a more reliable way to test this kind of code than the current method of having people who've forgotten more about math than most of us ever even heard of stare at it until drops of blood appear on their foreheads. Until then, we'll just have to be ready to roll out patches, scramble passwords, and revoke certificates when the next inevitable vulnerability or system compromise happens.

If It Weren't For My Horse...

2008-04-26T21:30:00.002-04:00

Lewis Black has this great bit he does, where he describes being out and about one day, and hearing a stray phrase from a nearby woman grab his attention like flypaper grabs a mosquito:

If it weren’t for my horse, I wouldn’t have spent that year in college.

Well, today, while doing a little research on network monitoring packages to see what's new out there, I ended up taking a peek at how Big Brother is doing these days. On their page, in the License section, lay another equally enigmatic phrase:

Big Brother is distributed under our Better than Free license. Clause 2 from that license determines whether you need to buy a Commercial license.

I'm sorry - what? It's better than free, but I have to worry about buying (as in, non-free) a license in some conditions. I really have no idea of how to reconcile those two sentences together without application of drugs. I recommend you don't try too hard, either, or you risk at minimum a migraine.

Let's just look at the first half. Better than free? So unless they're paying me to use their software, then not only is it free, but they throw in something else, too, like cake. Now, it is true that you can download and start using the software at no charge, but there are some strings attached. To be fair, let's compare their version of free to that famous poster child of free software, the GPL.

	Better than Free	GPL
Duration	30 days, and then you have to buy a commercial license.	Perpetual, until the terms are violated.
Source Code	Only if they feel like giving it to you.	If you obtain a copy of a GPL binary, you are entitled to the source code that generated it.
Derivative Works	Standard commercial no modifications, no reverse engineering, "hands off!" clause.	Ensuring that each user has full rights to create, modify, and distribute derivative works is the entire purpose of the GPL.
Termination	License is terminated on breach, or also with a 30 day notification on the web site.	Only on breach of terms.

Uh-huh. So you've only got 30 days, you're not guaranteed the right to delve into and modify the software, you can't give it or any changes you make to anyone else, and they can change the terms on you whenever they like. And no cake.

If that's "better", then I'll stick with plain old GPL and boring old "just free".

Microsoft and "Vista Ready"

2008-03-30T21:27:00.003-04:00

There have already been oodles of articles out there talking about the fiasco that is the "Vista Ready" vs "Vista Capable" fiasco. Simply put, there's evidence (including internal emails) that Microsoft lowered the standards required to meet Vista Capable. This resulted in machines with the Capable sticker barely being able to run Vista at all, let along certain advanced features, most noticeably the Aero interface, and much user annoyance and confusion.

This gives me a good opportunity to point out a common misconception about Microsoft. One that most people have, and which leads to a great deal of confusion about why Microsoft does what it does.

The confusion, simply stated, is that people think Microsoft makes software. It doesn't.

Now, now, I know what you're all thinking. What about Windows? and Office? and SQL Server, and MS Money, and all of the other Microsoft products lining the shelves at Best Buy? Okay, so Microsoft also makes advertisements. So is it an ad company? How about a payroll company, since it pays its employees?

My point is, those boxes of bits are, when you really get right down to it, in the same category as ads and pay stubs - nothing more than a means to an end. And that end is, of course, money. (The green paper stuff, that is, not the aforementioned program.) In other words, Microsoft doesn't make software; it makes money through its expertise at making software.

So why should you as a random consumer of Microsoft care? Because each and every decision that makes will have an implied footnote, a hidden subtext that reads like a banner out of Office Space: "Is this good for the company?" Each potential action will be weighted based on how much money it makes, or loses.

Sure, there will be plenty of consideration about what's good for customers, but let's face it - if Microsoft went out of its way to screw over consumers, it would have a difficult time convincing those same customers to give it money. Beyond that, Microsoft is a big company with a lot of people in it, and no doubt quite a few of them really do try to do right by their customers.

But in the end, like every other publicly traded corporation, Microsoft has to answer to it's shareholders. And each and every decision is evaluated, not on how popular it is, or on technical merit, or if it follows standards, or even ethics - but what it does for the bottom line.

Energy

2008-03-14T22:21:00.003-04:00

Looking back at the small bits of noise I've added to that gossip factory we call the Internet, I can see that I've only really bothered to talk about tech stuff, and somewhat esoteric bits at that. So, I've decided to change bandwagons mid-stream (to quote Eric Raymond, I like my metaphors shaken, not stirred) and talk a little bit about environmental issues.

More specifically, I'd like to talk about what is, at least in the long run, the single most important issue: energy.

Everything we do, from research, to cooking, to transportation, to taking a walk around the room requires energy. As our societies become larger and more technologically sophisticated, we require our energy be delivered both in greater total quantities, and in higher density packaging. A millennium ago livestock, crops, and farm animals were enough; a century ago modest amounts of petroleum products and electricity sufficed; now it's all we can do to keep energy production matching pace with ever increasing demand.

One of hot topics that obviously flows from this discussion is where we should be squeezing all of that energy from. So what choices do we have?

Oil
Coal
Wind
Water (hydroelectric dams)
Nuclear
Hydrogen
Solar

Now all of these various sources have advantages and disadvantages. Water and wind are clean and renewable, but hydrocarbons have far higher energy densities. But what happens when we step back and take a longer term view? And when I say "longer", I'm not talking about the "not one, but two quarters ahead!" view that seems to dominate most companies and public debates, but a true, seven generations out view.

Let's start with hydrogen. Hydrogen does occur naturally, but not in huge quantities. The big challenge with hydrogen isn't using it, it's creating it. Hydrogen is not an energy source, it's an energy transport. It's one solution to a huge piece of the problem, but it's not a source. Since we're only talking about energy sources, let's vote it off the island and see who's left.

Oil
Coal
Wind
Water (hydroelectric dams)
Nuclear
Solar

Let's go pick on nuclear power next. Now, if handled right, nuclear could potentially offer up quite a bit of power for quite some time. A string of properly set up breeder reactors can pass material down from one to the next, extracting additional energy at each stage. Even assuming that somehow, someone could muster up the political and financial capitol to make it happen, there is still only a finite amount of glowing rocks laying around to throw in the reactors. Once those pockets are used up, it's done, so let's throw it off our list too.

Oil
Coal
Wind
Water (hydroelectric dams)
Solar

Now let's go all "green" here and go after the Big Bad Carbon Producers: oil and coal. When you get right down to it, they're nothing more than dinosaur and plant extract. And where did the stored energy that we pour into our gas tanks every day come from? Why, the sun, of course, as any middle school level science textbook could show you with one of those near little diagrams with arrows pointing in circles, and a picture of the sun off to one side pumping energy into the picture of plants. So since they're really just pockets of condensed sunshine (energy-wise) let's consolidate the list further by taking those fossil fuels off.

Wind
Water (hydroelectric dams)
Solar

That list is getting might short, isn't it? But hey! At least what's left are all clean, renewable resources, right? But wait a minute. Wind and water are great, but what makes them move? What drives them? Or, as an actor would say, what's their motivation? Well, for wind, it's heating and cooling caused by - c'mon, guess - that's right! It's the sun again.

Water? We get energy out of water falling downhill, but something has to push that water uphill in the first place to store up that kinetic energy. More specifically, on the scale we're talking about, something has to evaporate it so it can condense into rain that lands at a higher altitude than it evaporated from. Which implies heating, which... yes, yes, it's the sun again. So, now our continually shortened list.

Solar

And there you have it. All of the various energy sources we argue over, discriminate between, and tweak to squeeze more out of, are either solar, concentrations of stored solar, or finite resources doomed to run out, short of mining other planets in the solar system.

Now, I'm quite aware that solar power, as it currently exists, has problems. It's output is heavily influenced by weather conditions, efficiencies are still relatively low (especially when compared against the energy in a gallon of gasoline), and it's only in the last decade or so that a solar cell could be expected to produce more energy in its entire lifetime than it took to manufacture it. It's output is also limited to electrical or simple raw heat, and again, we don't have any kind of batteries that can compare to the energy transportation and storage of petroleum products.

(Some companies are looking at ways of making hydrogen more easily used by binding it up with other elements to make it more stable, such as carbon. Which gets you volatile hydrocarbons - aka, petroleum products!)

So in the end, we really don't have much of a choice. We can take advantage of little caches of energy, stored in plutonium or crude oil. We can pick the path we take to get to solar energy, whether it's through an intermediate, such as manufacturing hydrocarbons, or direct, such as boiling water or solar cells. But in the end, the sun is really the only source of energy that's going to hang around long enough for us to pretend that it's going to last forever.

(At this point, the pedants out there will point out that eventually, the sun will let us down by expanding out and destroying the earth, rather than providing us with a gentle stream of life giving radiation. I, for one, fervently hope that the human race is around long enough to have to worry about this.)

(At Least) One Of These Things Is Not Like The Others...

2008-03-09T22:37:00.002-04:00

I just recently read Freaknomics, an interesting book on economics with interesting ideas put forth by a guy who economists claim is more of a sociologist, while sociologists claim he's an economist. The book throws out and makes a stab at answering bizarre questions about seemingly unrelated topics, like "how are teachers like sumo wrestlers?"

So, in the same spirit, I'll start off this post with the same kind of question: What do Freakonomics, Netflix, and my last hospital visit all have in common?

Now, unless you've been stalking me, I really wouldn't expect you to guess how my last hospital visit comes into play, so I'll give you a hint. My hospital is well into the process of converting from thick, massive folders of paper records, over to digital records with a PC in every exam room. While the nurse was going through medical records and scheduling procedures, she apologized for taking so long, and complained that the software layout made no sense for her field, and obviously wasn't designed by someone who knew it.

Figured it out yet? One more hint - the challenge Netflix is currently running to find a better movie recommendation looks like it just might be won, not by some MIT team of CS majors, but by a phsycologist!

Every field, be it computer science, psychology, or medicine, has boundaries. Various ideas and concepts get sorted into the right field based on those lines. An algorithm for traversing a graph? CS. A study of the effects of a new drug? Medicine, further narrowed down by specialty.

There's a problem with those lines, though. No one told the problems we're trying to solve about them!

In the first two examples, exceptional results were found by doing work that happily straddled those lines. In the Netflix example, without psychology, he likely never would have had the insight required, and without CS, he never would have been able to actually implement it.

Likewise, in the hospital example, the fact that the software engineers who created the software weren't intimately familiar with the actual job created a system that didn't match the workflow. Instead of helping the nurses, they end up stumbling around looking for options and fighting the system.

We've all heard the joke about a bunch of blind men who stumble across an elephant, and try to figure out what it is by feeling it: "It's a snake!" "No, it's a tree trunk!" "No, it's a wall!" Well, guess what? We're all a little guilty of it now and then. It's only human to try to look for solutions within the one or two fields we're comfortable in.

So what should we do about it? Recognize that the sum of human knowledge may be sorted by the Dewey Decimal System, but it is not defined by it. Read outside of your field, and see what kind of tricks those guys who went to college in a different building may have up their sleeve. Working in a college myself, I can tell you that it's not too uncommon for someone who's sacrificed any pretense of breadth for incredible depth in one field to struggle with a problem solved decades ago in an apparently unrelated field.

And in the end, ask yourself which one you'd rather be - the guy winning a Netflix prize by fusing together two superficially unrelated fields, or the software engineer who gets yelled at because he used the wrong kind of chicken guts when divining a nurses workflow?

Windows World (Slowly) Learning From Unix History

2008-02-29T19:59:00.003-05:00

The excellent Coding Horror blog has a short article up about one way of categorizing software: UsWare vs. ThemWare. The idea is simple enough. ThemWare is software that's only used by "Them" - ie, none of the users are also developers of it. UsWare is software that is used by the developers as well as others.

Jeff comes to the conclusion - which I happen to agree with 100% - that creating software as UsWare will, all other things being equal, lead to vastly higher quality than software created as ThemWare. To help this process along, he encourages his software developer readers to work to gain the user perspective, to eat their own dogfood. This is certainly a good idea.

But as I thought about it a bit, I realized that this is only half the picture. The focus here is to give the programmers more of a user perspective so they meet user needs better. But what if things could go the other way? What if users could get more of a programmer perspective, so they actually could communicate their needs effectively? And maybe, in the case of users who have some programming experience, be allowed to help out and contribute bits of code that demonstrate what they want with far more precision than any prose description. Either way, the end result is to break down barriers, and blur the line between developer and user.

Oh, wait. There's a name for that already - open source.

That phenomenon called open source software hasn't really caught on too strongly in the Windows world, in no small part because Microsoft does everything in its power to keep all of its source code under heavy lock and key. With how much Microsoft depends on license keys to enforce paying for software, there really isn't much of an alternative for them. Even more important, I believe, is the fact that Microsoft began from square zero by selling software to non-programmers. The people using those original DOS systems didn't want computers for their own sake, they just wanted them to run their business.

In the Unix world, however, things began completely different. While Microsoft was busy trying to sell computers to people who didn't want to know anything more about them than they had to, Unix was a programmers playground. Researchers used Unix, and often had to create their own applications, and were able to with compilers being commonplace on Unix systems. Unix was an environment created by programmers for programmers, and the result is that once you begin to feel a little comfortable as a Unix user, the bar to becoming a Unix programmer is fairly low.

As the Free Software and OSS movements had propelled Linux systems as the successor to the Unix heritage, this trend has only become more pronounced. These days, a typical Linux system will have two or three programmer-friendly editors, an IDE, compilers for C, C++, and possibly Fortran, lisp (if you count Emacs), java (Sun, or an open source alternative), and a handful of powerful scripting languages such as Perl, Python, and Ruby.

And that's just the typical stuff! For the Linux user truly interested in becoming a programmer, there are debuggers, Ada, Smalltalk, Rexx, Haskell, and countless other languages and development aids just a Freshmeat search away. With all those tools just waiting to be picked up, each and every open source user is a potential contributor, of anything from a bug fix, to feature enhancement, to documentation, all the way up to becoming a full fledged maintainer.

Jeff is absolutely right that programmers who learn what it's like to be users will end up producing higher quality software. But as long as you freeze out your users from becoming contributors, you're throwing away valuable resources that you often couldn't buy if you wanted to. And that's why Linux will always have an edge over Windows, no matter how many animations they add to Aero.

Plan For Failure

2008-02-15T21:26:00.004-05:00

Vista "enhancements" include removing the ability to do repair installs. Screw Windows up a little too badly, and your only option is to reformat and reinstall.

Rim has an undisclosed problem with servers off in Canada, and suddenly every Blackberry everywhere goes offline.

Congress starts ramping up surveillance and blanket data retention, but never seems to worry about the fact that those same tools are equally useful for criminals.

What do these three disparate events all have in common? Simple. All of the design was built around what happens when things go right, not wrong. All three cases display a horrific lack of preemptive failure analysis.

Failure analysis is something that is taught to more established professions, such as mechanical or civil engineering. In these professions, where a screw up frequently can mean people die, worrying about when happens when - not if - something breaks is beaten into students until they think about it the way a deep sea diver thinks about his air supply.

When a civil engineer designs a bridge, he can easily end up putting in thousands of pieces. Some pieces, when they fail, are rather unimportant. If the dedication plaque rusts or falls off, a donor may be upset, but the operation of the bridge isn't compromised. On the other hand, if a rivet or weld holding a support in place cracks, then the engineer who signed off on the design is going to be very interested in what will happen. Will the bridge hold for a year? Six months? A day?

Every part has an MTBF. Just as important as knowing when that part is likely to fail is every bit as important as knowing what will happen when it does fail. Often times, an early analysis can find hidden critical dependencies that can be fixed or mitigated with simple design changes.

Take the Vista removal of recovery restores. Strictly speaking, removing this feature didn't add any failure modes. Unlike a new driver or filesystem, it didn't add any new ways for an existing Windows system to break. What it does, is ensure that once a failure beyond a threshold does happen, the impact will go from being recoverable, to being a death sentence for that copy of Windows. Without adding any new failure modes, the number of critical failures just went up.

Now if you ask the people who put these systems together, I highly doubt that they intended for these systems to fail. This seems obvious... But it's also the problem.

Every system out there will have a failure sooner or later. Let's be fair to Microsoft, by giving them a plus side. All Blackberries have their data go through Rim servers, despite having a perfectly good data connection from the cell provider. This adds a wonderful single point of failure. By contrast, Microsoft based smart phones don't need any such assistance. They're perfectly capable of talking on their own, without an extra translator.

Microsoft could take their entire infrastructure offline, and the phones wouldn't care. By keeping their own servers out of the data path, they've reduced the number of failure modes of Windows Mobile phones out in the wild.

If we programmers and IT guys want to be taken seriously, we absolutely have to start planning for failure. Throwing redundant servers at problems reduces the likelihood of failure, but doesn't reduce it to zero. RAID protects you against a single hard drive failure, but not multiples.

We have to start asking ourselves, with each and every component we build or install, what will happen when this system breaks? That's how you notice things like a pair of high end servers both plugged into the same $4.95 ValuePak power strip. That's how you put in exception handlers that, when that exception that can't possibly happen happens, at least ensure the program goes down gracefully instead of exploding with a corrupted database.

That's how we can start building systems where a single, simple stupid failure doesn't turn into a headline generating, career limiting fiasco. Then maybe those civil and ME guys will stop snickering whenever one of us calls himself a software "engineer".

WTF is Google Thinking?

2008-02-07T20:57:00.000-05:00

Google. The projects they do, the reactions they provoke, even the cooking in the cafeteria - whatever they do, almost always ends up being big. Unfortunately, with their latest "It seemed like a good idea at the time!" they're most likely about to piss of even more IT staff than when Google Desktop started copying files onto Google servers indiscriminately.

The description from the press release sounds innocuous enough:

Google (NASDAQ: GOOG) today announced Google Apps Team Edition as the simplest and fastest way for groups of employees and students to collaborate within an organization using Google Apps.

But then they go on:

Once users verify their business or school email address, they can instantly share documents and calendars securely without burdening IT for support.

ARS Technica had it right when they described this as Google trying to "sneak Team Edition suite past IT help desk". To those IT help desks Google is referring to, this is roughly like working to bring new an exciting drugs to market without burdening the FDA, or opening a new restaurant without burdening those poor health inspectors.

The problem is, Google is offering to host some set of end user data, but those end users quite simply lack the ability to evaluate whether or not Google is a suitable custodian of that data. Random end users shouldn't be expected to make those kinds of evaluations on their own. After all, why should an accountant worry about going over technical details of colocation and outsourcing details, such as key escrow management, encryption, etc, when you already have an IT department to worry about them?

In any decent sized company, this is how things are supposed to work. The business side of the house sets the priorities, then passes the goals and requirements off to the IT of the house, who picks the best solution on suitability and technical merit. Management sets the why and what, IT decides the how.

Google, on the other hand, appears to be trying to take that away. Now, I'll be the first to say that expanding the online Google tool suite is great. And adding in collaboration features is a pretty obvious next step.

But damnit all, Google has a responsibility to make sure this loaded gun is at least pointed in the right direction! If you want to sell liquor, fine - but that doesn't mean you should open up shop across the street from a high school. The last story that I heard of where users decided to go off and create a working solution on their own, the end results included an SSL free commerce web site and credit card numbers were tossed around in plain text email to be typed in. Collaboration definitely sounds like a powerful tool in the right hands, but IT still has to have a prominent role in picking which tool to use and how to use it.

Now I'm sure that the good folks at Google never intended to have sensitive data, like business plans or credit card numbers, passed around. The problem is, to an ordinary user, only moderately technically literate, the only difference between storing that top secret business plan on a secured server and Google docs is which bookmark they click on.

In a a managed corporate IT environment, the IT and business sides of the house have a close working relationship. The IT side understands enough of the business side to create a working system. At Boeing, the IT staff understand that plans for new airplanes are highly sensitive, and so can set up servers and encryption to protect it, and train users in how to use it to protect data. With Google, however, you get what they offer, and that's it. If Google apps doesn't meet your needs, you either end up with a hole that Google apps can't fill, or even worse, leaving data inadequately protected.

So the next time that someone who has no chance of understanding the implications of the fine print in the acceptable use policy goes off and leaks the company crown jewels by clicking the wrong checkbox in a Google app, will Google accept any of the blame? Or even more importantly, any of the responsibility of cleaning up the resulting mess? Tracing the extent of data leaks? Buying credit protection for identity theft victims?

Somehow I suspect that Google won't mind burdening the IT help desk with that half of the job.

CS Majors Need Not Apply

2008-01-13T09:12:00.000-05:00

Usually, I like Coding Horror. I just read a post, though, where he argues that CS majors should be taught more software engineering. He quotes CS students who have never been formally exposed in their entire undergraduate program to things that the professional field lives and dies by, such as deployment management and revision control. But then, he goes one step too far, right off the cliff:

If we aren't teaching fundamental software engineering skills like deployment and source control in college today, we're teaching computer science the wrong way.

Sorry Jeff, but I'm going to have to call you on this one. If you're teaching fundamental software engineering skills like deployment and source control, we're not teaching computer science at all - we're teaching software engineering!

Think about cars, and the people who design them. Typically, they went through a Mechanical Engineering degree. While this means that they did get a solid grounding in some underlying physics, such as heat transfer, stress transfer, and material analysis, the focus is on how to apply those areas to the real world. Complicated, precise physics formulas are replaced for approximations and tables designed to quickly and easily give an answer that may be less accurate, but errs on the side of safety.

On the other end are the actual guys who do the real hard-core science - physics. Mostly done on chalkboards and computers, these guys only delve into the real world to gather data or test out a hypothesis. There are quite a few good physicists out there who could explain to you in great detail how and why a tire has a particular amount of traction on asphalt, but couldn't actually change a tire if their live depended on it.

The important point here is that even though ME and physics of the properties of physical things, there is still a distinction between the abstract, research oriented side, and the dirty, messy, practical side. This is a distinction which most of the computer "science" majors out there seem to pretend doesn't exist.

Most of true computer science doesn't even have anything to do with computers. Take Big O notation. In computer science, if an algorithm takes an hour, a day, or a mon, as long as they scale linearly as the size of the input goes up, they're all O(n). Try to argue to a customer that they should be considered equal in any way, though, is likely to make for a short career as a programmer.

The harsh reality is that most companies advertising for computer science majors don't really want computer science majors. Sure, they want someone with a good knowledge of algorithms, but - as Jeff pointed out - the ability to use version control is at least as important. Grungy skills, such as creating crash dumps that allow you to get good diagnostics info about customer problems without having to ship them custom builds, while utterly boring from a pure CS standpoint, are worth their weight in gold outside of academia.

This isn't to say that software engineers shouldn't have a grounding in CS theory. There's going to be a lot of overlap. The difference is one of focus. Once we accept that there are really two majors trying to fit into one curriculum in most schools, we can start the process of trying to make a one size fits all, and stop trying to turn out physics majors that we expect to be able to design a camshaft.

Typing Puppet Strings Onto Your Servers

2008-01-06T19:53:00.000-05:00

Just like a good Perl programmer, a system administrator should strive for a certain degree of laziness.

Now, this is not the kind of laziness that leads one to think "Eh, I'm not going to bother installing that update." No, this is the form of efficient laziness that says "I could download and install that update, but there's got to be a way to get it done automatically without wasting my time." These are the kind of people who have libraries of shell scripts and packed cron jobs.

Now, those libraries of shell scripts are great, but they can be an awful lot of work to write and maintain. Not very lazy at all! So, rather than going that route, I've been working with (and on) a package called Puppet.

Puppet is a client/server package written in Ruby. Essentially, you configure the server with the configuration settings you want all of your machines to look like. The clients get pointed at the server, pull all of their settings down, and make them happen.

It's got a decent library of native types (such as packages, files, users, etc) right out of the box. If you need something that's not covered, it's fairly straightforward to write your own custom code (assuming you know Ruby) that allows you to extend what kinds of files and setting Puppet is able to directly manage. Thanks to some good helper libraries, I was able to whip up a custom module that allows me to manage entries in /etc/sysctl.conf is only 59 lines of code!

Some of the cooler features of Puppet:

All communication is XML-RPC based, making it easier to write custom programs that communicate with Puppet
Collections of facts about client systems (OS, OS version, etc) are reported back to the server and can be stored in a database
Defines and Exec allow you to create complex configurations without writing any Ruby code
ERb templating system (the same one of Ruby on Rails fame) allows you to generate complex configuration files with per-host settings

Ask anyone who manages big numbers of systems - hundreds, or thousands - and they'll tell you that the ability to automatically manage systems from provision to decommission without manual intervention is absolutely essential. Whether it's built in, like GPO in Windows, or an add-on package like Puppet, trying to manage any more than one or two systems without this kind of help is just making more work for yourself.

And that's not very lazy at all.

Coding and Complexity

2007-12-19T15:53:00.000-05:00

First off, let me just make a quick confession - while my undergraduate degree was stamped with "Computer Science" as my major, I don't really consider myself to primarily be a programmer. Sure, I do actually spend a good number of my days mucking around with writing code (usually Perl, occasionally Ruby), but my job is really IT support, specifically networking. I deal with switches, routers, wireless, VPN, and a handful of Linux servers supporting the network with DNS, DHCP, etc. The code writing that I do is almost exclusively to support everything else, such as working on a host registration system or device monitoring scripts. The software I write is to directly address a need, rather than to be sold to address someone else's need.

That said, when I just read Steve Yegge's latest rant, Code's Worst Enemy, it struck a chord with me.

I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add, that the worst thing that can happen to a code base is size.

Now, as someone who does not consider himself a spectacular coder by any means, I would certainly feel quite daunted by tackling a 500k line codebase by myself. On the other hand, as a professional coder, Stevey ought to be able to casually fling around great swaths of code, using advanced software repositories and indexing tools, right? But no - he feels that, all other things being equal, less is more.

One feature of large code bases that I think he gave short shrift to was the idea of complexity. He talks a little bit about how complexity certainly makes a given code base harder to work on, and also that some of the automatic tools, such as refactoring, that try to deal with it just make the problem worse by bloating the code base even more.

This is something significant in his argument, I think. In this example, we have two code bases, before and after being run through the automatic refactoring tool. The initial state has a given level of functionality, size, and (for lack of a better word), "goodness". The final state greater size, and therefore less goodness, but identical functionality! This mirrors his stated goal of taking his existing game, and rewriting it with identical functionality but less than half the lines of code.

I think the explanation boils down to this: we can only fit so much in our brains at a time. Great programmers can mentally swap in more of the big picture at once, but everyone has their limit. This limit is why we decompose programs down into manageable subroutines, each of which can be understood (at least partially) in isolation from the rest. It is why we hide massive chunks of functionality behind a handful of calls into a library. The smaller chunk size we're working on, the more likely we are to be able to fully understand it and not screw up.

From here, the trick to making sense of Stevey's size argument is realizing that there are two completely different kinds of complexity at play here. If you're writing code to do, say, an FFT, you've got to know the math behind it and how it works. That's a fair bit of complexity that you've got to hold in your heard, and it's going to remain constant regardless of whether you're developing in Java, Ruby, C++, Assembly or BF.

This invariant portion of the complexity is what I call inherent complexity. (Please don't tell me if that term isn't original; I know if probably isn't, but I like to pretend.) It's the piece that you can't get away from, since it's what defines the actual problem you're trying to get that hunk of copper and silicon to solve for you. It's the tax code embodied in Quicken, the rules of mathematics in Mathematica, the graph theory in Garmin and TomTom. Remove the inherent complexity from a problem, and all you've got left is a very complex, boring video game with executables instead of high scores and compiler errors instead of health damage.

If the inherent complexity were all there was to it, then knowledge of the problem domain would be all that's required. You wouldn't need a programmer to write Mathematica, just a mathematician to sit down and tell the computer everything she knows about math. Easy, right?

Sadly (or fortunately, if you make a living as a programmer) this is not the case. The person coding has to know extra details that are outside of the problem domain, like the fact that the number 0.1 cannot be represented with absolute precision in a floating point number. Or that if you accidentally tell a computer to loop forever, it will do so. Or that each of these three different sort routines will produce the same final product, but the memory and time requirements can vary by an order of magnitude or more - and not always in the same order, depending on the data set. Not to mention nitty language details, like dealing with pointers in C or "bless" in Perl.

All of these other layers upon layer of crap that gets wrapped around the real problem is just extraneous complexity. I mean, let's be honest - learning objected oriented design or unit testing may help you write code faster and with fewer bugs, but won't help with bullet point one of the design requirements for an ERP (or online order system, or factory automation, or... ). It's all work that is, in the end, unquestionably important to creating a finished product, but any time spent working on that extraneous complexity is time not spent on the inherit complexity.

Or, to put it more bluntly, any time you spend appeasing your programming environment is time that you're not spending on solving the actual problem.

Based on this, the best development languages are ones that are fairly thin, succinct, and in general just get the hell out of your way and let you work. Go back a few decades, and compared to the alternatives of the time, this is what C was. The book that was for many years the definitive guide to C was under 300 pages long, and let the programmer almost completely ignore the messy details of things like programming in assembly. Loops and conditionals were suddenly a simple, easy mnemonic syntax.

More recently, I think this "thinness" is a huge portion of the success of Ruby on Rails. Starting from a database schema, you can literally create a functional skeleton application in minutes with just a few commands, with all of the components already laid out neatly organized and slots already created for niceties such as porting to different databases, unit testing, and version control.

Sure, it's all stuff that any competent programmer can easily handle, but automating it frees up that many more brain cells to do whatever it is the client or employer wants to give you money for.

Frank's Law of Foreign Key Constraints

2007-12-17T15:59:00.000-05:00

While bouncing around between a handful of typical LAMP style applications, I've come to a harsh realization of a brutal truth:

Those who do not learn proper foreign key constraints are doomed to create an incomplete, buggy implementation of them in their application.

Minus 50 million points to MySQL for creating an entire generation of web programmers who have only a vague, fuzzy idea of what constraints are by shipping a version that either didn't have them, or defaulted to a table type without them, for so long.

Blacklists and You

2007-12-12T08:03:00.000-05:00

Blacklists. Whether they're for virus signatures, firewall rules, or spam filters, every security guy who's spent more than 15 minutes in the business knows then, loves them, and hates them. Coding Horror has a mostly right article up summing it up titled, quite simply, Blacklists Don't Work.

On the one hand, all of the downsides he lists are dead on. Most of the reason that we frantically run around installing anti-virus software on Windows boxes are directly traceable to horribly shortsighted design decisions made as far back as MS-DOS. (Heck, search around, and you'll still occasionally find people having problems due to 8.3 filename restrictions!) And yes, blacklists are horribly inefficient, a royal pain the maintain, and often easily bypassed. After all, there's nothing whatsoever stopping our Evil Virus Author from taking his latest malware and running it through the dozen most popular virus scanners to make sure it slips by all of them.

But really, what are the other options? Are we to truly believe that there is some magic silver bullet waiting in the wings, parked next to the car that runs on water and an eclipse plugin that can tell when you typed ">" but meant ">="? Jeff puts forward the same idea that Microsoft has been painfully pushing in for years - forcing users to run as regular users instead of as administrators all of the time. Now, to be sure, this is absolutely something worth pursuing, both for security and general reliability issues. Ask anyone who maintains an open lab on a college campus how much fun it is trying to keep the right printer drivers installed and working when anyone can do anything they want on the machines!

Even this idea falls short, though. Most of those lab computers and corporate desktops, where you have site administrators who can hoard admin privs to themselves, aren't the real problem. Those computers are the ones with people babying them already, making sure passwords are strong, patches are up to date, and virus scanners are running. Sadly, it falls short when applied to Aunt Millie. She will gleefully open that email from her anonymous new best friend, follow the directions to open the encrypted zip virus, and do whatever is necessary to firmly embed the virus deep in her computer.

Even if you take away administrative rights, in a few months those same hackers will quickly start installing programs in My Documents, and use the same startup mechanisms that legit apps do. After all, it's not like you really need full system control to send spams or participate in a DoS attack. And if you do, once you get a program running on the computer, there are usually plenty of privilege escalation bugs and attacks that can get you the rest of the way, regardless of what level the user launched the program at.

The problem is that, as bad as they are, it's not quite fair to say unconditionally that blacklists don't work. They're slow, annoying, have lots of holes - in other words, they work quite horribly - and, like democracy, also happen to work better than any other workable solution out there right now. I'll agree 100% that we need to start building systems where security is just as important a design goal as reliability and profitability, but until we figure out a way to divine the intent of a given program, some form of blacklisting will always be with us.

Shared dedicated or dedicated shared?

2007-12-02T13:47:00.000-05:00

I like having internet at home. Sure, it's not quite the same as multiple 30M+ pipes at work, but it's plenty fast enough to waste time on youtube and settle arguments with wikipedia. These days, most people have pretty much two options for home connections with decent speed: DSL over phone lines, or cable modem over CATV lines. (At this point, I'm not really counting FIOS yet.)

Now the primary thing that you want from an ISP is a reliable, fast internet connection. All of the other fluffy, feel good benefits like more free email addresses, little bits of web storage, etc don't really count for much if your web pages take minutes to load. One of the little canards that DSL providers love to throw that really, really bugs me is the "DSL is dedicated! Cable is shared!"

I'm a network guy. I build and maintain 'em for a living. Now, it's true that with cable modems, the bandwidth is shared per coaxial segment among all of the customers on that segment, while each DSL customer gets to use all available bandwidth on that particular dedicated pair of lines. But guess what all those dedicated lines do? That's right, they go into a set of equipment (routers and uplinks) that are - horrors! - shared.

There isn't a network on this planet that doesn't do some level of oversubscription. Cable modem providers simply have to allocate enough bandwidth to each neighborhood loop to satisfy the actual demands, just like DSL providers have to do with their aggregation points.

Now, when an ISP starts advertising with promises of no hidden bittorent filters, secret P2P filters, or anti-criticism termination clauses - in short, the things that the Net Neutrality people have been lobbying for - then I'll care.

Swatting (computer security) flies with a bazooka

2007-11-21T21:35:00.000-05:00

In every field, there are some bad ideas that just won't die. In medicine, you have ideas such as curing cancer by pushing bones around with chiropracty. In the audio world, people get suckered into searching for "oxygen free" cables to somehow make that song, recorded in 1973 onto an 8 track, sound perfect. In the field of computer security, one of my personal favorite bad ideas is to make a magic virus that, instead of going around doing bad things, will slip in through existing security holes and clean out other viruses, install patches, and turn on firewall rules for the poor uneducated users. Dan Geer is the latest one to tie some strings onto this zombie and make it dance around.

Now, to someone who's only vaguely familiar with computers, or is used to dealing with one or two systems, this may sound like a good idea. I mean, who wouldn't love to have a magic program swoop in and clean house like an uber-l33t Mary Poppins with a keyboard? For added fun, Dan has added in an extra twist. Rather than the usual infection vector of scanning around the network (just like the viruses that people don't like), Dan proposes that secure web sites should ask users if they want security. If they say "yes", pretend that they're obviously competent and should be trusted, and run as normal. If they say "no", pretend that they're obviously idiots and fling the magic pixie dust back at their computer that keeps the Big Bad Scary Hackers hiding under the bed.

Let's start with us poor slobs stuck actually supporting the reality of computers, often in the hundreds or thousands, rather than in the idealistic realm pundits love to live in. To us, the idea of some random local bank or knick-knack vendor actually running arbitrary code on machines we have to keep going is downright terrifying. Writing this kind of code is hard - really hard. Don't believe me? Just ask Microsoft, who managed to release a silent, unblockable patch to the automatic update system that in some casesstopped updates from being installed. And that was an update only applied to Windows XP machines at a minimum patch level - imagine trying to make something so complex perfectly reliable and secure on all patch levels of Windows 98, 2000, XP, and Vista, not to mention Mac OS and Linux!

What shall we poke at next? I know! How about assumption that this code that gets downloaded to the poor computer is somehow safe itself? I mean, the whole purpose of this magic program is to make things safe on already infected machines - easy, right? Hah! Just ask the folks who spent millions creating the content protection scheme used in Blu-Ray about the impenitrability of BD+. (I'll give you a hint - it's been cracked.) Fundamental computer security 101 - once the OS is compromised, it's pretty much game over for any other programs running on it. Half the viruses out there disable the most popular virus scanners; if this magical security bit becomes at all popular, there's no reason to think it won't be targeted as well.

Okay, I think we have time for one more, so let's make it a good one. Let's assume for a moment that Mr. Geer manages to hire Tinkerbell, ensuring an adequate supply of pixie dust to the magical program work as designed. How much would that actually accomplish?

That single transaction - secured.
Any time the user logs into another site with the same password - unsecured!
Executables sent in email or instant message links - unsecured!
Phishing emails telling users to type their passwords into malicious sites - unsecured!
Malicious sites lurking on common typos of legitimate domains - unsecured!
Users picking bad passwords - unsecured!

I could go on listing other things that this idea wouldn't protect against, but I think you can see the pattern. Even if this idea could somehow be made to work completely properly, it's pretty doubtful that it would make a substantial dent in the problem of securing the computers of unskilled end users.

Well, it's time for me to head off to bed, so I'll leave you with this closing thought. Let's stick with the assumption for a moment that someone does come up with some magic <make-it-secure> HTML tag you can stick onto any web page. Instead of trying to use a yes/no dialog box as an ouija board to guess whether the computer is secure or not, why not just use the damn thing on every sensitive page?

Too much logging is almost enough

2007-11-15T22:06:00.000-05:00

Not too long ago, I stumbled across Log Everything All the Time on the site High Scalability. The short version is that the classic method of turning on debug logging when needed to track down a problem is useless because it implies that logging was off when the problem actually happened. Anyone who's dealt with an intermittent problem (only happens two or three times day, or one out of a thousand transactions, or if at least six people are wearing blue jackets) in a complex system can attest to the futility of trying to divine which combination of the tens or hundreds of thousands of events flowing by per day actually blew up. This is the deadly and elusive Heisenbug, an easily startled creature which runs away from the debug switch like a cockroach from a light switch. This particular article was written from the perspective of writing a large scale application. Building a system in this way has the huge advantage that you (where you is the programmer) get to drop debug and analysis code in at any point in the program flow. Suspect that data structures aren't getting initialized right? Print out a dump of the data structure right before the initialization routine sends it back. Garbage strings showing up in the database? Trace the entire lifecycle of the data that go into it, with a before and after every piece of code that touches them. Now that's all well and good for you lucky guys are busy actually building systems on general purpose computers from code up. But what about us poor schmucks who are running the networks that let those web 2.0 apps fling AJAX across the globe? By comparison, we're fighting with one hand tied behind our back while blindfolded.

Most network devices are, for all intents and purposes, embedded devices. This means the code running on them is far more rigidly fixed than on, say a Linux server. Open source apps can be patched and recompiled, PHP pages can have extra print statements thrown in, and even fussy vendor binaries can be spied on with strace and tweaked with library preloading. With your typical network device, though, unless you've got a really good service contract, you're pretty much stuck with whatever you've got.
The built in even logging on most network devices tends to be a bit on the anemic side. At best, you'll typically get reports of what the device considers to be unusual events. At worst, you'll get nothing at all. I've worked with at least one switch that taunts you by by just logging when an event occurs. No details, not even what even occurred - just a timestamp.
The state of a network is almost all transitory. Transactions on servers, even without logging, tends to leave various breadcrumbs behind - database entries, file timestamps, emails, etc. Once a state table entry for a stateful firewall expires, though, no trace is left behind.

The best remaining option is to constantly squeeze snapshots of the state of the network and track them. Then, later on, when someone comes to you with a report of a problem from three days ago, you at least stand a fighting chance of piecing together the half dozen otherwise normal events that line up to point at the real problem. For example, we once had a professor who's machine kept falling off the network over weekends. All of the usual troubleshooting techniques - look for errors on the port, test out the cabling, etc - showed everything perfectly normal. Looking through our more extensive state monitoring logs, we found two sets of interesting events, however. On Saturday morning, the port briefly lost a link, and a different MAC address started showing up on the port. Saturday evening, the link flickered again, and the original MAC address showed up again. Looking up the address showed that it belonged to a grad student working for the professor. The professor's own assistant was unplugging his machine! Without logging all of the "normal" state of the network, we'd probably never have found the real cause. So what information should you log? Here are what we log for network troubleshooting.

Switch FDB: This gives you the record of where all of your machines have been. More importantly, it can tell whether or not a machine was talking on a port at a specific time, rather than relying on a users guestimate.
ARP Tables: Utterly indispensable for catching misconfigurations. We've all had that bozo who set his IP address to the local gateway, or a nearby server. With this info, you already have a record of the offender, rather than tracking it down manually.
DHCP Leases: If you don't have fixed addresses everywhere, then without tracking DHCP leases, it's much harder to tell what machine a given IP address belonged to at a given time.
DHCP Fingerprints: By capturing extra information about DHCP lease requests, such as the option request list and the VCID, it is possible to do passive OS identification using techniques like those used by PacketFence. This can be particularly useful when tracking down disallowed or problematic devices, such as SOHO NAT routers.
IP Flows: An invaluable record of who talked to who when. You can either grab it from your routers directly with flow exports, capture it using span ports and a package like Argus, or even better, both. By capturing snapshots of IP flows from multiple points and looking for discrepancies, you can narrow down exactly which one of the three routers and five switches in the data path threw away a few packets. Also, your security guys will love this for helping to track down patient zero in virus outbreaks.
SNMP Traps: These are often your best insight into what the network devices think are going on. BGP session flapping, port security violations, failed login attempts - often times you can find events like these being generated in traps that don't even show up in the device log files.

Privacy without anonymity?

2007-11-11T18:21:00.000-05:00

So by now, anyone who cares has no doubt heard about the comments from Donald Kerr, deputy director of national intelligence. His claim is that nowadays, the American public has no choice but to abandon the idea of anonymity, and instead simply place their faith in the government and corporations to handle their private data properly.

Give me a break! If there's a more classic example of the foxes guarding the henhouse that doesn't involve an actual farm, I certainly can't think of it.

Let's ignore for a moment the fact that so far, these foxes have proved to be horribly incapable of properly handling data. Forget for a moment that from Choicepoint selling data to thieves, to TJX utterly failing to secure their own systems, to the countless laptops and backup tapes lost or stolen, the IT industry as a whole has not given consumers much reason to sleep soundly.

An essential aspect of free speech is the ability to speak your mind without fear of reprisal. While it's a wonderful ideal to be able to simply declare you can't be punished for speaking something unpopular, the harsh reality is that sometimes it can't be avoided. Look, for example, at the whistle blower protection laws. The fact that you can report illegal activities that a company you work for without that company knowing you did the reporting is absolutely essential. Would you turn if your boss if it was going to be months or years before results happened, and he would know what you did?

Kerr's statement is the worst kind of lie - one hidden in the middle of a bundle of truths. Do we all need to adjust our expectations and behavior regarding privacy in the age of Google? Clearly. Should people be able to expect that their government and companies will take care of their data properly? Absolutely.

But to suggest that these are an acceptable substitute for anonymity is both foolish and dangerous.

DRM vs Consumers...

2007-11-07T16:48:00.000-05:00

Ah, so it begins. For some time, consumer rights advocates have been concerned about the power that Digital Rights Management (or, as some more accurately call it, Digital Restrictions Management) grants to conent producers over consumers. Now, MLB has apparently decided that it really doesn't care about screwing over customers who bought DRM locked content.

In short, MLB has decided to change the technical details in how they protect any content they sell (in this case, videos of old games you can download and burn to disc). Who cares, right? Well, it turns out that the old DRM they use checks back in to a central server every time you go to play it. And, as part of the change, MLB sorta removed the magic server bits that convinced the player to show you what you paid for. All of those files you downloaded - and paid for! - are now just encrypted noise taking up space on your hard drive.

Oops.

I'd bet that, given the noise that's starting up, and the complete idiocy of what MLB has done, they'll bashfully find some what to at least look like they're making up for it - gift cards for screwed customers towards repurchasing the content (still under DRM, of course), or something equally feeble. A more interesting question is, what lessons will be learned from this on the industry side?

Microsoft Vista is shoving DRM features deeper and deeper into the core OS. Heck, the volume licensing arrangement already includes a component that talks to a server every 30 days, and being out of contact for too long will cripple your computer. The issues with performance, stability, application compatibility, and overall quirkiness have been enough to make resellers revolt and continue selling XP.

In both of these cases, the guys selling stuff have a de facto monopoly. No one but Microsoft gets the final say in what happens with new versions of Windows, and hanging out at your local high school baseball games just isn't the same as watching Don and Remy at Fenway. In the end, they'll be able to push through a lot of this kind of crap, and still come out reasonably well. This doesn't mean it's a good thing to build a business model around pissing off your consumers, however. Apple has had enough pressure that they're even starting to sell music without any DRM, and plenty of people have talked about Radiohead and the unencumbered downloads.

So the question now is, which lesson will the industry learn? That if you have enough momentum to not care about your customers, you can get away with DRM, or that if you offer your customers enough value, you won't need to bother with the expense of DRM in the first place?