Who is really to blame for the San Fran network lockout?

A strange sort of techno-drama is playing out in the city of San Francisco, California right now. The blame for the fiasco may not be as easily assigned as it first appeared.

——————————————————————————————————————-

Last Sunday, the 13th of July, a computer network administrator named Terry Childs was taken into custody by police and charged with four felony counts of “computer tampering.” The San Francisco Department of Technology has a new FiberWAN, and Childs was intimately involved in its implementation. In fact, he is apparently the only person in the city with administrative access to most of the Cisco routers on the network — and his unwillingness to turn over the administrative password is the reason he’s now in jail, unable to come up with the $5 million he would need for bail.

Beyond that, the story gets a little muddy. The DA’s office is keeping a lot of what’s going on under wraps. The press doesn’t have free access to Childs; his lawyer must of course be careful what he says; city officials aren’t talking much; and Childs’ supervisors and coworkers aren’t rushing to tell their stories either. There are quite a few articles reporting parts of the story:

• The basic, official story, as it stood on Monday, was reported by The San Francisco Chronicle in an article titled S. F. officials locked out of computer network.
• The San Francisco Chronicle also reports in Computer engineer keeping quiet on lockout from Wednesday.
• InfoWorld’s Why San Francisco’s network admin went rogue tells the story from the point of view of an anonymous source on the inside.
• An InformationWeek article, Suspect in Hijacking Of San Francisco Computer Network ‘Willing To Cooperate’, offers an update on Childs’ disposition from Friday.

What really happened — and who’s really to blame?

The following tale is my view of what happened, pieced together from the above and other sources.

In the beginning . . .

Terry Childs was a smart, talented network engineer, an indispensable resource for the City of San Francisco Department of Technology. He was also not the friendliest man on the planet, with a touchy temper and perhaps a bit arrogant — and had a criminal record, convicted 25 years ago in Kansas for aggravated robbery. He had a low opinion of some of his colleagues and his non-technical supervisors. Despite all this, his skills were respected, and some — including Mayor Gavin Newsom — described him using terms such as “well-liked.” When the lead network architect for rolling out the new FiberWAN wanted to make sure things worked, he relied heavily on Childs’ expertise, and, in fact, Childs ended up performing most of the implementation for the new network.

To ensure the continued stability of the network, he took on the task of maintaining it almost single-handedly, having a great deal of difficulty trusting any of his colleagues to do network administration tasks for the FiberWAN without screwing something up. His bosses, colleagues, and intra-organizational clients understood that someone who did not know the network sufficiently well could innocently do a lot of damage, and were mostly content to just let Childs handle it. His local authentication system with sole access to administrative functionality in his own hands had been in place for months, if not years, and in the words of an anonymous source inside SF’s Department of Technology, “everyone more or less accepted it.”

Childs was frustrated with his circumstances, however. Among his many problems as the lead network engineer for the entire city, he complained that his direct superior was “intrusive, incompetent, and obstructive”; the managers above his direct supervisor “had no real idea of what was going on, and were more interested in office politics than in getting anything done”; he was overworked, putting in far more than 40 hours a week (receiving comp time for overtime work that he would never have time to use); “many of his colleagues were incompetent freeloaders.” Apparently, there was some truth in a lot of his complaints, and personally I’m not surprised at the thought that they might all be true.

Things come to a head

Things came to a head when a new information security manager was brought on board by the City, and Childs came up for a performance review. The security manager started prying into things, trying to get Childs to give up administrative passwords for the network, and in his review he was told he was performing poorly. It quickly began to look like Childs’ professional head was going to be on the chopping block before long. Considering his circumstances, it seems reasonable he might be flabbergasted to discover that he — the only person willing and able to do much of what he did for the city every day, the man who kept everything running smoothly, who was effectively on-call 24/7/365 and put in as much overtime as anyone for effectively no reward at all — was on the fast-track to being fired for poor performance.

Part of his frustration revolved around the fact that, according to the anonymous source from inside the Department of Technology, Childs had told him, “I’ve been trying to get them to approve [a security policy] for years. I’ve written ones up and submitted them, but they don’t want to do it, because they don’t want to be held to it.” Finally, the brand new security manager has put a policy in place — a policy that is unenforceable, essentially just an unmodified template from the CCISDA that’s still awaiting discussion and alteration by a committee that doesn’t yet exist.

He was dismissed from work for “insubordination” on the 9th of July, but still received his regularly scheduled $127,735 a year salary paychecks. He may very well have become verbally confrontational with his superiors and the new security manager — the latter of which became so disturbed after an encounter with him that she locked herself in her office to escape having to deal with him. He is alleged to have begun monitoring others’ communications over the network with regard to his personnel evaluation case.

He refused to give up administrative passwords. He was threatened with arrest and continued to refuse to give up administrative passwords for the FiberWAN routers. Finally, he was charged with four felonies, and now sits in jail, with a bail determination five times what you’d expect many murderers to receive. As of Tuesday at the latest, according to his lawyer, he has been willing to cooperate — but the DA’s office refuses to comment on talks with Childs and his lawyer.

Officials suggest there may be reason to believe Childs gave access to some third party, outside of law enforcement, the DA’s office, and the city’s Department of Technology, and even said he might possess some kind of “electronic device” that might be used to gain access to the network and destroy “hundreds of thousands of sensitive documents,” presumably including jail bookings and other law enforcement documents, payroll files, and e-mails. These bold assertions of the danger Childs represents were followed up by searches of his home and car for devices that may be used to compromise network security, which turned up — nothing.

According to the mayor, “There’s nothing to be alarmed about, save the inability to get into the system and tweak the system. Nothing dramatic has changed in terms of our ability to govern the city.” By all accounts, the entire network continues to run smoothly, in Childs’ absence — perhaps because nobody who doesn’t understand the workings of the FiberWAN configuration can change anything.

The blame

It’s obvious that, officially, Terry Childs is taking all the heat for this. Based on what I’ve read, and the mental picture I have of the situation (as explained above), that’s not really a fair assessment of the situation.

1. Terry Childs treated the network like his own personal kingdom. That’s not the best way to deal with such things, by any means. He should have documented everything, created additional administrative passwords and procedures by which others can access them if he gets hit by a bus, and otherwise done what he could to make sure that the sanctity of network performance, stability, and security didn’t rest entirely on his shoulders. It’s not an uncommon state of affairs in the IT world, though, and at times is even a necessary state of affairs when a network administrator has too little power and too few resources for the responsibility he must assume.
2. His superiors gravely mishandled the situation, obviously. Even if only half the story told by the anonymous source and only half the implications of what has been offered in the official stories are true, they’ve done everything wrong from one end to the other. It’s possible the only reason things have worked as smoothly as they have so far is Childs’ skill and dedication.
3. The prosecutors and law enforcement officers involved, judging by what I know of prosecutors and law enforcement officers (to say nothing of human nature in general), are probably more interested in convicting Childs than they are in resolving this matter. If he’s really willing to cooperate, and if the whole matter was the result of a misunderstanding (as Childs’ lawyer contends) and mutual mismanagement, most of the felony charges against him should be dropped and the passwords recovered. As long as Childs maintains his innocence and refuses to plead guilty to several felony offenses, though, it’s likely the DA’s office will do more to hinder attempts to get access to administrative passwords out of Childs than to help.

As far as I can see, there’s blame all over everyone, like someone filled up a gigantic balloon with the stuff and everybody involved stabbed it with a letter opener at the same time.

The solution

It’s possible I gravely misunderstand the circumstances, given the fact I surely don’t know all the important details, but considering the way these things tend to play out, I find it unlikely that they diverge much from my guesses above.

If I’m close to the mark, there’s a simple solution: dismiss any charges against Childs except those related to obviously illegal and unethical behavior; keep charges related to actions such as monitoring others’ e-mails without authorization; and reduce the severity of the remaining charges. Then let him go with probation in exchange for completely divulging all information necessary for managing the network, including passwords, backups of router configurations, and so on.

Then, when all’s said and done, let the Department of Technology suffer the consequences of firing the only man in the city who could do his job and siccing the police on him. They’ve certainly made an uncomfortable bed for themselves.