Dodgy assertions from CASH's head medic

The salt-haters have been praising the reduction in dietary salt for an important role in the 42% fewer stroke fatalities and 40% drop in those dying from coronary heart disease:

The researchers, who include Britain's leading campaigner against added salt in food, claim that diminishing levels of salt was "an important contributor" to falls in blood pressure over the eight-year period. "As a result, the decrease in salt intake would have played an important role in the reduction of stroke and ischaemic heart disease mortality during this period," say the authors.
"Would have played"? That's a funny way of saying "was shown at a 95% confidence level to have played"... Co-author Graham MacGregor is the chair of CASH; his daytime job is Professor of cardiovascular medicine at the Wolfson Institute of Preventive Medicine at Queen Mary. So surely we can expect a rigorous and impartial analysis of the data from him.

If I'd been looking to prove or disprove this assertion, I'd have looked at stroke and heart disease rates in a range of patients over this time frame, where I had some objective measure of salt in their diet (urine samples), and looked to see whether patients with lower salt levels (in a group of patients with otherwise similar exercise, age, gender, racial stats) were correlated with lower stroke and heart disease rates. Is this what they did?

Patrick Wolfe, professor of statistics at University College London, took issue with the authors for assuming that the improved blood pressure seen in the 2003-2011 was largely the result of reduced salt intake. "Plausibility of assumption does not equal evidence," he said.
Oh. Apparently not, then. That's a piss-poor basis for the claims CASH (and international co-conspirator WASH) have been touting around about salt reduction. As commentor ID4968047 notes this reduction in strokes and heart disease could equally have come from the reduction in smoking in the past 10 years - the obligation is on Prof. MacGregor to show otherwise. Looking at CASH's writeup of the paper (the link to the paper isn't available yet, looks like) they say:
Confounding factors that were looked at include age, gender, ethnicity, education, incomes, alcohol consumption, fruit and vegetable intake and BMI.
Exercise and smoking are not mentioned. Nor do they reference the increase in statin use - and indeed Aseem Malhotra from Action on Sugar claims that statins are harmful and don't reduce mortality which is interesting as they seem to be a prime competitor to CASH/Action on Sugar's crusades against sugar and salt. Malhotra's claims got panned for lack of evidence by Prof. Rory Collins from Oxford.

It seems that others in the medical stats community have doubts too:

David Spiegelhalter, professor of the public understanding of risk at Cambridge university, cited the researchers' admission that the fall over that time in systolic blood pressure would be expected to reduce strokes by just 11% and heart attacks by 6%, small amounts of the total falls. [my emphasis] Reduced blood pressure did not represent the authors' claimed "substantial contribution" to the reduced death rates.
This is not to say that Graham MacGregor is obviously wrong in his claims. They might be true but it is a real reach to claim that this study supports them. And if this is the best he can do, I'd suggest the Marcela Trust / OMC Investments crowd who are backing CASH find someone with a better stats background to organise their crusade against salt and sugar.

Update: just managed to dig up the link to the full text in BMJ Open. From a quick look the focus was on linking salt reduction with BP reduction but not explicitly with stroke/CVD reduction.

The authors themselves admit:

It is likely that several factors, that is, the fall in BP, total cholesterol and smoking prevalence, the reduction in salt intake and the increase in the consumption of fruit and vegetables, along with improvements in the treatments of BP, cholesterol and CVD, contributed to the decrease in stroke and IHD mortality.
They have a stab at isolating the effect of salt by casting tea leaves:
it was estimated that a 2.7 mm Hg reduction in systolic BP that occurred with salt reduction would be predicted to reduce stroke by approximately 11% and IHD by 6%.
but even then the 2.7mmm Hg reduction figure they quote is the net over 8 years including factors such as decrease in smoking and increase in statins, so to attribute it to just salt reduction is "optimistic". They appeal to studies in Japan and Finland in the late 60's / early 70's but the huge gaps in time, diet and environment between now and then render the comparison unconvincing. If that's the best argument they've got to offer, I'd hate to see the ones that didn't get selected for use in the paper.

The conclusions are what kill the paper for me:

The reduction in salt intake is likely to be an important contributor to the falls in BP in England from 2003 to 2011. As a result, the decrease in salt intake would have played an important role in the reduction in stroke and IHD mortality during this period. [my emphasis]
That's a terribly weak conclusion even to my relatively untrained eyes. If they could state this more strongly, they would. Instead, they reserve their strength for polemic:
... the mean salt intake in England (8.1 g/day in 2011) was still 35% higher than the recommended level of 6 g/day, and 70% of the adult population (80% men and 58% women) had a daily salt intake above the recommended level.[14] Therefore, continuing and much greater efforts are needed to achieve further reductions in salt intake to prevent the maximum number of stroke and IHD deaths.
Reference 14 doesn't justify the 6g/day level, it's just a measurement of sodium levels. The authors don't make any reference I can see to why the recommended level should be 6g/day and not (say) 10g/day or 3g/day. If you're appealing to magic figures in your conclusion it doesn't give great confidence in the rest of your article.


A lesson from OpenSSL

If you are paranoid about secrecy on the web, today's news about a bug in OpenSSL may make you feel justified. OpenSSL is an open source library that is used by companies, individuals and governments around the word to secure their systems. It's very widely used for two reasons: 1) a very useful set of licensing conditions that essentially say you're fine to use it as long as you credit the right authors in the source and 2) because so many commercial firms depend on it, its source has been scrutinised to death to spot both performance and functional bugs.

A one-paragraph primer on SSL (Secure Sockets Layer): it's the method by which a regular web browser and a secure web server communicate. You're using it whenever the address bar in your browser displays a URL starting with "https:" instead of "http" - so that's your online banking, Facebook, Google, Twitter, Amazon... Most of these secure web servers will be using OpenSSL - there are alternatives to OpenSSL but none of them are compellingly better, and in fact the widespread usage of OpenSSL probably makes it less likely to contain security bugs than the alternatives so there's safety in belonging to the herd.

Anyone who's thinking "aha, my company should avoid this problem by developing their own SSL implementation" or better yet "my company should develop a more secure protocol than SSL, and then implement that!" has not spent much time in the security space.

And yet, someone has just discovered a bug in a very widely used version of OpenSSL - and the bug is bad.

To get some perspective on how bad this is, the Heartbleed.com site has a nice summary:

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.
Sounds dire, no? Actually the above description is the worst case; the bug gives an attacker access to memory on the secure server that they shouldn't have, and that memory *might* contain secrets, but the attacker doesn't get to control which area of memory they can read. They'd have to make many queries to be likely to gain access to secrets, and it's not too hard to spot when one small area of the Internet has that kind of unusual access pattern to your server. Even if they make 1000 reads and get one secret, they still have to be able to recognise that the data they get back (which will look like white noise) has a secret somewhere in it. I don't want to downplay how serious the bug is - anyone running an OpenSSL server should upgrade it to get the fix as soon as humanly possible - but it's not the end of the world as long as you're paying attention to the potential of attacks on your servers.

Still, isn't this bug a massive indictment of the principle of Open Source (that you'll have fewer bugs than commercial alternatives)? It's appropriate here to quote Linus's Law, codified by Open Source advocate Eric Raymond and named after the founder of the Linux operating system Linus Torvalds:

"Given enough eyeballs, all bugs are shallow"
or more formally:
"Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone."
Unfortunately, the larger and more complex your codebase, the larger the tester and developer base has to be and the longer it takes to find problems...

It's tempting to look at this security alert and declare that Open Source has allowed a critical bug to creep into a key Internet infrastructure component (clearly true) and declare that this can't be the right approach for security. But you have to look at the alternatives: what if OpenSSL was instead ClosedSSL, a library sold at relatively low cost by respected security stalwart IBM? ClosedSSL wouldn't have public alerts like this; if IBM analysis found bugs in the implementation then they'd just make an incremental version release with the fix. But the bug would still be there and would not be any less exploitable for the lack of announcement. You'd have to assume that government agencies (foreign and domestic) would bust their guts to plant someone or something with access to the ClosedSSL team mail, and in parallel apply object code analysis to spot flaws. The flaw would not be much less exploitable for lack of publicity, and would likely be in the wild longer because IBM would never announce a flaw so vocally and so users would be more lax about upgrades.

There are then two lessons from OpenSSL: 1) that even Open Source inspection by motivated agencies can't prevent critical bugs from creeping into security software and 2) that no matter how bad the current situation is, it would be worse if the software was closed-source.


Diversity in everything except opinion

This is terrifying. Mozilla CEO Brendan Eich has been forced out because of his personal position against gay marriage. If you don't believe me, read the mail from painfully hip lawyer and Mozilla Foundation chair Mitchell Baker:

We didn’t act like you’d expect Mozilla to act. We didn’t move fast enough to engage with people once the controversy started. We’re sorry. We must do better.
Brendan Eich has chosen [ah! free choice? really?] to step down from his role as CEO. He's made this decision for Mozilla and our community. [He's been given the choice to resign or be fired.]
Mozilla believes both in equality and freedom of speech. Equality is necessary for meaningful speech. And you need free speech to fight for equality. Figuring out how to stand for both at the same time can be hard.
Yes, it can be hard to stand for equality and freedom of speech, but clearly you've opted out of that stance and gone for sucking up to the media-approved line of thought. 60% of Americans support gay marriage (at least publicly) so it may be a majority opinion but opposition can't reasonably be dismissed as a small clique of bigots. It seems that opposing the majority opinion is only allowed when the majority opinion is "wrong". Let's remember that the California Supreme Court (no bastion of political orthodoxy) did not see anything wrong with allowing Californian voters to vote freely on whether marriage should be restricted to male-female partners in California.

I'm reminded of Bob Hope's quote after the 1975 Consenting Adult Sex Bill was passed:

I've just flown in from California, where they've made homosexuality legal. I thought I'd get out before they make it compulsory.
You're still allowed to choose to be heterosexual, but if you value your job and career you'd be a brave person to even hint at wavering on the issue of gay marriage.

Gay arch-blogger Andrew Sullivan can see where this is leading and he's really not convinced it's a good idea:

If this is the gay rights movement today – hounding our opponents with a fanaticism more like the religious right than anyone else – then count me out. If we are about intimidating the free speech of others, we are no better than the anti-gay bullies who came before us.

Allowing the media to dictate the acceptable positions on thought is not going to end well, and I wonder if the gay marriage lobby have really thought this one through. If Fox News continues its rise and the (more left-leaning) rest of the media continues its decline, do they want opposition to Fox editorial policies to become grounds for hate campaigns against people?

Obamacare is going to have a doctor problem

An aspect of the American Affordable Care Act (aka Obamacare) which I hadn't appreciated until this week was the degree to which it may be scuppered by primary care physicians (PCP, the USA equivalent of a GP). Chatting with a middle-aged friend who is moving to a small-ish American town and changing jobs, she opened my eyes to the mysteriously unpublicised problems which the implementation of the ACA is about to cause everyone.

She was visiting her new town last week, sorting out her house and meeting her new workmates in preparation for moving there for good in June. One item on her list was sorting out a new PCP. She had the details for her new insurance plan with one of the state providers, so dropped by a doctor's office near her house to register. No joy, that doctor wasn't accepting new patients. OK, so go back to the insurance provider website to identify a wider shortlist of providers in town.

No joy at the first doctor. Or the second. Or the third. This was getting ridiculous. She had a friend already working in her new workplace, so asked him if he could help. His doctor wasn't accepting new patients either, but the friend had a good relationship with the doctor, so asked him directly. Still no luck. Eventually my friend managed to find a large medical center open to new patients - over 60 miles away. She signed up, because that was the best offer there was.

There was one doctor in a nearby town open to new patients, but after a quick check on his reviews it became rapidly clear why. It sounds like he hadn't progressed much past the "trial by ordeal" approach to illness, and was still reading pre-Pasteur literature on infection control.

Why this insanity of unavailability? It turns out that the coming avalanche of patients signed up on ACA-compliant plans is not making doctors sleep well at night. Not only are these patients much more likely to be sick than their current patients, the main concern of practices is that they're going to lose money on treating these patients. The remuneration rate for doctors for ACA patients is - at least in some states, and I suspect all - based on 80% of the standard practice fees from 3 years ago. That's not great, but much worse is the experience with Medicare (federal medical coverage for the elderly). My friend used to work in a doctor's office, and they had a profitably employed office employee whose sole job was to push back against government agencies spuriously denying payment for Medicare claims. If there were any abnormalities at all in a claim, Medicare would deny it in the hope that only solidly valid claims would be retried. A denial is, after all, cheap.

I digress, but bear with me. Do you older readers remember when The Simpsons started and they introduced us to Marge's sisters Patty and Selma, stalwart misanthropic employees of the local Department of Motor Vehicles? There's a reason why the American viewers of the show laughed, and it wasn't to do with absurdity. There's no-one more un-fireable than a federal or state bureaucrat, and their attitude to their customers is exactly what you'd expect - entirely down to their general disposition to humanity. That's what the doctors' billing teams are going to be dealing with. Only by being better acquainted with the rulebook will they be able to get their due payments out of the ACA system, and even then those payments will be small and paid late.

Having a new government agency and new set of rules means that small practices are extremely worried about their costs and ability to maintain existing patients. Because (I believe, from chatting to the front desks of a few offices) you can't be selectively open to new patients based on their particular insurer, the easiest option for small providers is to batten down the hatches until the picture is clearer. Only the very large healthcare centers can take advantage of their economy of scale to accept the new stream of ACA-covered patients - and with them the standard employer-covered and easier to manage new patients.

I wonder whether this problem with finding doctors is behind the proposed rise in rates for ACA-compliant plans? Perhaps you will be able to look at the rate rise in the next few years and extrapolate the "marginal cost of doing business with government". As it stands, the best they can hope for is to break even:

Cigna, which is selling Obamacare plans in five states and is considering whether to expand that further in 2015, has said it won't make money on the business this year. It expects to have signed up as many as 100,000 new members under the program for this year.

So for the Affordable Care Act, if you're sick then you may be able to get coverage where you couldn't before, but sure as heck if you're outside a major city then you're going to find it a challenge to get a primary care physician to see you and get your care started. Now I'm wondering what the ER departments of ACA-plan-accepting hospitals are going to experience in the next year.


When protecting minorities screws them over

Fascinating. I came across this Dave Winer story of tech hiring and firing in 1985's Silicon Valley via the money quote:

...every time a company hires someone who is not a young male, they run the risk that the new hire isn't there to work, rather is there to scam you.
since from that quote I wondered "what the hell planet is this author on?"

Then I read the story. And blow me down if I didn't end up at least partially agreeing with the author. Go read the whole thing.

Commenter sep332 clarifies that the problem described (an older tech worker using his age to file a discrimination claim after being fired) isn't actually about age, sex or anything in particular:

The laws about protected classes are not about classes of people. Anyone can claim that they were discriminated against for gender reasons, not just women. White people can claim they were targeted because of their race, etc. So the people who couldn't realistically claim discrimination are the people who are most like the rest of the company. [my emphasis] I mean, if women are the majority of your company, then it would be hard for a woman to claim gender discrimination.
In the tech sector, the majority of your company are likely to be young, male, and (in Silicon Valley) a mix of white, Chinese and Indian in race; probably also straight although I have known a couple of small firms where that was not the case. If you're middle-aged, female, transgender, black, Inuit, Pacific Islander or Hispanic then you're almost certainly in a clear minority and hence possibly a "protected class".

As Winer notes, in a small struggling tech firm if someone comes at you with a discrimination lawsuit then you haven't the money or time to fight it. Unless it's a complete no-brainer (an 18 year old white male alleging discrimination on the grounds of inability to get out of bed) your best option is to pay up and move on. So what do you do when you have the risk of recruiting people who can sit back and do nothing while being practically un-fireable? Simply minimise the risk of recruiting them, by avoiding anyone who is in a good position to do this to you.

An ironic "well done" to everyone who has pushed through these employment laws, and a special raspberry to everyone who has filed (or backed) an abusive lawsuit exploiting these laws. You've screwed over everyone in the tech sector who's not a young male.


Storing in the cloud

So this is interesting. Google is dropping its cloud storage rates to $10 per month per TB (though 100GB costs $2, twice that rate). Amazon S3 storage is currently $85 per TB per month and Microsoft Azure is $64 per TB per month for their cheapest option (Locally Redundant Storage). I'd expect these prices to be dropping fairly soon in response to Google's move.

How much does this actually cost to provide? Let's look at the cost of storing and accessing 1 TB of data. An internal SATA 1TB hard drive costs about $60 on Amazon - but a 2TB costs $85, and a 4TB costs $160 (retail). So we can figure on about $40 per TB of storage. How long will this drive last? Mean time between failures of hard drives is between 18 months and 3 years depending on manufacturer and usage; let's split the difference and say 2 years. Buying 1TB of storage over 2 years will cost the supplier about $40 in capital costs. Isn't this a rip-off?

Well, having a hard drive is one thing - being able to access it is another. You've got to get data into that hard drive, and probably you want to get it out again. Assuming that the entire volume of that drive is written once and read twice per 2 years (probably a lowball estimate) at a rate of about 5 Mbits/s, that means that in 1 day (86400 seconds) you could read (86400 * 5 / 8) MB or about 54 GB per day so it would take up about 10 days per year per user, and so you could support about 36 users on a 5 Mbit connection. Let's say we're using 4 TB drives so you need a 5 Mbit connection for each 9 computers in your storage.

It's not quite that straight forward though. Cloud storage is supposed to be reliable, and hard drives are manifestly not - they die all the time. Therefore you want at least a second copy of your data on a separate hard drive, and ideally you want that second copy to be in at least a separate building in case of a physical disaster (flooding, fire, tornado). Generally the further away the better, at least up to 100 miles or so, though distance tends to increase the expense of hosting because for every write to the data you need to send the write to your remote facility as well. Azure gives you the explicit option of how physically separate you want your data to be; Locally Redundant Storage vs Geographically Redundant Storage.

There's also the distinction between data loss and data unavailability; if the primary copy of the user's data is unavailable (e.g. because the data center has a planned or unplanned availability outage) cloud providers may give their customers the option of reading data from (or writing data to) the backup copy of the data. Customers can buy this kind of read access from Azure as an additional option (Read Access GRS).

If you as the cloud storage operator rack up 3600 users, then, you'll need 900 computers with a total of 3600 TB of storage in each of 2 sites. You'll need 500 Mbit/s of bandwidth on each site if you want to offer read redundancy, and about 170 Mbit/s of bandwidth between sites to replicate writes. You don't need customised hardware for this amount of traffic, but you do need to buy the bandwidth to get the data to and from the user. Azure quotes $120 for egressing 1TB of data; if we estimate that it actually costs them $100 then each user will cost you $200 (reading their data twice), so you will have $720,000 of bandwidth cost.

If the computers last about as long as the hard drives you'd expect them to cost you about $300 plus the storage ($160), say about $500 once you take into account rack and network switch hardware. I suspect power won't be too much of an issue since storage isn't a CPU intensive operation and user access is intermittent - idling computers without a display consume about 20W. So each site will cost you 900 x $500 over two years, and consume a steady 18 KW of power. Electricity costs about $100 per MWh to make in the cheaper parts of the USA, so power will cost you $2 per hour per site. So power is only about 7% of your equipment costs for a 2 year lifetime, and you end up paying about $1.8M in total in hardware, power and bandwidth to provide 1TB of cloud storage to 3600 users over 2 years. Each user pays $240 over that time, or $860K in total. So it would seem that $10 per hour is a massively losing proposition for the provider even before we take into account the human costs of designing, building and operating the system.

The real picture is more nuanced. We implicitly assumed that every user would use all the storage (and bandwidth) they paid for. In practice, they could conceivably be consuming only half of what they've paid for. As long as we can dynamically provision for users (having a small amount of storage headroom, and adding on more computers and drives as that headroom is threatened) we could provide maybe 60% of the maximum hardware needed, so instead of $1.8M our costs would be down to $1M or so. Still something of a loss.

I think the way cloud companies can make money - or at least avoid a loss - on this is to make use of the fact that the computers providing access to the drives are seldom even slightly busy. Instead of buying $300 of low power computing hardware to support each 4TB drive, just chain a few 4TB drives onto an existing computer that you're using for something else (say, Bing search, Google maps, Amazon website serving). When a user starts to access their data, temporarily reserve a core or two on the machines holding that data to serve it. That way you save nearly 60% of your hardware costs and might just be bringing your operation into slight profit.

You'll still need to pay the design and implementation costs for your system, not to mention the usual marketing and business operations, but these don't scale in proportion to your number of users. The more users you have, the better your business looks.

$10 per month per TB is a bit of a game changer. Suddenly storing in the cloud isn't massively more expensive than storing on your hard drive. I wonder what the next couple of years will bring?


Piling on Piers Morgan

I was initially surprised that the news of Piers Morgan's fall from grace resulting in the cancellation of his CNN prime time talk show had been news in the UK, but I guess the prospect of Morgan returning to Blightly was sufficiently generally appalling that the Brits were quite concerned about the prospect.

Yesterday Piers had comedienne Chelsea Handler on his show. Noted for brutal honesty, such as discussing personal abortion and DUI stories, Chelsea didn't disappoint but possibly not in the way Piers expected. The discussion following the show's commercial break was quite enlightening:

Chelsea: I mean, in the middle of the commercial break – I want your viewers to know; they must know, because they're probably following you on Twitter. I mean you can't even pay attention for 60 seconds. You're a terrible interviewer.
Piers: Well, you just weren't keeping my attention. It's more of an issue with you than me.
Chelsea: That's not my problem. This is your show, you have to pay attention to the guest that you invited on your show.
Piers: If they're interesting enough ...[cut off]
Chelsea: Listen, it doesn't matter how interesting I am. You signed up for this job. [..] Well, maybe that's why your job is coming to an end.
Piers has - or had - the 9pm-10pm slot on CNN, which is prime time. (West Coast viewers, 3 hours behind the East Coast times quoted, can usually see the show live at 6pm Pacific or repeated 3 hours later). There's a constant battle for viewers between CNN and Fox in the evening, and Fox announced in August that newscaster Megyn [yes, really] Kelly would be taking over the Fox 9pm slot from previous incumbent Sean Hannity. Piers seemed keen on the challenge, tweeting "Bring it on, Megyn Kelly". Kelly duly brought it on, starting in early October and two months later was beating Piers 5 to 1 in viewers, up 10%-20% from Hannity.

Why did CNN viewers turn away from Piers in droves? Kelly is clearly easier on the eyes than Piers, but that can't be the whole story. I think Tim Stanley in the Telegraph gets closest to it:

But he acted as though no one had ever thought to discuss the subject[gun control] before. Like, ever. He tried to make gun control his own personal crusade, to "school" the Americans on law and order. And he displayed a crass insensitivity towards issues such as the importance of the Constitution or the American tradition of self-reliance. The scale of his ego was extraordinary. No US liberal has ever managed to challenge their country's fundamental respect for gun ownership. Why did he imagine that a guy with an English accent – the accent of George III no less – would succeed where Bill Clinton, Teddy Kennedy or Barack Obama had failed?
The embarrassing thing about the whole Piers Morgan affair is that it has turned the tables on us Brits. We always insisted that we were the courteous ones and the Americans were the boobs. In this case, it's been the other way around.
Americans really don't like being told, imperiously, what to do. Piers did this all the time, and worse wouldn't admit when he'd been beaten. He invited young Republican Ben Shapiro to debate guns with him after the Sandy Hook shooting, and had his clock unexpectedly cleaned - Shapiro tried to load the game by going for a "town hall" format which would have stuffed the audience with shooting victims and Shapiro wisely refused, telling him 1-on-1 or nothing. So, nothing.

Looking at interview styles, probably the best comparison on Fox with Piers is Bill O'Reilly who hosts the 8pm-9pm slot. O'Reilly has a similar format, inviting guests to discuss contentious issues 1-on-1, or sometimes 2-on-1, with him. One key difference, I think, is that O'Reilly doesn't try to lay traps - he relies on his (formidable) preparation and debating skills. Unlike Piers he has regular guests from a range of political views - liberal Juan Williams is one of the favourites - and the debate can get quite shouty at times but the guests generally know O'Reilly, know the position he'll take, and have wrestled with him often enough to give a good showing. More important, O'Reilly doesn't belittle them. He tries to reason, and even if you don't agree with his (often reactionary) position you can see that he's playing the man rather than the ball. Piers by contrast is a classic bully, trying to belittle opponents.

CNN's Anderson Cooper is a marked contrast to Piers; I've seen Anderson do interviews, and he can be tough - Greg Smith who resigned publicly from Goldman Sachs did an Anderson Cooper interview, and Cooper didn't go easy on him at all - but he's fair. You feel as if he's trying to make the interviewee explain what he's hiding from the audience, rather than browbeating the interviewee from a position of power just to establish a pecking order like a bully does. It would be hard to overstate how much most Americans look down on bullies. The War of Independence was essentially a reaction against perceived bullying (imposition by means of might) of the American colony by King George III. Piers is a classic bully, and Americans simply don't like that kind of person.

Interestingly it seems that even his own staff weren't keen on him:

"The makeup girls suffered the worst — he was rude and belligerent," says our source. "The general feeling is Morgan didn't show any respect to anyone working under him — the people who were trying to make him look good."
It would be uncharitable to note how hard a job that is. But this is Piers Morgan, so screw charity.


The next step in the gentrification wars?

I'm going out on a limb here and saying that the massive fire in a new apartment building in San Francisco is not unrelated to the past year's increasingly violent struggle between long-term residents and new arrivals from the tech sector. According to local TV station KVTU:

Another Strata resident, 25-year-old Hisham Bajwa, said he could see the fire start to burn outside his window shortly before 5 p.m.
"There were two main points of fire, one on the left and one on the right," Bajwa said. "It got pretty big pretty fast."
An interesting observation; it would be surprising for an accidental conflagration to manifest in two separate points. Now eyewitness reports are famously inaccurate, and the fire could have spread internally before being visible in two external points, but it does make you wonder...

You can see the building under construction to the west of the intersection between 4th Street and China Basin Street. The implication of the size of the construction is that it was a new apartments block - opposite it is Strada Apartments which had to be evacuated, and just up the street is Channel Mission Bay Apartments. So why does a huge new apartment building start to burn down? A welding accident? A gas leak igniting? Or something more deliberate?

The Mission District in San Francisco next to where these apartments are located has been Ground Zero for the protests against the influx of tech and biotech workers from Apple, Google, Facebook, Genentech and others. Assaulting Google Glass wearers in bars, blockading shuttle buses or just generally protesting tech nerds has become an increasingly popular sport in central San Francisco. Since it's nearly impossible to increase rents significantly in San Francisco apartments or evict a renter who doesn't want to leave - even if their contract is at an end they can require the landlord to renew it at substantially the same rent and terms - the only way that most landlords can improve their rent income is to invoke the Ellis Act to "go out of business" and sell their properties to another company, which then changes the use of the building (often via a drastic knock-down and rebuild).

The organised protests to date have mostly been focused around the shuttle buses which take the tech workers to Cupertino, Mountain View, Sunnyvale and other places across the South Bay; they are a very visible and concentrated target. Perhaps now the anti-tech movement is changing tactics: instead of impeding the transport, make building luxury apartments a much more expensive and chancy business. If they cause enough disruption maybe they'll be able to slow the influx of tech money and keep their existing apartments.

The investigation into this (hundred millions of dollars?) fire could be very interesting.

Update: KTVU confirms a $220M+ project with 360 apartments and "Arson investigators were on the scene Tuesday and will return Wednesday morning. " I bet they will.


Fixing Healthcare.gov - the inside story

The new Time covers in depth the work of the team who fixed Healthcare.gov. It's a fantastic read, with good access to the small but extremely competent team who drove the fix - go absorb the whole thing.

The data coming out of the story confirms a lot of what I suspected about what was wrong and how it needed to be fixed. Breaking down by before-and-after the hit team arrived:


  1. By October 17 the President was seriously contemplating scrapping the site and starting over.
  2. Before this intervention, the existing site's teams weren't actually improving it at all except by chance; the site was in a death spiral.
  3. No one in CMS (or above) was actually checking whether the site would work before launch.
  4. The engineers (not companies) who built the site actually wanted to fix it, but their bosses weren't able to give them the direction to do it.
  5. There was no dashboard (a single view) showing the overall health of the site.
  6. The key problem the site had was being opened up to everyone at once rather than growing steadily in usage.
  7. The site wasn't caching the data it needed in any sensible way, maximising the cost of each user's action; just introducing a simple cache improved the site's capacity by a factor of 4.
I refer the reader in particular to my blogpost The Curse of Experts where CMS head Marilyn Tavenner was trying to dodge blame.
During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days. "In retrospect, we could have done more about load testing," she said.
As the Time article shows, this was anything but the truth about what was actually wrong.


  1. There wasn't any real government coordination of the rescue - it was managed by the team itself, with general direction but not specific guidance from the White House CTO (Todd Park)
  2. The rescue squad was a scratch team who hadn't worked together before but was completely aligned in that they really wanted to make the site work, and had the technical chops to know how to make this happen if it was possible.
  3. Fixing the website was never an insurmountable technical problem: as Dickerson noted "It's just a website. We're not going to the moon." It was just that no-one who knew how to fix it had been in a position to fix it.
  4. The actual fixes were complete in about 6 weeks.
  5. One of the most important parts in improving the speed of fixing was to avoid completely the allocation of blame for mistakes.
  6. Managers should, in general, shut up during technical discussions: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."
  7. The team refused to commit to artificial deadlines: they would fix it as fast as they could but would not make promises about when the fixes would be done, refusing to play the predictions game.
  8. Having simple metrics (like error rate, concurrent users on the site) gave the team a good proxy for how they were doing.
  9. Targeted hardware upgrades made a dramatic difference to capacity - the team had measured the bottlenecks and knew what they needed to upgrade and in what order.
  10. Not all problems were fixed: the back-end communications to insurance companies still weren't working, but that was less visible so lower priority.

The overall payoff for these six weeks of work was astonishing; on Monday 23rd December the traffic surged in anticipation of a sign-up deadline:

"We'd been experiencing extraordinary traffic in December, but this was a whole new level of extraordinary ... By 9 o'clock traffic was the same as the peak traffic we'd seen in the middle of a busy December day. Then from 9 to 11, the traffic astoundingly doubled. If you looked at the graphs, it looked like a rocket ship." Traffic rose to 65,000 simultaneous users, then to 83,000, the day's high point. The result: 129,000 enrollments on Dec. 23, about five times as many in a single day as what the site had handled in all of October.
Despite this tremendous fix, however, President Obama didn't visit the team to thank them. Perhaps the political fallout from the Healthcare.gov farce was too painful for him.

The best quote that every single government on the planet should read:

[...] one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change. "It was only when they were desperate that they turned to us," says Dickerson. "I have no history in government contracting and no future in it ... I don't wear a suit and tie ... They have no use for someone who looks and dresses like me. Maybe this will be a lesson for them. Maybe that will change."
The team who pulled President Obama's chestnuts out of the fire didn't even think they were going to be paid for their work initially; it looks like they did eventually get some money, but nowhere near even standard contracting rates. And yet, money wasn't the motivator for them - they deeply wanted to make Healthcare.gov work. As a result they did an extraordinary job and more or less saved the site from oblivion. This matches my experience from government IT developments: it's reasonable to assume that the government don't care about whether the project works at all, because if they did then they'd run it completely differently. Though if I were President I'd be firing Marilyn Tavenner, cashing in her retirement package and using it to pay bonuses to the team who'd saved my ass.

If you have a terribly important problem to solve, the most reliable way to solve it is to find competent people who will solve it for free because they want it to work. Of course, it's usually quite hard to find these people - and if you can't find them at all, maybe your problem shouldn't be solved in the first place.

Don't give the guests power over the residents

Top legal blog "The Volokh Conspiracy", now at the Washington Post, analyses the recent California 9th Circuit decision that wearing American flag shirts at high school can legally be prohibited. Eugene Volokh notes that the actions of the principal (Mr. Rodriguez) in banning wearing of American flag clothing in fear of it causing violent disruption may be constitutional but not at all a good idea:

This is a classic "heckler's veto" — thugs threatening to attack the speaker, and government officials suppressing the speech to prevent such violence. "Heckler's vetoes" are generally not allowed under First Amendment law; the government should generally protect the speaker and threaten to arrest the thugs, not suppress the speaker’s speech. But under Tinker's "forecast substantial disruption" test, such a heckler's veto is indeed allowed.
I have to confess sympathy for Mr. Rodriguez in his predicament - his job is to ensure order and prevent disruption at his school, and students who wear the American flag did seem very prone to be correlated with disruption:
At least one party to this appeal, student M.D., wore American flag clothing to school on Cinco de Mayo 2009. M.D. was approached by a male student who, in the words of the district court, "shoved a Mexican flag at him and said something in Spanish expressing anger at [M.D.;s] clothing."
Now there are plenty of legal Mexican immigrants in the USA, so we shouldn't assume anything about the angry student's immigration status, but if (for instance) a student of Scottish heritage took offense to a Live Oak student wearing an American flag on St. Andrew's Day and threatened him "I'll cae the pins o' ye!" I can't imagine Mr. Rodriguez reacting the same way. The principal does seem to be bowing to the opinions of a category of "guest" students in preference to those who are citizens of the country. (Irish students aren't likely to cause problems on St Patrick's Day because it's celebrated in the USA as much if not more than in Ireland).

If you want to know more about what Live Oak High School is like, take a look at the California department of education stats for the school. It's about 50-50 demographic split between white and Hispanic/Latino students. The standardised scoring indicates that white, Asian and black students improved significantly in the past academic year but the Hispanic/Latino students went backwards. It seems that pandering to them isn't doing them any favours academically. Incidentally, I'm dubious about the "Two or more races" stats - only 1 student of mixed heritage out of 858? My arse.

I can do no better than quote Volokh's takeaway:

The school taught its students a simple lesson: If you dislike speech and want it suppressed, then you can get what you want by threatening violence against the speakers. The school will cave in, the speakers will be shut up, and you and your ideology will win. When thuggery pays, the result is more thuggery. Is that the education we want our students to be getting?
If Live Oak High School really wants to help its Hispanic/Latino students, it should insist that they meet the standards expected of all other students.