2014-04-23

Tanya Gold on mountaineering

Alas, Tanya Gold of The Guardian has descended to clickbait headlines to keep up interest in her articles; her latest on the Everest avalanche that killed thirteen Sherpas is particularly painful:

As commercial climbing has exploded, Everest has shifted from an explicit wasteland to a moral and internal one which also serves as a perfect metaphor for the contempt in which we hold the planet.
It is not simply the ordinary exploitation of the Sherpas, which is soothed away with the knowledge that in Nepal, where the average annual wage is $700, a Sherpa can make $5,000 in a two-month season – although it is impossible to imagine this kind of death rate being tolerated if the dead were rich and white.
The fall of a serac in the Khumbu icefall was tragic, but by no means a bolt from the blue. Jon Krakauer's account of the 1996 Everest disaster that killed eight climbers (which Tanya references) describes how perilous the icefall can be - falls of huge, building-sized chunks of ice can happen without any more warning than a "look out!" from your companions, and even an injury such as a broken leg can prove fatal in the perilously hazardous conditions of 18000+ feet altitude where simply being there can provoke debilitating or fatal altitude sickness.

The Everest climber fatality rate between 1922 and 2006 is about 2% overall, and about 1.4% if you exclude porters (Sherpas and others). Certainly, being a Sherpa is more dangerous than being a paying climber, but as a Sherpa you're still about the fatality rate of an astronaut and your relative compensation is better - if a Sherpa makes 7x the average annual wage, look at a top-end salary for an astronaut based in Houston, TX which is about $141K, or about 2.5x the mean US wage. This is not comparing apples to apples, but at least gives you a ballpark picture of how well Sherpas are compensated. Sure, they have a dangerous job, but no-one is forcing them into it and they probably have a better understanding of the dangers than most of the climbers. So when Tanya says:

...although it is impossible to imagine this kind of death rate being tolerated if the dead were rich and white.
it turns out we actually tolerate this death rate already, even though the dead are moderately well paid and mostly white. Perhaps Tanya doesn't have a great imagination.

Looking at Tanya, I think it's safe to say that she's in no danger of attempting to summit Everest (or indeed any peak more challenging than Brown Willy) any time soon. Perhaps then she cannot appreciate what drives people to push themselves to their physical and mental limits to overcome the imposing challenge of high altitude mountaineering, and we should not blame her for that. We should, however, nail her to the wall for comments claiming that money trumps humanity for climbers:

But more tourists claim "tunnel vision" and "summit fever". They do not pause; they are slaked on their own fantasies; they paid too much. Madness indeed.
At Everest summit altitudes, even a very fit climber has to draw on all their reserves of strength to survive the Death Zone. Even a small amount of additional exertion in aiding a fellow climber can cause them to collapse and double the number of people that need help. When you're on a climb to the summit of Everest, your survival is your own responsibility; it's unlikely that anyone will be able to help you; trying to help someone else can make you pay much, much more than just your climbing fees. All climbers who have reached the final base camp will know and understand this, much more than someone like Tanya can even strain to appreciate.

2014-04-20

Strategy in 2048

I've been playing the 2048 game which (if you haven't played it yet) is the most phenomenonal time sink invented. To save people from sanity, here are some general game hints; using this strategy I manage to obtain the 2048 tile in roughly 50% of games.

  1. Start by building up the big numbers in the lower left corner, spreading along the bottom row.
  2. As the bottom row is nearly full (say you have [16, 8, 4, _] start filling the third row with numbers starting from half the lower left number (say, [8, 4, 2, _)
  3. When the opportunity arises to have the bottom row full and the third row filling 3 of 4, right shift the board so the third row numbers line up above the same numbers on the bottom row and then drop them down and shift left to increase your bottom row numbers by a factor of 2.
  4. When the bottom row starts getting big (say, 64+ as the left corner number) start trying to order the third row in the opposite direction. If the bottom row is [128, 64, 32, 16] start trying to create 16 at the end of the third row. Whenever you can, drop numbers into the bottow row.
  5. At nearly all cost, avoid filling row 2 so that you have no option but to move the board up - that will trap small numbers under your row of big numbers. If that happens, drop the board again immediately (and hope).
  6. If you end up with 2 or 4 tile to the left of a big number tile on your bottom row, focus on increasing that tile number until it matches your big number so you can left-shift the bottom row and have the biggest tile in the bottom left corner.
Good luck!

2014-04-17

Assuaging public concern by taking the piss

It seems that homeopathy is gaining adherents in Portland, USA: 38 million gallons of water had to be flushed from a reservoir after a guy took a leak in it:

The Portland Water Bureau there's little risk to the public's health but bureau administrator David Shaff says, "Our customers have an expectation that their water is not deliberately contaminated. We have the ability to meet that expectation."
Um, yes, but ability and requirement are not the same thing...

Now this wasn't as expensive as it could have been elsewhere in the country, e.g. neighbour California is in a drought and water is a precious resource there, whereas Portland is famous for its damp climate. Still, treating water costs actual money. Why did the Portland Water Bureau feel compelled to take this drastic action? A pint of urine in a 38M gallon reservoir would have been practically impossible to detect, let alone have any effect. You'd get more contamination every day from birds pooping.

It's all about perception, of course. Once it became known that someone had peed in the reservoir, if the PWB had done nothing then there would have been a popular outcry (fuelled by the press, who love a story like this; can you imagine the puns if the Sun wrote this up?) The PWB is merely pre-empting these protests, saving itself from the grief by expending someone else's money. Wouldn't you?

Mind, anyone in Portland who still drinks Budweiser or Coors wouldn't have much room to complain...

2014-04-14

Dodgy assertions from CASH's head medic

The salt-haters have been praising the reduction in dietary salt for an important role in the 42% fewer stroke fatalities and 40% drop in those dying from coronary heart disease:

The researchers, who include Britain's leading campaigner against added salt in food, claim that diminishing levels of salt was "an important contributor" to falls in blood pressure over the eight-year period. "As a result, the decrease in salt intake would have played an important role in the reduction of stroke and ischaemic heart disease mortality during this period," say the authors.
"Would have played"? That's a funny way of saying "was shown at a 95% confidence level to have played"... Co-author Graham MacGregor is the chair of CASH; his daytime job is Professor of cardiovascular medicine at the Wolfson Institute of Preventive Medicine at Queen Mary. So surely we can expect a rigorous and impartial analysis of the data from him.

If I'd been looking to prove or disprove this assertion, I'd have looked at stroke and heart disease rates in a range of patients over this time frame, where I had some objective measure of salt in their diet (urine samples), and looked to see whether patients with lower salt levels (in a group of patients with otherwise similar exercise, age, gender, racial stats) were correlated with lower stroke and heart disease rates. Is this what they did?

Patrick Wolfe, professor of statistics at University College London, took issue with the authors for assuming that the improved blood pressure seen in the 2003-2011 was largely the result of reduced salt intake. "Plausibility of assumption does not equal evidence," he said.
Oh. Apparently not, then. That's a piss-poor basis for the claims CASH (and international co-conspirator WASH) have been touting around about salt reduction. As commentor ID4968047 notes this reduction in strokes and heart disease could equally have come from the reduction in smoking in the past 10 years - the obligation is on Prof. MacGregor to show otherwise. Looking at CASH's writeup of the paper (the link to the paper isn't available yet, looks like) they say:
Confounding factors that were looked at include age, gender, ethnicity, education, incomes, alcohol consumption, fruit and vegetable intake and BMI.
Exercise and smoking are not mentioned. Nor do they reference the increase in statin use - and indeed Aseem Malhotra from Action on Sugar claims that statins are harmful and don't reduce mortality which is interesting as they seem to be a prime competitor to CASH/Action on Sugar's crusades against sugar and salt. Malhotra's claims got panned for lack of evidence by Prof. Rory Collins from Oxford.

It seems that others in the medical stats community have doubts too:

David Spiegelhalter, professor of the public understanding of risk at Cambridge university, cited the researchers' admission that the fall over that time in systolic blood pressure would be expected to reduce strokes by just 11% and heart attacks by 6%, small amounts of the total falls. [my emphasis] Reduced blood pressure did not represent the authors' claimed "substantial contribution" to the reduced death rates.
This is not to say that Graham MacGregor is obviously wrong in his claims. They might be true but it is a real reach to claim that this study supports them. And if this is the best he can do, I'd suggest the Marcela Trust / OMC Investments crowd who are backing CASH find someone with a better stats background to organise their crusade against salt and sugar.

Update: just managed to dig up the link to the full text in BMJ Open. From a quick look the focus was on linking salt reduction with BP reduction but not explicitly with stroke/CVD reduction.

The authors themselves admit:

It is likely that several factors, that is, the fall in BP, total cholesterol and smoking prevalence, the reduction in salt intake and the increase in the consumption of fruit and vegetables, along with improvements in the treatments of BP, cholesterol and CVD, contributed to the decrease in stroke and IHD mortality.
They have a stab at isolating the effect of salt by casting tea leaves:
it was estimated that a 2.7 mm Hg reduction in systolic BP that occurred with salt reduction would be predicted to reduce stroke by approximately 11% and IHD by 6%.
but even then the 2.7mmm Hg reduction figure they quote is the net over 8 years including factors such as decrease in smoking and increase in statins, so to attribute it to just salt reduction is "optimistic". They appeal to studies in Japan and Finland in the late 60's / early 70's but the huge gaps in time, diet and environment between now and then render the comparison unconvincing. If that's the best argument they've got to offer, I'd hate to see the ones that didn't get selected for use in the paper.

The conclusions are what kill the paper for me:

The reduction in salt intake is likely to be an important contributor to the falls in BP in England from 2003 to 2011. As a result, the decrease in salt intake would have played an important role in the reduction in stroke and IHD mortality during this period. [my emphasis]
That's a terribly weak conclusion even to my relatively untrained eyes. If they could state this more strongly, they would. Instead, they reserve their strength for polemic:
... the mean salt intake in England (8.1 g/day in 2011) was still 35% higher than the recommended level of 6 g/day, and 70% of the adult population (80% men and 58% women) had a daily salt intake above the recommended level.[14] Therefore, continuing and much greater efforts are needed to achieve further reductions in salt intake to prevent the maximum number of stroke and IHD deaths.
Reference 14 doesn't justify the 6g/day level, it's just a measurement of sodium levels. The authors don't make any reference I can see to why the recommended level should be 6g/day and not (say) 10g/day or 3g/day. If you're appealing to magic figures in your conclusion it doesn't give great confidence in the rest of your article.

2014-04-08

A lesson from OpenSSL

If you are paranoid about secrecy on the web, today's news about a bug in OpenSSL may make you feel justified. OpenSSL is an open source library that is used by companies, individuals and governments around the word to secure their systems. It's very widely used for two reasons: 1) a very useful set of licensing conditions that essentially say you're fine to use it as long as you credit the right authors in the source and 2) because so many commercial firms depend on it, its source has been scrutinised to death to spot both performance and functional bugs.

A one-paragraph primer on SSL (Secure Sockets Layer): it's the method by which a regular web browser and a secure web server communicate. You're using it whenever the address bar in your browser displays a URL starting with "https:" instead of "http" - so that's your online banking, Facebook, Google, Twitter, Amazon... Most of these secure web servers will be using OpenSSL - there are alternatives to OpenSSL but none of them are compellingly better, and in fact the widespread usage of OpenSSL probably makes it less likely to contain security bugs than the alternatives so there's safety in belonging to the herd.

Anyone who's thinking "aha, my company should avoid this problem by developing their own SSL implementation" or better yet "my company should develop a more secure protocol than SSL, and then implement that!" has not spent much time in the security space.

And yet, someone has just discovered a bug in a very widely used version of OpenSSL - and the bug is bad.

To get some perspective on how bad this is, the Heartbleed.com site has a nice summary:

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.
Sounds dire, no? Actually the above description is the worst case; the bug gives an attacker access to memory on the secure server that they shouldn't have, and that memory *might* contain secrets, but the attacker doesn't get to control which area of memory they can read. They'd have to make many queries to be likely to gain access to secrets, and it's not too hard to spot when one small area of the Internet has that kind of unusual access pattern to your server. Even if they make 1000 reads and get one secret, they still have to be able to recognise that the data they get back (which will look like white noise) has a secret somewhere in it. I don't want to downplay how serious the bug is - anyone running an OpenSSL server should upgrade it to get the fix as soon as humanly possible - but it's not the end of the world as long as you're paying attention to the potential of attacks on your servers.

Still, isn't this bug a massive indictment of the principle of Open Source (that you'll have fewer bugs than commercial alternatives)? It's appropriate here to quote Linus's Law, codified by Open Source advocate Eric Raymond and named after the founder of the Linux operating system Linus Torvalds:

"Given enough eyeballs, all bugs are shallow"
or more formally:
"Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone."
Unfortunately, the larger and more complex your codebase, the larger the tester and developer base has to be and the longer it takes to find problems...

It's tempting to look at this security alert and declare that Open Source has allowed a critical bug to creep into a key Internet infrastructure component (clearly true) and declare that this can't be the right approach for security. But you have to look at the alternatives: what if OpenSSL was instead ClosedSSL, a library sold at relatively low cost by respected security stalwart IBM? ClosedSSL wouldn't have public alerts like this; if IBM analysis found bugs in the implementation then they'd just make an incremental version release with the fix. But the bug would still be there and would not be any less exploitable for the lack of announcement. You'd have to assume that government agencies (foreign and domestic) would bust their guts to plant someone or something with access to the ClosedSSL team mail, and in parallel apply object code analysis to spot flaws. The flaw would not be much less exploitable for lack of publicity, and would likely be in the wild longer because IBM would never announce a flaw so vocally and so users would be more lax about upgrades.

There are then two lessons from OpenSSL: 1) that even Open Source inspection by motivated agencies can't prevent critical bugs from creeping into security software and 2) that no matter how bad the current situation is, it would be worse if the software was closed-source.

2014-04-03

Diversity in everything except opinion

This is terrifying. Mozilla CEO Brendan Eich has been forced out because of his personal position against gay marriage. If you don't believe me, read the mail from painfully hip lawyer and Mozilla Foundation chair Mitchell Baker:

We didn’t act like you’d expect Mozilla to act. We didn’t move fast enough to engage with people once the controversy started. We’re sorry. We must do better.
Brendan Eich has chosen [ah! free choice? really?] to step down from his role as CEO. He's made this decision for Mozilla and our community. [He's been given the choice to resign or be fired.]
Mozilla believes both in equality and freedom of speech. Equality is necessary for meaningful speech. And you need free speech to fight for equality. Figuring out how to stand for both at the same time can be hard.
Yes, it can be hard to stand for equality and freedom of speech, but clearly you've opted out of that stance and gone for sucking up to the media-approved line of thought. 60% of Americans support gay marriage (at least publicly) so it may be a majority opinion but opposition can't reasonably be dismissed as a small clique of bigots. It seems that opposing the majority opinion is only allowed when the majority opinion is "wrong". Let's remember that the California Supreme Court (no bastion of political orthodoxy) did not see anything wrong with allowing Californian voters to vote freely on whether marriage should be restricted to male-female partners in California.

I'm reminded of Bob Hope's quote after the 1975 Consenting Adult Sex Bill was passed:

I've just flown in from California, where they've made homosexuality legal. I thought I'd get out before they make it compulsory.
You're still allowed to choose to be heterosexual, but if you value your job and career you'd be a brave person to even hint at wavering on the issue of gay marriage.

Gay arch-blogger Andrew Sullivan can see where this is leading and he's really not convinced it's a good idea:

If this is the gay rights movement today – hounding our opponents with a fanaticism more like the religious right than anyone else – then count me out. If we are about intimidating the free speech of others, we are no better than the anti-gay bullies who came before us.

Allowing the media to dictate the acceptable positions on thought is not going to end well, and I wonder if the gay marriage lobby have really thought this one through. If Fox News continues its rise and the (more left-leaning) rest of the media continues its decline, do they want opposition to Fox editorial policies to become grounds for hate campaigns against people?

Obamacare is going to have a doctor problem

An aspect of the American Affordable Care Act (aka Obamacare) which I hadn't appreciated until this week was the degree to which it may be scuppered by primary care physicians (PCP, the USA equivalent of a GP). Chatting with a middle-aged friend who is moving to a small-ish American town and changing jobs, she opened my eyes to the mysteriously unpublicised problems which the implementation of the ACA is about to cause everyone.

She was visiting her new town last week, sorting out her house and meeting her new workmates in preparation for moving there for good in June. One item on her list was sorting out a new PCP. She had the details for her new insurance plan with one of the state providers, so dropped by a doctor's office near her house to register. No joy, that doctor wasn't accepting new patients. OK, so go back to the insurance provider website to identify a wider shortlist of providers in town.

No joy at the first doctor. Or the second. Or the third. This was getting ridiculous. She had a friend already working in her new workplace, so asked him if he could help. His doctor wasn't accepting new patients either, but the friend had a good relationship with the doctor, so asked him directly. Still no luck. Eventually my friend managed to find a large medical center open to new patients - over 60 miles away. She signed up, because that was the best offer there was.

There was one doctor in a nearby town open to new patients, but after a quick check on his reviews it became rapidly clear why. It sounds like he hadn't progressed much past the "trial by ordeal" approach to illness, and was still reading pre-Pasteur literature on infection control.

Why this insanity of unavailability? It turns out that the coming avalanche of patients signed up on ACA-compliant plans is not making doctors sleep well at night. Not only are these patients much more likely to be sick than their current patients, the main concern of practices is that they're going to lose money on treating these patients. The remuneration rate for doctors for ACA patients is - at least in some states, and I suspect all - based on 80% of the standard practice fees from 3 years ago. That's not great, but much worse is the experience with Medicare (federal medical coverage for the elderly). My friend used to work in a doctor's office, and they had a profitably employed office employee whose sole job was to push back against government agencies spuriously denying payment for Medicare claims. If there were any abnormalities at all in a claim, Medicare would deny it in the hope that only solidly valid claims would be retried. A denial is, after all, cheap.

I digress, but bear with me. Do you older readers remember when The Simpsons started and they introduced us to Marge's sisters Patty and Selma, stalwart misanthropic employees of the local Department of Motor Vehicles? There's a reason why the American viewers of the show laughed, and it wasn't to do with absurdity. There's no-one more un-fireable than a federal or state bureaucrat, and their attitude to their customers is exactly what you'd expect - entirely down to their general disposition to humanity. That's what the doctors' billing teams are going to be dealing with. Only by being better acquainted with the rulebook will they be able to get their due payments out of the ACA system, and even then those payments will be small and paid late.

Having a new government agency and new set of rules means that small practices are extremely worried about their costs and ability to maintain existing patients. Because (I believe, from chatting to the front desks of a few offices) you can't be selectively open to new patients based on their particular insurer, the easiest option for small providers is to batten down the hatches until the picture is clearer. Only the very large healthcare centers can take advantage of their economy of scale to accept the new stream of ACA-covered patients - and with them the standard employer-covered and easier to manage new patients.

I wonder whether this problem with finding doctors is behind the proposed rise in rates for ACA-compliant plans? Perhaps you will be able to look at the rate rise in the next few years and extrapolate the "marginal cost of doing business with government". As it stands, the best they can hope for is to break even:

Cigna, which is selling Obamacare plans in five states and is considering whether to expand that further in 2015, has said it won't make money on the business this year. It expects to have signed up as many as 100,000 new members under the program for this year.

So for the Affordable Care Act, if you're sick then you may be able to get coverage where you couldn't before, but sure as heck if you're outside a major city then you're going to find it a challenge to get a primary care physician to see you and get your care started. Now I'm wondering what the ER departments of ACA-plan-accepting hospitals are going to experience in the next year.

2014-03-18

When protecting minorities screws them over

Fascinating. I came across this Dave Winer story of tech hiring and firing in 1985's Silicon Valley via the money quote:

...every time a company hires someone who is not a young male, they run the risk that the new hire isn't there to work, rather is there to scam you.
since from that quote I wondered "what the hell planet is this author on?"

Then I read the story. And blow me down if I didn't end up at least partially agreeing with the author. Go read the whole thing.

Commenter sep332 clarifies that the problem described (an older tech worker using his age to file a discrimination claim after being fired) isn't actually about age, sex or anything in particular:

The laws about protected classes are not about classes of people. Anyone can claim that they were discriminated against for gender reasons, not just women. White people can claim they were targeted because of their race, etc. So the people who couldn't realistically claim discrimination are the people who are most like the rest of the company. [my emphasis] I mean, if women are the majority of your company, then it would be hard for a woman to claim gender discrimination.
In the tech sector, the majority of your company are likely to be young, male, and (in Silicon Valley) a mix of white, Chinese and Indian in race; probably also straight although I have known a couple of small firms where that was not the case. If you're middle-aged, female, transgender, black, Inuit, Pacific Islander or Hispanic then you're almost certainly in a clear minority and hence possibly a "protected class".

As Winer notes, in a small struggling tech firm if someone comes at you with a discrimination lawsuit then you haven't the money or time to fight it. Unless it's a complete no-brainer (an 18 year old white male alleging discrimination on the grounds of inability to get out of bed) your best option is to pay up and move on. So what do you do when you have the risk of recruiting people who can sit back and do nothing while being practically un-fireable? Simply minimise the risk of recruiting them, by avoiding anyone who is in a good position to do this to you.

An ironic "well done" to everyone who has pushed through these employment laws, and a special raspberry to everyone who has filed (or backed) an abusive lawsuit exploiting these laws. You've screwed over everyone in the tech sector who's not a young male.

2014-03-15

Storing in the cloud

So this is interesting. Google is dropping its cloud storage rates to $10 per month per TB (though 100GB costs $2, twice that rate). Amazon S3 storage is currently $85 per TB per month and Microsoft Azure is $64 per TB per month for their cheapest option (Locally Redundant Storage). I'd expect these prices to be dropping fairly soon in response to Google's move.

How much does this actually cost to provide? Let's look at the cost of storing and accessing 1 TB of data. An internal SATA 1TB hard drive costs about $60 on Amazon - but a 2TB costs $85, and a 4TB costs $160 (retail). So we can figure on about $40 per TB of storage. How long will this drive last? Mean time between failures of hard drives is between 18 months and 3 years depending on manufacturer and usage; let's split the difference and say 2 years. Buying 1TB of storage over 2 years will cost the supplier about $40 in capital costs. Isn't this a rip-off?

Well, having a hard drive is one thing - being able to access it is another. You've got to get data into that hard drive, and probably you want to get it out again. Assuming that the entire volume of that drive is written once and read twice per 2 years (probably a lowball estimate) at a rate of about 5 Mbits/s, that means that in 1 day (86400 seconds) you could read (86400 * 5 / 8) MB or about 54 GB per day so it would take up about 10 days per year per user, and so you could support about 36 users on a 5 Mbit connection. Let's say we're using 4 TB drives so you need a 5 Mbit connection for each 9 computers in your storage.

It's not quite that straight forward though. Cloud storage is supposed to be reliable, and hard drives are manifestly not - they die all the time. Therefore you want at least a second copy of your data on a separate hard drive, and ideally you want that second copy to be in at least a separate building in case of a physical disaster (flooding, fire, tornado). Generally the further away the better, at least up to 100 miles or so, though distance tends to increase the expense of hosting because for every write to the data you need to send the write to your remote facility as well. Azure gives you the explicit option of how physically separate you want your data to be; Locally Redundant Storage vs Geographically Redundant Storage.

There's also the distinction between data loss and data unavailability; if the primary copy of the user's data is unavailable (e.g. because the data center has a planned or unplanned availability outage) cloud providers may give their customers the option of reading data from (or writing data to) the backup copy of the data. Customers can buy this kind of read access from Azure as an additional option (Read Access GRS).

If you as the cloud storage operator rack up 3600 users, then, you'll need 900 computers with a total of 3600 TB of storage in each of 2 sites. You'll need 500 Mbit/s of bandwidth on each site if you want to offer read redundancy, and about 170 Mbit/s of bandwidth between sites to replicate writes. You don't need customised hardware for this amount of traffic, but you do need to buy the bandwidth to get the data to and from the user. Azure quotes $120 for egressing 1TB of data; if we estimate that it actually costs them $100 then each user will cost you $200 (reading their data twice), so you will have $720,000 of bandwidth cost.

If the computers last about as long as the hard drives you'd expect them to cost you about $300 plus the storage ($160), say about $500 once you take into account rack and network switch hardware. I suspect power won't be too much of an issue since storage isn't a CPU intensive operation and user access is intermittent - idling computers without a display consume about 20W. So each site will cost you 900 x $500 over two years, and consume a steady 18 KW of power. Electricity costs about $100 per MWh to make in the cheaper parts of the USA, so power will cost you $2 per hour per site. So power is only about 7% of your equipment costs for a 2 year lifetime, and you end up paying about $1.8M in total in hardware, power and bandwidth to provide 1TB of cloud storage to 3600 users over 2 years. Each user pays $240 over that time, or $860K in total. So it would seem that $10 per hour is a massively losing proposition for the provider even before we take into account the human costs of designing, building and operating the system.

The real picture is more nuanced. We implicitly assumed that every user would use all the storage (and bandwidth) they paid for. In practice, they could conceivably be consuming only half of what they've paid for. As long as we can dynamically provision for users (having a small amount of storage headroom, and adding on more computers and drives as that headroom is threatened) we could provide maybe 60% of the maximum hardware needed, so instead of $1.8M our costs would be down to $1M or so. Still something of a loss.

I think the way cloud companies can make money - or at least avoid a loss - on this is to make use of the fact that the computers providing access to the drives are seldom even slightly busy. Instead of buying $300 of low power computing hardware to support each 4TB drive, just chain a few 4TB drives onto an existing computer that you're using for something else (say, Bing search, Google maps, Amazon website serving). When a user starts to access their data, temporarily reserve a core or two on the machines holding that data to serve it. That way you save nearly 60% of your hardware costs and might just be bringing your operation into slight profit.

You'll still need to pay the design and implementation costs for your system, not to mention the usual marketing and business operations, but these don't scale in proportion to your number of users. The more users you have, the better your business looks.

$10 per month per TB is a bit of a game changer. Suddenly storing in the cloud isn't massively more expensive than storing on your hard drive. I wonder what the next couple of years will bring?

2014-03-13

Piling on Piers Morgan

I was initially surprised that the news of Piers Morgan's fall from grace resulting in the cancellation of his CNN prime time talk show had been news in the UK, but I guess the prospect of Morgan returning to Blightly was sufficiently generally appalling that the Brits were quite concerned about the prospect.

Yesterday Piers had comedienne Chelsea Handler on his show. Noted for brutal honesty, such as discussing personal abortion and DUI stories, Chelsea didn't disappoint but possibly not in the way Piers expected. The discussion following the show's commercial break was quite enlightening:

Chelsea: I mean, in the middle of the commercial break – I want your viewers to know; they must know, because they're probably following you on Twitter. I mean you can't even pay attention for 60 seconds. You're a terrible interviewer.
Piers: Well, you just weren't keeping my attention. It's more of an issue with you than me.
Chelsea: That's not my problem. This is your show, you have to pay attention to the guest that you invited on your show.
Piers: If they're interesting enough ...[cut off]
Chelsea: Listen, it doesn't matter how interesting I am. You signed up for this job. [..] Well, maybe that's why your job is coming to an end.
Piers has - or had - the 9pm-10pm slot on CNN, which is prime time. (West Coast viewers, 3 hours behind the East Coast times quoted, can usually see the show live at 6pm Pacific or repeated 3 hours later). There's a constant battle for viewers between CNN and Fox in the evening, and Fox announced in August that newscaster Megyn [yes, really] Kelly would be taking over the Fox 9pm slot from previous incumbent Sean Hannity. Piers seemed keen on the challenge, tweeting "Bring it on, Megyn Kelly". Kelly duly brought it on, starting in early October and two months later was beating Piers 5 to 1 in viewers, up 10%-20% from Hannity.

Why did CNN viewers turn away from Piers in droves? Kelly is clearly easier on the eyes than Piers, but that can't be the whole story. I think Tim Stanley in the Telegraph gets closest to it:

But he acted as though no one had ever thought to discuss the subject[gun control] before. Like, ever. He tried to make gun control his own personal crusade, to "school" the Americans on law and order. And he displayed a crass insensitivity towards issues such as the importance of the Constitution or the American tradition of self-reliance. The scale of his ego was extraordinary. No US liberal has ever managed to challenge their country's fundamental respect for gun ownership. Why did he imagine that a guy with an English accent – the accent of George III no less – would succeed where Bill Clinton, Teddy Kennedy or Barack Obama had failed?
[...]
The embarrassing thing about the whole Piers Morgan affair is that it has turned the tables on us Brits. We always insisted that we were the courteous ones and the Americans were the boobs. In this case, it's been the other way around.
Americans really don't like being told, imperiously, what to do. Piers did this all the time, and worse wouldn't admit when he'd been beaten. He invited young Republican Ben Shapiro to debate guns with him after the Sandy Hook shooting, and had his clock unexpectedly cleaned - Shapiro tried to load the game by going for a "town hall" format which would have stuffed the audience with shooting victims and Shapiro wisely refused, telling him 1-on-1 or nothing. So, nothing.

Looking at interview styles, probably the best comparison on Fox with Piers is Bill O'Reilly who hosts the 8pm-9pm slot. O'Reilly has a similar format, inviting guests to discuss contentious issues 1-on-1, or sometimes 2-on-1, with him. One key difference, I think, is that O'Reilly doesn't try to lay traps - he relies on his (formidable) preparation and debating skills. Unlike Piers he has regular guests from a range of political views - liberal Juan Williams is one of the favourites - and the debate can get quite shouty at times but the guests generally know O'Reilly, know the position he'll take, and have wrestled with him often enough to give a good showing. More important, O'Reilly doesn't belittle them. He tries to reason, and even if you don't agree with his (often reactionary) position you can see that he's playing the man rather than the ball. Piers by contrast is a classic bully, trying to belittle opponents.

CNN's Anderson Cooper is a marked contrast to Piers; I've seen Anderson do interviews, and he can be tough - Greg Smith who resigned publicly from Goldman Sachs did an Anderson Cooper interview, and Cooper didn't go easy on him at all - but he's fair. You feel as if he's trying to make the interviewee explain what he's hiding from the audience, rather than browbeating the interviewee from a position of power just to establish a pecking order like a bully does. It would be hard to overstate how much most Americans look down on bullies. The War of Independence was essentially a reaction against perceived bullying (imposition by means of might) of the American colony by King George III. Piers is a classic bully, and Americans simply don't like that kind of person.

Interestingly it seems that even his own staff weren't keen on him:

"The makeup girls suffered the worst — he was rude and belligerent," says our source. "The general feeling is Morgan didn't show any respect to anyone working under him — the people who were trying to make him look good."
It would be uncharitable to note how hard a job that is. But this is Piers Morgan, so screw charity.