2016-03-23

Hillary Clinton to give counter-terrorism speech at Stanford University

"Do as I say, not as I do!" Hillary Clinton will urge listeners to stand in solidarity with Europe in order to defeat the Islamic State group:

"Today's attacks will only strengthen our resolve to stand together as allies and defeat terrorism and radical jihadism around the world."
Heaven forfend that we strenuously deny any connection between organized terrorism and this event and instead speciously blame random YouTube videos for incitement.

Hillary certainly has balls. She probably took Bill's in exchange for keeping quiet about his dalliances...

2016-02-29

Trump's Republican problem

Long-time readers (both of them) will know of my affection for "my favorite wonk", Megan McArdle. She has been canvassing on Twitter for information about where Donald Trump support is coming from and where it won't ever come from, and has just published a great list of anonymous quotes from lifelong Republicans who won't vote Trump even if he's the Republican candidate:

  • I've always voted Republican [...] I have generally avoided voting third-party for fear of helping the Democratic candidate win. However, if Trump wins the nomination, I will vote for the winner of the Libertarian party nomination. I will not support Trump under any circumstances.
  • [I will] stay home or not vote for President if Trump is the Republican nominee. After voting basically a straight ticket Republican since I have been eligible to vote, this is truly amazing.
  • I have never voted for a Democrat before, but I care too much about the future of this country to let a blithering imbecile become the President.
  • I hate Hillary Clinton, but at the very least I know she will do what [she thinks] is best for this country. I cannot say the same about Trump.
I'd say "read the whole thing", but be aware that you'll be there a while.

I really can't imagine many Democrats voting for (say) Republican Ted Cruz if the Democrat candidate was either Sanders or Clinton, no matter what they though of the Democrat. I wonder if this will turn out to be the most compelling reason for the Republicans to band together and stop Trump - not so much to stop him being President, but to stop him being such a horrible Republican candidate that he would keep Republican supporters at home and let in Clinton or Sanders.

2016-02-20

Analysing the blue-red hat problem in the face of user error

Everyone knows computers are getting smarter - unless they're being programmed by a major corporation for a government contract - but there has recently been another leap in the level of smart. DeepMind (now part of Google) has built an AI that has successfully deduced the optimal solution to the hat problem:

100 prisoners stand in line, one in front of the other. Each wears either a red hat or a blue hat. Every prisoner can see the hats of the people in front – but not their own hat, or the hats worn by anyone behind. Starting at the back of the line, a prison guard asks each prisoner the colour of their hat. If they answer correctly, they will be pardoned [and if not, executed]. Before lining up, the prisoners confer on a strategy to help them. What should they do?
Tricky, n'est ce pas?

The obvious part first: the first prisoner to answer, whom we'll designate number 1, has no information about his hat colour. Assuming blue and red hats are assigned with equal probability, he can answer either "red" or "blue" with a 50% chance of success and 50% chance of getting executed; he has no better strategy for self-survival. What about the other prisoners?

Applying information theory, our system has 100 binary bits of state - 100 people, each with 1 bit of state relating to whether their hat is blue or not. We generate 99 bits of knowledge about that state as the hat-wearers give answers. So the maximum we can expect to discover reliably is 99/100 hat values. How can we get close to this?

If everyone just guesses their own hat colour randomly, or everyone says "blue", or everyone says "red", then on average 50% of people survive. How to do better? We need to communicate information to people further down their line about their hat colour.

Let's get the first 50 people in line to tell the next 50 people in line about their hat colour. Person 1 announces the hat colour of person 51, person 2 of person 52 and so on. So the last 50 people are guaranteed to survive because they have been told their hat colour. The first 50 people each have a 50-50 chance of survival because the colour they "guess" has no necessary relation to the colour of their hat. On average 25 of them survive, giving an average survival of 75% of people.

The DeepMind algorithm relies on an insight based on the concept of parity: an 0/1 value encapsulating critical state, in this case the number of blue hats seen and guessed, modulo 2. The first user counts the number of blue hats seen and says "blue" if that number is even, and "red" if odd. He still has a 50-50 chance of survival because he has no information about his hat. The second user counts the number of blue hats. If even, and the first person said "blue", then he and the first person both saw the same number of blue hats - so his hat must be red. If even, and the first person said "red", his hat must be blue because it changed the number of blue hats seen between the first person and him. Similar reasoning on the odd case means that he can announce his hat colour with full confidence.

What about person 3? He has to listen to person 1 and person 2, and observe the hat colours in front of him, to deduce whether his hat is blue; his strategy, which works for all others after him too, is to add the parity values (0 for blue, 1 for red) for heard and seen hats modulo 2, and if 0 then announce "blue", if 1 then announce "red". Follow this down the line, and persons 2 through 100 are guaranteed survival while person 1 has a 50-50 chance, for an average 99.5% survival rate.

Of course, this is a fairly complicated algorithm. What if someone mis-counts - what effect does it have? We don't want a fragile algorithm where one person's error can mess up everyone else's calculations, such as with "Chinese whispers." Luckily, a bit of thought (confirmed by experiment) shows us that both the future-casting and parity approaches are resilient to individual error. For future-casting, if one of the first 50 people makes an error then it makes no difference to their chance of survival but their correspondent in the second half of the line is doomed. If one of the second 50 people makes an error then they are doomed unless their correspondent also makes a mistake - generally unlikely, a 10% chance. So if 10% of users make errors then the approximate number of survivors is (75 - 10) + 1, i.e. 66%.

Surprisingly, the parity approach is also robust. It turns out that if user N makes a mistake then they doom themselves, and also doom user N+1 who relies on user N's calculation. But because both user N and N+1 make erroneous guesses, this brings the parity value back in line for user N+2, whose guess will be correct (absent any other errors). So the approximate number of survivors given a 10% error rate is 99.5 - 10*2 = 79.5%

Here's Python code to test the various algorithms: save it as "hats.py" and run it (e.g. "chmod 0755 hats.py ; ./hats.py" on OS X or Linux). It runs 10 trials of 100 people and reports the average number of survivors, based on a 10% error rate in hat wearers following their strategy. Default strategy is the parity algorithm.

#!/usr/bin/python

import random

person_count = 100
half_person_count = person_count / 2
# Hat choices
hat_choices = ['r','b']
hat_opposite = {'b':'r', 'r':'b'}
# 10% error rate in guesses
error_rate = 0.1

def guess_constant(heard_guesses, seen_hats):
    return 'b'

def guess_random(heard_guesses, seen_hats):
    return random.choice(hat_choices)

def guess_future(heard_guesses, seen_hats):
    """ First half of list calls out hat of correspondent in second half of list """
    full_list = heard_guesses + ['x'] + seen_hats
    my_index = len(heard_guesses)
    if my_index < half_person_count:
        # Call out the hat of the person in the second half of the list, hope same as mine
        return full_list[my_index+half_person_count]
    else:
        # Remember what was called out by my corresponding person in first half of list
        return heard_guesses[my_index - half_person_count]

def guess_parity(heard_guesses, seen_hats):
    """ Measure heard and seen parity of blue hats, call out blue for even, red for odd."""
    heard_blue_count = len([g for g in heard_guesses if g == 'b'])
    seen_blue_count = len([s for s in seen_hats if s == 'b'])
    if (heard_blue_count + seen_blue_count) % 2 == 0:
        return 'b'
    else:
        return 'r'

def run_test(guess_fun):
    hat_list = [ random.choice(hat_choices) for i in range(0, person_count) ]
    print "Actual: " + "".join(hat_list)
    answer_list = []
    score_list = []
    error_list = []
    correct = 0
    for i in range(0, person_count):
        guess = guess_fun(answer_list, hat_list[i+1:])
        if random.random() < error_rate:
            guess = hat_opposite[guess]
            error_list.append('X')
        else:
            error_list.append('-')
        answer_list.append(guess)
        if guess == hat_list[i]:
            correct += 1
            score_list.append('-')
        else:
            score_list.append('X')
    print "Called: " + "".join(answer_list)
    print "Score:  " + "".join(score_list)
    print "Errors: " + "".join(error_list)
    print "%d correct" % correct
    return correct

if __name__ == "__main__":
    trial_count = 10
    correct_total = 0
    for i in range(0, trial_count):
        print "\nTrial %d" % (i+1)
        correct_total += run_test(guess_parity)
    print "\nAverage correct: %d" % (correct_total / trial_count)
You can change the "guess_parity" value in the run_test() invocation on the penultimate line to "guess_future" for the "warn the second half" strategy, or "guess_random" for the random choice.

This is a lousy problem for use in software engineering job interviews, by the way. It's a famous problem, so candidates who have heard it are at a major advantage to those who haven't. It relies on a key and non-obvious insight. A candidate who hasn't encountered the problem before and solves it gives a very strong "hire" signal, but a candidate who fails to find the optimal solution could still be a valid hire. The least worst way to assess candidates based on this problem is whether they can write code to evaluate these algorithms, once the algorithms are described to them.

2016-01-20

Putting Twitter's loss in perspective

Today, Twitter (NYSE symbol TWTR) lost 7% of its value to close at $16.69/share at a market cap of $11.4bn. That's a loss of approximately $800m of of share capital.

To put that in perspective, that's 8M $100 bills. The NYSE (New York Stock Exchange) is open from 9:30am to 4pm; 6.5 hours, or 23,400 seconds. A well-tuned toilet flush cycle is 35 seconds, so you could get in 668 back-to-back flushes during NYSE opening hours. Therefore you'd have to flush 12,000 $100 bills each time in order to match TWTR's loss. At 150 bills/stack that's 80 stacks, and I can't see you getting more than 1 stack per flush in a single toilet, so I would characterise today's loss as a rate of 80 NYSE-toilets.

I hesitate to ascribe all this loss to Twitter's de-verification of arch-gay-conservative @Nero on 9th January when Twitter was $20, but its share price has descended in more or less a straight line since then. Today the NYSE actually went very slightly up but Twitter still plummeted.

It certainly wasn't helped by 6 hours of partial unavailability of Twitter today, but I suspect that was the straw breaking the camel's back.

2015-12-30

How to build the perfect man bun

Note: this is not to be confused with how to build the perfect man buns, which is an entirely separate topic and not one I will be addressing any time soon in this blog.

  1. Grow your hair out to at least 10 inches in length. This may take you a while. Any length above this is also fine.
  2. Wash your hair. Yes, this really does matter. Any shampoo will do, but do not use conditioner or a shampoo+conditioner combined product - you don't want smooth conditioned hair.
  3. Blow dry hair, thoroughly. Comb out so that it is moderately straight. Don't add any styling product.
  4. Obtain a pair of hairdressing scissors, either new or well sharpened.
  5. Visit any workplace with large rotating machinery.
  6. Holding the scissors firmly, carefully lower the end of your hair into the rotating machinery. Once it obtains a grip, frantically shear away at your trapped hair with the scissors until your head is free.

The net effect should be a lifelong aversion to having long hair at all, let alone a man bun.

2015-11-24

ISIS and the matriarchy

Dilbert cartoonist Scott Adams has stuck his oar into a conjunction of the current fiery discussions around a) the treatment of women in the workplace and b) how to handle ISIS with a blog article titled, discretely, Global Gender War:

Now compare our matriarchy (that we pretend is a patriarchy) with the situation in DAESH[ISIS]-held territory. That’s what a male-dominated society looks like. It isn’t pretty. The top-ranked men have multiple wives and the low-ranked men either have no access to women, or they have sex with captured slaves.

There's no way this could possibly be controversial, amirite?

Adams has started a new blog series today on the question "Is the United States a Patriarchy or a Matriarchy?" where his evidence for "Matriarchy" is moderately compelling so far: "Women have the most political power in the United States because more women than men vote". True, we haven't had a female President yet, but it's not obvious that this is because they're being discriminated against. Personally, I'm astonished that Hillary Clinton has got as far as she has given all the shenanigans she has been involved in, which would have torpedoed any other candidate's aspirations before starting, and how astonishingly un-gregarious and un-likeable she is.

Anybody remember Democrat hopeful Gary Hart in the 1988 campaign? Sunk without trace by an accusation of marital infidelity. Michael Dukakis who eventually became the nominee? Lost mostly because of an insufficiently warm personality. Hillary should have sunk without trace by now, and the fact that she's still the Democrat front-runner by far is a sign of how much leeway the population - and the media - is giving her. One can only surmise that it's because she is a woman.

Returning to Adams' original article, he points out that the gender pay gap in the USA doesn't really exist - as the estimable Tim Worstall has pointed out time and time again with respect to the UK, which (if anything) is more traditional than the USA. He addresses the assertion that women are interrupted more in meetings by pointing out that people who talk more are more likely to be interrupted, and while this is not a slam-dunk answer it's at least a point that suggests a need for more analysis. He also points out the strong societal push to give sanctuary to women and children from Syria, as opposed to young single men.

It's quite possible that Adams is completely wrong and that women are systematically discriminated against in the USA, but it's not obviously false.

Anyway, the point of the article is to contrast the USA against the vast majority of Middle East states which are indisputable patriarchies - really, would anyone like to argue the opposite? - and to hypothesize that one way that the Daesh/ISIS leadership are controlling their low-level followers is to restrict their access to good nookie on Earth with the promise of wonderful nookie in Heaven if they blow themselves up in the right place, with appropriate Koranic citations to back this up. Again, Adams' thesis is not obviously wrong. These men seem to have significant "issues" with women and something is motivating them to suicide, while the harems for their leadership are a matter of record.

If you're in any doubt about the position of Western women in the Middle East, read the travel guidance for women in Saudi Arabia:

- Women traveling alone are not allowed to enter the country unless they will be met at the airport by a husband, a sponsor or male relative.
- Women relocating to Saudi Arabia to marry, study or stay with a Saudi family need to be aware that leaving the country requires the permission of the Saudi male head of their household.

So if the hormone-crazed late-teen ISIS recruits can't get access to women because of the restrictions that their leadership imposes, Adams' assertion that he as a teenager in the same position would gladly strap on explosives to get access to the forbidden fruit is not obviously insane. Deliberately provocative, yes. But can those piling on Adams provide a more plausible explanation of the current suicide bombers' motivation?

2015-10-15

The logistics of de-immigration

Eminent social justice activist Shaun King raises a pertinent point on the current topic (in the sphere of the US presidential candidate selection process) of what to do with the "immigrants of dubious legality" in the USA:

This is, as several people has observed, quite a hard problem.

The first problem you have is finding the immigrants, and this is probably the killer. You've got 360M people in the USA, illegal immigrants are 10M-20M in number by various estimates, so for every 1000 illegals found you have to trawl (naively) about 20,000 legal citizens - and at 450K illegals/month constant rate you're looking at 2 years to remove nearly everyone. So every month you need to annoy 9M legal residents at some level in order to meet your quota. As immigrant numbers fall, that number of recently annoyed legal residents will rise. You'll start with unobtrusive measures, but as time goes on you'll need to get more and more intrusive - and most of the annoyed legal residents are citizens, and can vote against representatives who are supporting this measure.

Then you need to make them leave the country. Detention is expensive, ask anybody in the Federal Bureau of Prisons - average is about $100/day and that assumes amortizing entry and exit costs over many months. The sooner you can export them, the better. You need to fund daily 1-way flights from a wide range of cities to the dominant countries of residence of illegal immigrants: Mexico (obviously), major nations in Central/South America, and Pakistan/India/Bangladesh. The immigrants won't be paying for this - they'd rather pass their US$ to legal resident friends and rely on that largesse being transmitted to their home country for later pick-up, at a generous margin. So the US government will be implicitly boosting illegal financial transactions as a result. Occupancy rate on those planes is going to be highly variable. Assuming average occupancy of an evacuation plane at 50% - realistically, you can't fill them with paying passengers, ask anyone in the UK - that $700 is a reasonable round trip fare to Latin America, and noting that the return journey will need to be empty (don't even think about eating the profit margins of existing airlines, there's no way this turns out well) you're spending about $700M/month just on the export. This assumes zero cost on detention and transport to the airport, which is "optimistic".

What's the end run around this? Make the illegals deport themselves. Illegal immigrants come to the USA to work and earn money for their family, with the (faint) hope that they can eventually stay. This might occur by having a baby in the USA who will be a US citizen, and applying for residency on compassionate grounds; alternatively they might eventually find an employer willing to sponsor them. So remove that attraction. There are definite areas of employment for illegal immigrants; it depends on the region, but generally agriculture (crop picking), daily manual labour and domestic service are the top areas. Focus tax audits on those areas, reduce the marginal cost of legal labor (e.g. by increasing the deductability of costs associated with a provably legal labourer) and watch the illegal employment rate plummet.

This isn't a free ride - the government will still need to fund the free no-questions-asked one-way flights home, but if they really want to make this happen then it's probably the cheapest way to achieve the goal. With no income, and easy access to return journeys to one's home country, the labour problem will mostly fix itself.

Of course, this reduces the government benefit of illegal employment - is an incumbent administration willing to forego all the income from illegal activity?

2015-10-11

Go easy on the tea, Lewis

Mercedes F1 racing top talent Lewis Hamilton may be well advised to steer clear of cups of tea for a while after soaking Vlad Putin's suit in champagne.

It's possible that Putin will take this in the humourous way it was intended, but if I were Lewis I'd be looking around for a watch with a built-in Geiger counter.

2015-10-08

The Silicon Valley Diversity Shitstorm

Talk about putting the cat among the pigeons. Journo Brian S. Hall wrote a short article about diversity in Silicon Valley: specifically "There Is No Diversity Crisis In Silicon Valley". It turns out to have been slightly controversial, as you can tell when you visit the original Forbes post:

The piece previously at this URL, titled "There Is No Diversity Crisis In Silicon Valley," published on 10/5/2015 [5th October 2015 for anyone using a sane date format], was deemed to have violated our Terms of Service and was removed.
Well, that's odd. What did it say? What could have violated the ToS?

Luckily, we can now read the original article on Brian's own site. An excerpt:

Silicon Valley doesn't just create greatness, it's probably the most open, welcoming, meritocratic-based region on the planet. Anecdotal evidence strongly suggests that disproportionately more Chinese, Indians, and LGBQT succeed in Silicon Valley than just about any place in America. Guess what? Everyone earned their job because of their big brains and ability to contribute.
The piece, to be fair, was rather heavy on anecdote. However in the ensuing Twitter blitzkreig there was some actual data posted including an illustrative stat on student study and degree achievements in STEM subjects which showed pretty clearly that white, Hispanic and black enroll in STEM programmes at about the same rate, but that the white students are disproportionately more successful in actually obtaining a degree. Asian students - of course! - enroll at twice the rate and obtain a degree disproportionately more often. So if you view a STEM degree as somewhat important in a Silicon Valley career then it's not entirely surprising that the ethnic makeup in SV correlates with those stats.

Anecdotally, Hall's assertions on race and LGBTQ seem about right to me, though I think he's missed a few letters off the latter term. The native Californians and other white Americans are distinctly in the numerical minority, and even obvious LGBTQ engineers are relatively plentiful. I also liked Hall's dig at the humanities as a contrast to "computer programming, engineering, chemistry — hard subjects that demand hard work", remembering the geography and history students lounging around after a couple of Finals exams early in the last semester after putting in a few strenuous 10-12 hour weeks, while the maths, physics and engineering students were still sweating away with 50 hour weeks revising for a series of painfully hard and objectively marked exams right at the end of the semester.

I'd probably take some issue with Hall's assertion that "Everyone earned their job because of their big brains and ability to contribute" - the latter is more aspirational than fact, SV hiring like anywhere else still has problems trying to determine whether someone who's obviously smart can actually be productive, and screws up that assessment reasonably frequently, but the basic idea is there. A SV company that does any discrimination other than by ability to do the job is going to shut off some of its source of talent, and in a hugely competitive hiring market that's a pretty dumb play.

Hall subsequently doubled down with the tweet

It's worth noting that Google CEO Pichai came from a poor family in India and rose based on brutal merit. Though he had problems with the humanities in school:
He was a brilliant student but his geography and history used to let him down so he was never top of the year.
That's probably what annoyed all the humanities graduates who piled onto Hall on Twitter... Although it's a little tricky to argue for a glass ceiling for minorities in SV given the above facts, it didn't seem to stop a lot of people from trying.

SV still has recruiting and retention problems, and I'd call out the experience of women in particular - the tendency of male engineers to act like baboons is off-putting to any women engineers who want to be something other than male engineers with a slightly different placement of genitals. But I don't find anything particularly jarring, scandalous or untrue in Hall's piece, so I wonder why exactly Forbes decided to withdraw it under pressure. If it wasn't pulled because of falsehoods, was it pulled because it was too true?

2015-09-04

Some illegals more equal than others - California edition

In a conversation at work today, a colleague mentioned that her Iceland-born spouse needed someone to go with him to the local branch of the California Department of Motor Vehicles (aka the First Circle of Hell) because he had to take a test. There was widespread surprise at this - didn't he have a valid licence from another country, and wasn't this OK? Yes he did, and no it wasn't; as of 15th May, the California DMV will no longer issue temporary driving licences when you pass their written test.

For context on why this matters: for foreign citizens, when you move to California and become resident (paying rent / utility bills locally) you're required to get a driving licence within 10 days of this event if you want to continue driving in California. Up until May, this was straight forward: you went to the DMV, took their written tests - tedious but not too hard - then booked a practical test and in return got a temporary driving licence that you could renew if the test got postponed. The practical test took 1-3 weeks to reserve a reasonable slot until recently, but this year's announcement that certain immigrants didn't have to prove any legal residence status has caused a huge rush of applications and backlog of tests.

Now that foreign citizens don't get the temporary licence, they can't drive unaccompanied from day 11 of their residency until the date that they pass the (admittedly easy) driving test. Sounds like a bit of a regression, so what's going on?

Let's look at the requirements for California DMV form AB60 guidelines on proving identity if you're not already a Californian:

Foreign Document that is valid, approved by the Department of Motor Vehicles (DMV) and electronically verified by DMV with the country of origin:
  • Mexican Federal Electoral Card (Instituto Federal Electoral (IFE) Credencial para Votar – 2013 version)
  • Mexican Passport (issued in 2008 or later and includes digital photo and digital signature)
  • Mexican Consular Card (Matricula Consular – 2006 and 2014 versions)
  • Foreign Passport that is valid and approved by DMV (see page 4 & 5 for list of DMV approved passports). The customer must also provide his/her social security number (SSN) that is electronically verifiable with the Social Security Administration.

Well, that's tough luck if you're an illegal immigrant (i.e. not able to get a legit Social Security number because you're not a legal resident) and not Mexican, right? Luckily there's an alternative if you have a foreign passport but not an SSN: if you have one of the following then you're OK:
  • Argentinian Identification Card (Documento Nacional de Identidad (DNI) – 2009 or 2012 version)
  • Brazilian Consular Card (Carteira de Matricula Consular – 2010 version)
  • Chilean Identification Card (Cedula de Identidad – 2013 version)
  • Colombian Consular Card (Consular Registration – 2015 version)
  • Ecuadorian National Identification Card (Cedula de Ciudadania – 2006 or 2009 version)
  • Ecuadorian Consular Card (Tarjeta De Identification Consular – 2015 version)
  • El Salvadorian Identification Card (Documento Unico de Identidad (DUI) – 2010 version)
  • Guatemalan National Identification Card (Documento Personal de Identificacion (DPI) – 2012 version)
  • Guatemalan Consular Card (Tarjeta de Identificacion Consular – 2002 version)
  • Peruvian Identification Card (Documento Nacional de Identidad (DNI) – 2005 version)
Or you can show another foreign passport: so if you're a dual national then by my reading, you're sorted. Other than that, if you're not from Central/South America and don't have legal residence then you're pretty much sunk. Yay for the major South American nations, except Venezuela or Uruguay, but boo for anyone else.

To recap: if you're an illegal immigrant then you don't really care about driving illegally in the short term. But long term it could be a problem, which is why California has the above AB60 guidance about handing out driving licenses. If you're from Central/South America then they have you covered, otherwise they really don't seem to care. It's perfectly fine for a country to be antagonistic to illegal aliens (that's me struck off Shahid Haque-Hausrath's Christmas card list) but to be arbitrarily receptive to citizens of some countries and not others smacks of, oh I don't know, naked political favouritism?

And now legal immigrants will find it substantially harder to comply with the laws of the state that they're living in - and paying taxes to. Nice one, California.