# Statistical Significance Explained

If you did­n’t read the House of Com­mons Lib­rar­y’s stat­ist­ic­al lit­er­acy guides recently (or you need a refresh­er on what, exactly, stat­ist­ic­al sig­ni­fic­ance means), then you can do much worse than stu­dent War­ren Dav­ies’ short run­down on the mean­ing of stat­ist­ic­al sig­ni­fic­ance:

In sci­ence we’re always test­ing hypo­theses. We nev­er con­duct a study to ‘see what hap­pens’, because there’s always at least one way to make any use­less set of data look import­ant. We take a risk; we put our idea on the line and expose it to poten­tial refut­a­tion. There­fore, all stat­ist­ic­al tests in psy­cho­logy test the prob­ab­il­ity of obtain­ing your giv­en set of res­ults (and all those that are even more extreme) if the hypo­thes­is were incor­rect – i.e. the null hypo­thes­is were true. [â€¦]

This is what stat­ist­ic­al sig­ni­fic­ance test­ing tells you – the prob­ab­il­ity that the res­ult (and all those that are even more extreme) would have come about if the null hypo­thes­is were true. [â€¦] It’s giv­en as a value between 0 and 1, and labelled p. So p = .01 means a 1% chance of get­ting the res­ults if the null hypo­thes­is were true; p = .5 means 50% chance, p = .99 means 99%, and so on.

In psy­cho­logy we usu­ally look for p val­ues lower than .05, or 5%. That’s what you should look out for when read­ing journ­al papers. If there’s less than a 5% chance of get­ting the res­ult if the null hypo­thes­is were true, a psy­cho­lo­gist will be happy with that, and the res­ult is more likely to get pub­lished.

Sig­ni­fic­ance test­ing is not per­fect, though. Remem­ber this: ‘Stat­ist­ic­al sig­ni­fic­ance is not psy­cho­lo­gic­al sig­ni­fic­ance.’ You must look at oth­er things too; the effect size, the power, the the­or­et­ic­al under­pin­nings. Com­bined, they tell a story about how import­ant the res­ults are, and with time you’ll get bet­ter and bet­ter at inter­pret­ing this story.

To get a real feel for this, Dav­ies provides a simple-to-fol­low example (a loaded die) in the post.

via @sandygautam

# Political Risk Assessments

“Safety is nev­er allowed to trump all oth­er con­cerns”, says Juli­an Bag­gini, and without say­ing as much gov­ern­ments must con­sist­ently put a price on lives and determ­ine how much risk to expose the pub­lic to.

In anÂ art­icleÂ for the BBC, Bag­gini takes a com­pre­hens­ive look at how gov­ern­ments make risk assess­ments and in the pro­cess dis­cusses a top­ic of con­stant intrigue for me: how much a human life is val­ued by dif­fer­ent gov­ern­ments and their depart­ments.

The eth­ics of risk is not as straight­for­ward as the rhet­or­ic of “para­mount import­ance” sug­gests. People talk of the “pre­cau­tion­ary prin­ciple” or “erring on the side of cau­tion” but gov­ern­ments are always trad­ing safety for con­veni­ence or oth­er gains. [â€¦]

Gov­ern­ments have to choose on our behalf which risks we should be exposed to.

That poses a dif­fi­cult eth­ic­al dilemma: should gov­ern­ment decisions about risk reflect the often irra­tion­al foibles of the popu­lace or the ration­al cal­cu­la­tions of sober risk assess­ment? Should our politi­cians opt for informed pater­nal­ism or respect for irra­tion­al pref­er­ences? [â€¦]

In prac­tice, gov­ern­ments do not make fully ration­al risk assess­ments. Their cal­cu­la­tions are based partly on cost-bene­fit ana­lyses, and partly on what the pub­lic will tol­er­ate.

# Statistical Literacy Guides

I am suit­ably impressed by the clar­ity and breadthÂ of the House of Com­mons Lib­rar­y’s stat­ist­ic­al lit­er­acy guide on How to spot spin and inap­pro­pri­ate use of stat­ist­ics (pdf, viaÂ @TimHarford).

A quick dig around the archives revealed a full series of stat­ist­ic­al lit­er­acy guides (all pdf), all of which are fant­ast­ic­ally read­able, access­ible and com­pre­hens­ive. These are must-read guides to what some people feel are com­plex, seem­ingly-mono­lith­ic­al sub­jects:

# The Evidence on Breastfeeding

In an art­icle the Roy­al Stat­ist­ic­al Soci­ety announced as the run­ner-up in their annu­al Awards for Stat­ist­ic­al Excel­lence in Journ­al­ism, Helen Rum­below thor­oughly invest­ig­ates the well-debated sub­ject of breast­feed­ing.

The con­clu­sion of the piece is that much of the evid­ence in sup­port of breast­feed­ing is massively mis­rep­res­en­ted or inher­ently flawed.

“The evid­ence to date sug­gests it prob­ably does­n’t make much dif­fer­ence if you breast­feed.” [â€¦]

“The con­clu­sion is that the evid­ence we have now is not com­pel­ling. It cer­tainly does not jus­ti­fy the rhet­or­ic,” [Amer­ic­an aca­dem­ic Joan Wolf] says. The prob­lem with the stud­ies is that it is very hard to sep­ar­ate the bene­fits of the mother­’s milk from the bene­fits of the kind of moth­er who chooses to breast­feed. In the UK, for example, the highest class of women are 60 per cent more likely to breast­feed than the low­est, so it is not sur­pris­ing that research shows that breast­fed infants dis­play all the health and edu­ca­tion­al bene­fits they were born into. But even if edu­ca­tion, class and wealth is taken into account, there is known to be a big dif­fer­ence between the type of moth­er who fol­lows the advice of her doc­tor and breast­feeds, and the one that ignores it to give the bottle. In oth­er words, breast­feed­ing stud­ies could simply be show­ing what it’s like to grow up in a fam­ily that makes an effort to be healthy and respons­ible, as opposed to any­thing pos­it­ive in breast milk.

This is not to say that breast­feed­ing is not good:

• Wolf acknow­ledges that it helps pre­vent gastrointest­in­al infec­tions (life-sav­ing in the devel­op­ing world, gen­er­ally a mild com­plaint in the West).
• Michael Kramer (one of the world’s most author­it­at­ive sources of breast­feed­ing research; advisor to the WHO, Unicef and the Cochrane Lib­rary) believes:
• The evid­ence is “encour­aging” in pre­vent­ing res­pir­at­ory prob­lems.
• The data on help­ing pre­vent breast can­cer is “sol­id”.

How­ever:

• The data on obesity, aller­gies, asthma,Â leuk­aemia, lymph­oma, bowel dis­ease, type 1 dia­betes, heart dis­ease and blood pres­sure are “weak” at best.
• The “highly respec­ted” Amer­ic­an Agency for Health­care Research and Qual­ity (AHRQ) warns that, “because the breast­feed­ing moth­ers were self-select­ing, ‘one should not infer caus­al­ity’ ”.
• The World Health Organ­isa­tion’s own research review con­cluded that gains were “mod­est” and also warned that “because none of the stud­ies it looked at dealt with the prob­lem of con­found­ing, the res­ults could be explained by the ‘self-selec­tion of breast­feed­ing moth­ers’ ”.

via @TimHarford

# Why Designers Need Statistics

The pro­lif­er­a­tion of infograph­ics online is help­ing to make a broad, some­what stat­ist­ic­ally illit­er­ate, audi­ence aware of import­ant data and trends.

For those design­ing these infograph­ics, there­fore, there is a need that they under­stand their pro­cess intimately–from data col­lec­tion to illustration–in order to ana­lyse it hon­estly and with mean­ing.

Through a “show­case of bad infograph­ics”, Smash­ing Magazine lam­basts the trend of inap­pro­pri­ate infor­graph­ics and offers an inter­est­ing essay on why design­ers need to be stat­ist­ic­ally lit­er­ate.

The import­ance of stat­ist­ic­al lit­er­acy in the Inter­net age is clear, but the concept is not exclus­ive to design­ers. I’d like to focus on it because design­ers must con­sider it in a way that most people do not have to: stat­ist­ic­al lit­er­acy is more than learn­ing the laws of stat­ist­ics; it is about rep­res­ent­a­tions that the human mind can under­stand and remem­ber.

As a design­er, you get to choose those rep­res­ent­a­tions. Most of the time this is a pos­it­ive aspect. Visu­al rep­res­ent­a­tions allow you to quickly sum­mar­ize a data set or make con­nec­tions that might be dif­fi­cult to per­ceive oth­er­wise. Unfor­tu­nately, design­ers too often for­get that data exists for more than enter­tain­ment or aes­thet­ics. If you design a visu­al­iz­a­tion before cor­rectly under­stand­ing the data on which it is based, you face the very real risk of sum­mar­iz­ing incor­rectly, pro­du­cing faulty insights, or oth­er­wise mangling the pro­cess of dis­sem­in­at­ing know­ledge. If you do this to your audi­ence, then you have viol­ated an expect­a­tion of sin­gu­lar import­ance for any con­tent cre­at­or: their expect­a­tion that you actu­ally know what you’re talk­ing about.

The two rules of infograph­ic pro­duc­tion:

1. If it would lead to the wrong con­clu­sions, not present­ing the data at all would be bet­ter.
2. Your pro­ject isn’t ready to be released into the wild if you’ve spent more time choos­ing a font than choos­ing your data.

I am reminded of this tan­gen­tially-related infograph­ic tem­plate from Flow­ing­Data.