# Statistical Significance Explained

If you didn’t read the House of Commons Library’s statistical literacy guides recently (or you need a refresher on what, exactly, statistical significance means), then you can do much worse than student Warren Davies’ short rundown on the meaning of statistical significance:

In science we’re always testing hypotheses. We never conduct a study to ‘see what happens’, because there’s always at least one way to make any useless set of data look important. We take a risk; we put our idea on the line and expose it to potential refutation. Therefore, all statistical tests in psychology test the probability of obtaining your given set of results (and all those that are even more extreme) if the hypothesis were incorrect – i.e. the null hypothesis were true. [â€¦]

This is what statistical significance testing tells you – the probability that the result (and all those that are even more extreme) would have come about if the null hypothesis were true. [â€¦] It’s given as a value between 0 and 1, and labelled p. So p = .01 means a 1% chance of getting the results if the null hypothesis were true; p = .5 means 50% chance, p = .99 means 99%, and so on.

In psychology we usually look for p values lower than .05, or 5%. That’s what you should look out for when reading journal papers. If there’s less than a 5% chance of getting the result if the null hypothesis were true, a psychologist will be happy with that, and the result is more likely to get published.

Significance testing is not perfect, though. Remember this: ‘Statistical significance is not psychological significance.’ You must look at other things too; the effect size, the power, the theoretical underpinnings. Combined, they tell a story about how important the results are, and with time you’ll get better and better at interpreting this story.

To get a real feel for this, Davies provides a simple-to-follow example (a loaded die) in the post.

via @sandygautam

# Political Risk Assessments

“Safety is never allowed to trump all other concerns”, says Julian Baggini, and without saying as much governments must consistently put a price on lives and determine how much risk to expose the public to.

In anÂ articleÂ for the BBC, Baggini takes a comprehensive look at how governments make risk assessments and in the process discusses a topic of constant intrigue for me: how much a human life is valued by different governments and their departments.

The ethics of risk is not as straightforward as the rhetoric of “paramount importance” suggests. People talk of the “precautionary principle” or “erring on the side of caution” but governments are always trading safety for convenience or other gains. [â€¦]

Governments have to choose on our behalf which risks we should be exposed to.

That poses a difficult ethical dilemma: should government decisions about risk reflect the often irrational foibles of the populace or the rational calculations of sober risk assessment? Should our politicians opt for informed paternalism or respect for irrational preferences? [â€¦]

In practice, governments do not make fully rational risk assessments. Their calculations are based partly on cost-benefit analyses, and partly on what the public will tolerate.

# Statistical Literacy Guides

I am suitably impressed by the clarity and breadthÂ of the House of Commons Library’s statistical literacy guide on How to spot spin and inappropriate use of statistics (pdf, viaÂ @TimHarford).

A quick dig around the archives revealed a full series of statistical literacy guides (all pdf), all of which are fantastically readable, accessible and comprehensive. These are must-read guides to what some people feel are complex, seemingly-monolithical subjects:

# The Evidence on Breastfeeding

In an article the Royal Statistical Society announced as the runner-up in their annual Awards for Statistical Excellence in Journalism, Helen Rumbelow thoroughly investigates the well-debated subject of breastfeeding.

The conclusion of the piece is that much of the evidence in support of breastfeeding is massively misrepresented or inherently flawed.

“The evidence to date suggests it probably doesn’t make much difference if you breastfeed.” [â€¦]

“The conclusion is that the evidence we have now is not compelling. It certainly does not justify the rhetoric,” [American academic Joan Wolf] says. The problem with the studies is that it is very hard to separate the benefits of the mother’s milk from the benefits of the kind of mother who chooses to breastfeed. In the UK, for example, the highest class of women are 60 per cent more likely to breastfeed than the lowest, so it is not surprising that research shows that breastfed infants display all the health and educational benefits they were born into. But even if education, class and wealth is taken into account, there is known to be a big difference between the type of mother who follows the advice of her doctor and breastfeeds, and the one that ignores it to give the bottle. In other words, breastfeeding studies could simply be showing what it’s like to grow up in a family that makes an effort to be healthy and responsible, as opposed to anything positive in breast milk.

This is not to say that breastfeeding is not good:

• Wolf acknowledges that it helps prevent gastrointestinal infections (life-saving in the developing world, generally a mild complaint in the West).
• Michael Kramer (one of the world’s most authoritative sources of breastfeeding research; advisor to the WHO, Unicef and the Cochrane Library) believes:
• The evidence is “encouraging” in preventing respiratory problems.
• The data on helping prevent breast cancer is “solid”.

However:

• The data on obesity, allergies, asthma,Â leukaemia, lymphoma, bowel disease, type 1 diabetes, heart disease and blood pressure are “weak” at best.
• The “highly respected” American Agency for Healthcare Research and Quality (AHRQ) warns that, “because the breastfeeding mothers were self-selecting, ‘one should not infer causality'”.
• The World Health Organisation’s own research review concluded that gains were “modest” and also warned that “because none of the studies it looked at dealt with the problem of confounding, the results could be explained by the ‘self-selection of breastfeeding mothers'”.

via @TimHarford

# Why Designers Need Statistics

The proliferation of infographics online is helping to make a broad, somewhat statistically illiterate, audience aware of important data and trends.

For those designing these infographics, therefore, there is a need that they understand their process intimately–from data collection to illustration–in order to analyse it honestly and with meaning.

Through a “showcase of bad infographics”, Smashing Magazine lambasts the trend of inappropriate inforgraphics and offers an interesting essay on why designers need to be statistically literate.

The importance of statistical literacy in the Internet age is clear, but the concept is not exclusive to designers. I’d like to focus on it because designers must consider it in a way that most people do not have to: statistical literacy is more than learning the laws of statistics; it is about representations that the human mind can understand and remember.

As a designer, you get to choose those representations. Most of the time this is a positive aspect. Visual representations allow you to quickly summarize a data set or make connections that might be difficult to perceive otherwise. Unfortunately, designers too often forget that data exists for more than entertainment or aesthetics. If you design a visualization before correctly understanding the data on which it is based, you face the very real risk of summarizing incorrectly, producing faulty insights, or otherwise mangling the process of disseminating knowledge. If you do this to your audience, then you have violated an expectation of singular importance for any content creator: their expectation that you actually know what you’re talking about.

The two rules of infographic production:

1. If it would lead to the wrong conclusions, not presenting the data at all would be better.
2. Your project isn’t ready to be released into the wild if you’ve spent more time choosing a font than choosing your data.

I am reminded of this tangentially-related infographic template from FlowingData.