login/register

Snip!t from collection of Alan Dix

see all channels for Alan Dix

Snip
summary

Data from a study of nearly 2,000 people seemed to show ... moderates saw shades of grey more accurately than did ei... right-wing extremists. “The hypothesis was sexy,” he...
But then reality intervened. Sensitive to controversies ...
It turned o

Scientific method: Statistical errors : Nature News & Comment
http://www.nature.com/news/scientific-method-statistical-errors-1.14700

Categories

/Channels/research methods

[ go to category ]

For Snip

loading snip actions ...

For Page

loading url actions ...

Data from a study of nearly 2,000 people seemed to show that political moderates saw shades of grey more accurately than did either left-wing or right-wing extremists. “The hypothesis was sexy,” he says, “and the data provided clear support.” The P value, a common index for the strength of evidence, was 0.01 — usually interpreted as 'very significant'. Publication in a high-impact journal seemed within Motyl's grasp.

But then reality intervened. Sensitive to controversies over reproducibility, Motyl and his adviser, Brian Nosek, decided to replicate the study. With extra data, the P value came out as 0.59 — not even close to the conventional level of significance, 0.05. The effect had disappeared, and with it, Motyl's dreams of youthful fame1.

It turned out that the problem was not in the data or in Motyl's analyses. It lay in the surprisingly slippery nature of the P value, which is neither as reliable nor as objective as most scientists assume. “P values are not doing their job, because they can't,” says Stephen Ziliak, an economist at Roosevelt University in Chicago, Illinois, and a frequent critic of the way statistics are used.

HTML

<p>Data from a study of nearly 2,000 people seemed to show that political moderates saw shades of grey more accurately than did either left-wing or right-wing extremists. &#x201c;The hypothesis was sexy,&#x201d; he says, &#x201c;and the data provided clear support.&#x201d; The <i>P</i> value, a common index for the strength of evidence, was 0.01 &#x2014; usually interpreted as 'very significant'. Publication in a high-impact journal seemed within Motyl's grasp.</p> <p>But then reality intervened. Sensitive to controversies over reproducibility, Motyl and his adviser, Brian Nosek, decided to replicate the study. With extra data, the <i>P</i> value came out as 0.59 &#x2014; not even close to the conventional level of significance, 0.05. The effect had disappeared, and with it, Motyl's dreams of youthful fame<sup><a class="ref-link" title="Nosek, B. A., Spies, J. R. &amp; Motyl, M. Perspect. Psychol. Sci. 7, 615&#x2013;631 (2012)." id="ref-link-1" href="#b1">1</a></sup>.</p> <p>It turned out that the problem was not in the data or in Motyl's analyses. It lay in the surprisingly slippery nature of the <i>P</i> value, which is neither as reliable nor as objective as most scientists assume. &#x201c;<i>P</i> values are not doing their job, because they can't,&#x201d; says Stephen Ziliak, an economist at Roosevelt University in Chicago, Illinois, and a frequent critic of the way statistics are used.</p>