Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Spencer Greenberg notes that replication efforts have curbed conventional statistical p-hacking, but a subtler issue has taken its place. Researchers now practice importance hacking by subtly shifting the meaning of their results. Peer reviewers believe the findings imply major real-world breakthroughs when the replicated data actually demonstrates something trivial.

Related Insights

The 'replication crisis' shows many studies fail to replicate. Researchers, biased by their desire for success, engage in 'p-hacking' and delude themselves into finding significance. Paradoxically, the most surprising and celebrated results are often the least likely to be true.

The scientific process relies on replication to verify findings, yet academic journals create a "replication crisis" by prioritizing novelty over truth. They often reject crucial follow-up studies, telling researchers "you already published this finding, so there's nothing novel here," which obstructs the validation of important research.

Daryl Bem's 2011 paper 'proving' ESP was a key trigger for psychology's replication crisis. Because it followed the field's accepted statistical practices (p-hacking) to reach an absurd conclusion, it forced a dilemma: either ESP is real, or the field's methods are fundamentally unreliable.

The "Batman Effect" study's choice of a superhero to test a "disruption" hypothesis introduces a glaring confound (priming heroism). This may be a deliberate strategy to create ambiguity, ensuring a stream of follow-up studies is needed to disentangle the effects, thus building a literature.

Much published research is false because scientists find correlations and then create a hypothesis retrospectively, like drawing a bullseye around a bullet hole. Requiring predictions *before* data collection forces intellectual honesty, a practice valuable for business A/B testing and market research.

The public appetite for surprising, "Freakonomics-style" insights creates a powerful incentive for researchers to generate headline-grabbing findings. This pressure can lead to data manipulation and shoddy science, contributing to the replication crisis in social sciences as researchers chase fame and book deals.

Contrary to popular belief, publication in a top academic journal doesn't guarantee a study is correct. The social sciences lack the precise experimental validation of hard sciences, allowing incorrect theories to have "long legs and survive" due to a lack of rigorous, focused scrutiny from peers.

Once-popular concepts like psychological "priming" have been largely disproven through replication studies. A reliable rule for marketers is that if a psychological input is ridiculously small and barely noticeable, it is unlikely to produce a significant or repeatable behavioral change.

An intuitive finding (swearing improves strength) is undermined by its proposed mechanism, "state disinhibition," which the hosts critique as meaningless jargon. This highlights a common flaw where psychology papers invent complex, unprovable explanations for simple observations.

Psychologist Lisa Miller overcame scientific skepticism by embedding her novel research on spirituality within familiar, rigorously designed studies. By using established datasets and methods for nine out of ten variables and adding only "personal importance of spirituality," her work passed peer review and was published in top journals.