Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Contrary to popular belief, publication in a top academic journal doesn't guarantee a study is correct. The social sciences lack the precise experimental validation of hard sciences, allowing incorrect theories to have "long legs and survive" due to a lack of rigorous, focused scrutiny from peers.

Related Insights

Tyler Cowen argues the AI risk community's reluctance to engage in formal peer review weakens their arguments. Unlike fields like climate change, which built a robust literature, the movement's reliance on online discourse lacks the rigorous scrutiny needed to build credible scientific consensus.

When a study is presented at a major conference like ASCO, it gains visibility and a perception of having been vetted. This can create a "tailwind," leading subsequent journal reviewers to be less critical, as they may assume the work has already undergone rigorous scrutiny, which is often not the case for conference abstracts.

Invoking 'studies say' or 'science backed' has become a rhetorical trick to claim intellectual authority and shut down conversation. It's wiser to adopt a default position of skepticism, as these phrases often precede weak or misrepresented claims, especially in soft sciences.

A philosophy paper defines an expert as someone with a PhD who cites peer-reviewed journals. This is mocked as an insular, academic view of expertise that absurdly discounts practical, real-world masters, such as a Bushman who knows how to find water in a desert.

The public appetite for surprising, "Freakonomics-style" insights creates a powerful incentive for researchers to generate headline-grabbing findings. This pressure can lead to data manipulation and shoddy science, contributing to the replication crisis in social sciences as researchers chase fame and book deals.

A key feature making economics research robust is its structure. Authors not only present their thesis and evidence but also anticipate and systematically discredit competing theories for the same outcome. This intellectual honesty is a model other social sciences could adopt to improve credibility.

Professor Asao Inouye's theory—that grading English promotes white supremacy—was presented not at a fringe event but as the keynote at his field's biggest conference. This shows how radical ideas can become centrally accepted dogma within academic fields, making dissent from peers seem heretical.

Historically, generating a good hypothesis was the most prestigious part of science. Now, AI can produce theories at near-zero cost, overwhelming traditional validation systems like peer review. The new grand challenge is developing scalable methods to verify and filter this flood of AI-generated ideas.

While commercial conflicts of interest are heavily scrutinized, the pressure on academics to produce positive results to secure their next large institutional grant is often overlooked. This intense pressure to publish favorably creates a significant, less-acknowledged form of research bias.

Despite multiple refutations, the "More Guns, Less Crime" debate continues. This persistence is fueled by ideology, powerful economic interests like the NRA, and the original author's refusal to concede. It shows that in academia, as the saying goes, "progress comes one death at a time."