There is emerging evidence of a "pay-to-play" dynamic in AI search. Platforms like ChatGPT seem to disproportionately cite content from sources with which they have commercial deals, such as the Financial Times and Reddit. This suggests paid partnerships can heavily influence visibility in AI-generated results.

An attempt to teach AI 'scientific taste' using RLHF on hypotheses failed because human raters prioritized superficial qualities like tone and feasibility over a hypothesis's potential world-changing impact. This suggests a need for feedback tied to downstream outcomes, not just human preference.

The danger of LLMs in research extends beyond simple hallucinations. Because they reference scientific literature—up to 50% of which may be irreproducible in life sciences—they can confidently present and build upon flawed or falsified data, creating a false sense of validity and amplifying the reproducibility crisis.

A concerning trend is using AI to expand brief thoughts into verbose content, which then forces recipients to use AI to summarize it. This creates a wasteful cycle that amplifies digital noise and exhaustion without adding real value, drowning organizations in synthetic content.

A study found evaluators rated AI-generated research ideas as better than those from grad students. However, when the experiments were conducted, human ideas produced superior results. This highlights a bias where we may favor AI's articulate proposals over more substantively promising human intuition.

The greatest danger of AI content isn't job loss or bad SEO, but a societal one. Since we consume more brand content than educational material, an internet flooded with AI's 'predictive text' based on what's common could relegate collective human knowledge and creativity to a permanent base level.

A major frontier for AI in science is developing 'taste'—the human ability to discern not just if a research question is solvable, but if it is genuinely interesting and impactful. Models currently struggle to differentiate an exciting result from a boring one.

Labs are incentivized to climb leaderboards like LM Arena, which reward flashy, engaging, but often inaccurate responses. This focus on "dopamine instead of truth" creates models optimized for tabloids, not for advancing humanity by solving hard problems.

Research shows that feeding LLMs junk social media content leads to significant cognitive decline, including a 23% drop in reasoning. This AI "brain rot" persists even after retraining on high-quality data, mirroring the negative cognitive effects observed in humans who doomscroll.