As more of the public internet and code repositories are generated by LLMs, any new model trained on this public data is, in effect, being 'distilled' from other models. This complicates accusations of direct distillation and blurs the line for what constitutes original training data.

Contrary to the popular belief that open-source AI will inevitably catch up, a NIST analysis indicates the performance gap between open and closed-source models is growing. The performance trend lines are diverging, suggesting frontier models are improving at a significantly faster rate.

Large, centralized AI models are vulnerable to 'distillation attacks,' where a smaller model can be trained cheaply by querying the larger one. This technical reality, combined with the moral hypocrisy of creators restricting copying after scraping the internet, strongly suggests a future dominated by decentralized, open-source models.

As more of the internet and code repositories are generated by leading AI models, any new model trained on this public data inadvertently "distills" the knowledge and quirks of those proprietary systems. This blurs the line between original training and outright copying.

The effort to shut down a "dangerous" model like Anthropic's Mythos is largely temporary. The rapid pace of open-source development means its capabilities will likely be replicated and universally available in 6-12 months, rendering current control measures moot.

History in tech shows that open systems like Linux and Android tend to defeat closed ones. The same dynamic is playing out in AI. Open-source models will likely win long-term because they optimize for widespread adoption and rapid innovation, while closed models focus on maximizing short-term profits within a ring-fenced environment.

The debate over distilling from other AI models is becoming moot. The internet is now so saturated with AI-generated content ('AI slop') that any new model trained on web data is already, by default, being trained on the outputs of its predecessors. Pure 'human data' is a dwindling resource.

As developers increasingly use AI coding assistants like Claude Code, they flood public repositories like GitHub with high-quality, AI-generated outputs. This effectively turns the internet into a massive, unavoidable training dataset for competing models, making it difficult to police "distillation" as a violation of terms.

New open-weight models like Inkling are not entirely 'pure'; they use 'distillation light' from other open models (e.g., Kimi). Since those models may be distilled from closed-source giants like OpenAI, it creates a multi-layered dependency chain where traits and biases are passed down, blurring the lines between truly independent and derivative models.