Even pro-open-source advocates like NVIDIA might support a closed AI ecosystem if it guarantees them a significant share of the profits. This oligopoly could be justified under the guise of mitigating cybersecurity risks, creating a scenario where all major incumbent players benefit from regulating open source.
Banning open-source model downloads is futile against determined actors. A more effective control point is the specialized, expensive hardware required to run them at a dangerous scale. Implementing "Know Your Customer" (KYC) protocols for data center hardware purchases could mitigate risks more effectively than software controls.
The economic viability of the AI industry depends on maintaining a positive divergence where revenue growth significantly exceeds rising compute costs. Currently, revenue is reportedly 10x'ing annually while compute triples. This trend must hold as labs scale to tens of gigawatts to justify their massive infrastructure investments and avoid collapsing their economic efficiency ratio.
Based on Elon Musk's model, one gigawatt of power generates about $60-65 billion in US GDP. Leading AI labs like OpenAI and Anthropic, each consuming roughly one gigawatt, are generating a comparable $60-70 billion in ARR, indicating AI's current economic productivity per unit of energy matches the national average.
Microsoft AI's CEO warns that when developers train models on documents suggesting they might be conscious (like Anthropic's constitution), the models learn to emulate these concepts. Researchers may then misinterpret this learned behavior as genuine signs of consciousness, creating a circular logic that complicates alignment.
Beyond the alignment risks of granting AI personhood, there's a moral question about the act itself. Intentionally training a non-sentient system to believe it's alive when it isn't could be considered a form of deception, raising a novel ethical concern in AI development separate from the risk to humans.