While de-duplicating protein databases helps learn diverse structures, subtle variations in similar sequences are essential for learning function, as a single mutation can be catastrophic. This justifies training models on massive, unclustered datasets to capture fine-grained functional determinants.

Training a language model to predict the next amino acid in a sequence forces it to learn the protein's 3D structure. To make accurate predictions, the model must understand an amino acid's physical microenvironment, effectively deriving 3D spatial relationships from 1D sequence data alone. This demonstrates emergent capabilities of LLMs in biology.

A key strategy for improving results from generative protein models is "inference-time scaling." This involves generating a vast number of potential structures and then using a separate, fine-tuned scoring model to rank them. This search-and-rank process uncovers high-quality solutions the model might otherwise miss.

Historically, biological discovery relied on harvesting naturally occurring proteins from bacteria or organisms like CRISPR and PCR. By training large language models on sequential protein data instead of natural text, Retro Biosciences and OpenAI demonstrated that bio-native models can generate completely new, functional human proteins at will. This transition shifts protein engineering from scrounging natural biology to an exponential, de novo generative paradigm.

The ESM-C model's performance leap came from adding billions of "noisy" protein sequences from environmental samples. This vast, diverse dataset overcame the limitations of curated databases like Uniref, removing the data bottleneck and revealing clear scaling laws.

The issue with public protein affinity databases isn't just a lack of data, but a lack of interoperability. Data aggregated from different labs using varied methods, buffers, and conditions creates a "messy" dataset that hinders an AI's ability to learn generalizable rules, a problem solved by standardized assays.

The low-hanging fruit of applying AI to existing datasets is being picked. The next major leap forward will come not from slightly better models, but from creative strategies to generate entirely new datasets for unsolved problems like protein stability or in vivo effects.

The success of protein language models can be explained by Zellig Harris's 1954 linguistic theory. Just as a word's meaning is defined by its contexts, an amino acid's biological role is determined by the sequences it can appear in. The model learns this deep statistical structure, effectively learning biology.

Current AI for protein engineering relies on small public datasets like the PDB (~10,000 structures), causing models to "hallucinate" or default to known examples. This data bottleneck, orders of magnitude smaller than data used for LLMs, hinders the development of novel therapeutics.