Unlike older methods that use an "unknown" placeholder, modern tokenizers built on Byte-Pair Encoding (BPE) can represent any string. When faced with a new word, they don't crash or lose information. Instead, they break it down into the smallest known components, ensuring universal representability.

Large, off-the-shelf multilingual models are often poor at less-common languages like Sanskrit due to data scarcity and improper tokenization. A smaller, focused team can outperform them by carefully curating a high-quality corpus, making domain-specific 'care' the key differentiator over raw computing power.

The most significant challenge in building a Sanskrit GPT wasn't coding the transformer model but extracting clean text from PDFs. Issues like scanned images, legacy fonts, and encoding errors required more time than the AI development itself, showing that high-quality data sourcing is the primary obstacle and competitive moat.

A generic tokenizer trained on web text might break a medical term like "myocardial" into meaningless pieces. A specialized tokenizer trained on medical texts would keep it as a single unit. This preserves meaning, improves learning efficiency, and prevents wasting the model's limited context window on fragmented words.

A unified tokenizer, while efficient, may not be optimal for both understanding and generation tasks. The ideal data representation for one task might differ from the other, potentially creating a performance bottleneck that specialized models would avoid.

Current LLMs abstract language into discrete tokens, losing rich information like font, layout, and spatial arrangement. A "pixel maximalist" view argues that processing visual representations of text (as humans do) is a more lossless, general approach that captures the physical manifestation of language in the world.

Beyond the obvious lack of non-English training data, Large Language Models are architecturally biased. Their tokenization process, designed for English, inefficiently breaks down other languages into more fragments. This increases operational costs and reduces comprehension, creating a structural disadvantage.

Models like ChatGPT struggle with basic string manipulation because their fundamental unit of understanding is a "token," which can be a whole word or a sub-word. If a word like "Strawberry" is tokenized into pieces, the model cannot "see" the individual letters to perform tasks like counting them.

Custom tokenizers and embeddings, created for a foundation model, can be repurposed to enhance other data engineering tasks. They can improve OCR accuracy on domain-specific documents, allowing for better text-based processing and avoiding the higher cost of vision models.