We scan new podcasts and send you the top 5 insights daily.
Unimodal datasets, such as bulk text or simple image categories, quickly saturate and depreciate in value as models solve them. In contrast, complex multimodal data that links video, audio, scripts, or temporal patient records resists decay and remains valuable across training runs.
While companies readily use models that process images, audio, and text inputs, the practical application of generating multimodal outputs (like video or complex graphics) remains rare in business. The primary output is still text or structured data, with synthesized speech being the main exception.
Google's Embedding 2 model is a significant infrastructure upgrade because it is 'natively multimodal.' This allows AI to directly understand and retrieve images, diagrams, and text without first converting non-text data into lossy captions. This makes internal knowledge bases and co-pilots dramatically more effective and accurate for enterprises.
The next significant evolution in AI infrastructure is the shift to multimodal systems. Future tech stacks must move beyond single-modality paradigms (like text-only) to seamlessly handle and integrate text, images, audio, and video within a single, unified architecture.
Contrary to the belief that AI requires perfect, clean data, the biggest opportunity lies in building technology that can find signals in messy, diverse data sets across different modalities and organisms. The tech should solve the data problem, not wait for it to be solved.
The Sora team views video as having lower "intelligence per bit" compared to text. However, the total volume of available video data is vastly larger and less tapped. This suggests that, unlike LLMs facing a data crunch, video models can scale with more data for a very long time.
Unlike text-based LLMs where simply increasing parameter count works, Verge Labs found the biggest AI performance gains in biology come from scaling data modalities—adding new types of data like proteomics and imaging. Fusing different data sources is more critical than just making the model bigger.
Research shows that AI models trained on smaller, high-quality datasets are more efficient and capable than those trained on the unfiltered internet. This signals an industry shift from a 'more data' to a 'right data' paradigm, prioritizing quality over sheer quantity for better model performance.
The repeated mention of the 'Duccio' framework for multimodal feature extraction signals a key trend. Advanced recommendation systems are moving beyond single data types, integrating audio, visual, and textual data to build a more holistic understanding of user preferences and products.
When pre-training a large multimodal model, including small samples from many diverse modalities (like LiDAR or MRI data) is highly beneficial. This "tempts" the model, giving it an awareness that these data types exist and have structure. This initial exposure makes the model more adaptable for future fine-tuning on those specific domains.
The era of building frontier AI models on easily scraped internet data is ending. The next competitive advantage lies in securing unique, proprietary, real-world datasets that reflect complex physical interactions, such as endoscopy videos or 3D object data. Synthetic data is proving insufficient, making access to this "reality" data the key differentiator.