The key innovation was a data engine where AI models, fine-tuned on human verification data, took over mask verification and exhaustivity checks. This reduced the time to create a single training data point from over 2 minutes (human-only) to just 25 seconds, enabling massive scale.

Instead of one component doing everything, SAM3 first uses a specialized token to answer a simple question: "Is this concept in the image at all?" Only then does it proceed to localization. This simplifies the model's task, improving its ability to avoid hallucinating objects that aren't there.

Sora doesn't process pixels or frames individually. Instead, it uses "space-time tokens" — small cuboids of video data combining spatial and temporal information. This voxel-like representation is the fundamental unit, enabling the model to understand properties like object permanence through global attention.

Traditional video models process an entire clip at once, causing delays. Descartes' Mirage model is autoregressive, predicting only the next frame based on the input stream and previously generated frames. This LLM-like approach is what enables its real-time, low-latency performance.

Current multimodal models shoehorn visual data into a 1D text-based sequence. True spatial intelligence is different. It requires a native 3D/4D representation to understand a world governed by physics, not just human-generated language. This is a foundational architectural shift, not an extension of LLMs.

To analyze video cost-effectively, Tim McLear uses a cheap, fast model to generate captions for individual frames sampled every five seconds. He then packages all these low-level descriptions and the audio transcript and sends them to a powerful reasoning model. This model's job is to synthesize all the data into a high-level summary of the video.

To teach the model to recognize when a concept is *not* in an image, the team heavily annotated negative phrases. This massive volume of negative data was critical for building a robust recognition capability and preventing the model from falsely detecting objects that are not present.

While SAM3 can act as a "tool" for LLMs, researchers argue that fundamental vision tasks like counting fingers should be a native, immediate capability of a frontier model, akin to human System 1 thinking. Relying on tool calls for simple perception indicates a critical missing capability in the core model.

The team views its comprehensive 'SeiCo' benchmark, with over 200,000 concepts, as a more lasting contribution than the SAM3 model itself. While models are quickly surpassed, a robust benchmark can guide and measure progress for the entire research community for years.