Instead of one component doing everything, SAM3 first uses a specialized token to answer a simple question: "Is this concept in the image at all?" Only then does it proceed to localization. This simplifies the model's task, improving its ability to avoid hallucinating objects that aren't there.
Related Insights
The key innovation was a data engine where AI models, fine-tuned on human verification data, took over mask verification and exhaustivity checks. This reduced the time to create a single training data point from over 2 minutes (human-only) to just 25 seconds, enabling massive scale.
Sora doesn't process pixels or frames individually. Instead, it uses "space-time tokens" — small cuboids of video data combining spatial and temporal information. This voxel-like representation is the fundamental unit, enabling the model to understand properties like object permanence through global attention.
Current multimodal models shoehorn visual data into a 1D text-based sequence. True spatial intelligence is different. It requires a native 3D/4D representation to understand a world governed by physics, not just human-generated language. This is a foundational architectural shift, not an extension of LLMs.
To teach the model to recognize when a concept is *not* in an image, the team heavily annotated negative phrases. This massive volume of negative data was critical for building a robust recognition capability and preventing the model from falsely detecting objects that are not present.
A significant real-world challenge is that users have different mental models for the same visual concept (e.g., does "hand" include the arm?). Fine-tuning is therefore not just for learning new objects, but for aligning the model's understanding with a specific user's or domain's unique definition.
While SAM3 can act as a "tool" for LLMs, researchers argue that fundamental vision tasks like counting fingers should be a native, immediate capability of a frontier model, akin to human System 1 thinking. Relying on tool calls for simple perception indicates a critical missing capability in the core model.
The team views its comprehensive 'SeiCo' benchmark, with over 200,000 concepts, as a more lasting contribution than the SAM3 model itself. While models are quickly surpassed, a robust benchmark can guide and measure progress for the entire research community for years.
The model uses separate components for detection and tracking. The detector needs an identity-agnostic representation (e.g., "dog"), while the tracker needs a unique representation for each instance (e.g., "this specific dog"). Decoupling these conflicting requirements was a key architectural breakthrough for video performance.
Contrary to common perception shaped by their use in language, Transformers are not inherently sequential. Their core architecture operates on sets of tokens, with sequence information only injected via positional embeddings. This makes them powerful for non-sequential data like 3D objects or other unordered collections.
Human intelligence is multifaceted. While LLMs excel at linguistic intelligence, they lack spatial intelligence—the ability to understand, reason, and interact within a 3D world. This capability, crucial for tasks from robotics to scientific discovery, is the focus for the next wave of AI models.