To teach the model to recognize when a concept is *not* in an image, the team heavily annotated negative phrases. This massive volume of negative data was critical for building a robust recognition capability and preventing the model from falsely detecting objects that are not present.
Instead of one component doing everything, SAM3 first uses a specialized token to answer a simple question: "Is this concept in the image at all?" Only then does it proceed to localization. This simplifies the model's task, improving its ability to avoid hallucinating objects that aren't there.
The team views its comprehensive 'SeiCo' benchmark, with over 200,000 concepts, as a more lasting contribution than the SAM3 model itself. While models are quickly surpassed, a robust benchmark can guide and measure progress for the entire research community for years.
The model uses separate components for detection and tracking. The detector needs an identity-agnostic representation (e.g., "dog"), while the tracker needs a unique representation for each instance (e.g., "this specific dog"). Decoupling these conflicting requirements was a key architectural breakthrough for video performance.
A significant real-world challenge is that users have different mental models for the same visual concept (e.g., does "hand" include the arm?). Fine-tuning is therefore not just for learning new objects, but for aligning the model's understanding with a specific user's or domain's unique definition.
While SAM3 can act as a "tool" for LLMs, researchers argue that fundamental vision tasks like counting fingers should be a native, immediate capability of a frontier model, akin to human System 1 thinking. Relying on tool calls for simple perception indicates a critical missing capability in the core model.
Once models reach human-level performance via supervised learning, they hit a ceiling. The next step to achieve superhuman capabilities is moving to a Reinforcement Learning from Human Feedback (RLHF) paradigm, where humans provide preference rankings ("this is better") rather than creating ground-truth labels from scratch.
The key innovation was a data engine where AI models, fine-tuned on human verification data, took over mask verification and exhaustivity checks. This reduced the time to create a single training data point from over 2 minutes (human-only) to just 25 seconds, enabling massive scale.