Current AI agents focus on "conversation memory" (what you tell them), completely missing the vast context of a user's actual work—like code commits, browsing sessions, or abandoned emails. This creates a significant blind spot in their understanding of user context and intent, as most work happens outside the chat window.
While using an LLM to summarize raw user activity seems intuitive, it is expensive, non-deterministic, and prone to hallucination. A superior approach is a "boring" deterministic compiler using plain code to transform raw data into structured, trustworthy, and recomputable memory episodes, reserving the LLM for higher-level interpretation.
AI agent memory is an emerging attack surface. To build trustworthy systems, memory must enforce a strict, auditable separation between "measured" data (recomputable facts from raw input) and "inferred" data (LLM-generated interpretations). This ensures a ground truth of pure fact remains, defending against memory poisoning attacks.
Analysis of 55 days of screen activity revealed that work is not composed of long, focused blocks. The median activity frame was just 0.3 minutes, with 74% lasting under a minute. This "confetti" pattern of rapid task-switching is a reality that AI summarizers would likely obscure, but which deterministic compilation reveals accurately.