The significant barrier of messy, legacy data is being overcome by AI. Snowflake is developing "agent-driven migrations" that automate the process of moving data from old systems onto modern platforms. This drastically reduces project timelines from multiple years to just a few weeks.

To avoid AI hallucinations, Square's AI tools translate merchant queries into deterministic actions. For example, a query about sales on rainy days prompts the AI to write and execute real SQL code against a data warehouse, ensuring grounded, accurate results.

Ingress, Stonebraker's first database, couldn't handle non-standard data types like polygons for GIS or custom calendars for financial bonds. Postgres was engineered with an extendable type system to solve this fundamental limitation, making it vastly more flexible for diverse applications beyond standard business data processing.

AI agents make it dramatically easier to extract and migrate data from platforms, reducing vendor lock-in. In response, platforms like Snowflake are embracing open file formats (e.g., Iceberg), shifting the competitive basis from data gravity to superior performance, cost, and features.

dbt's design philosophy is 'progressive complexity.' It uses accessible SQL to lower the entry barrier for analysts, who can start simple and only engage with more complex features as needed. This approach avoids the initial overwhelm common with powerful engineering tools like Spark.

Empower your entire team to perform data analysis safely by having analysts check verified SQL queries, table schemas, and analysis playbooks into a shared repository. This reduces reliance on the data team and prevents incorrect, "hallucinated" results from AI agents.

To enable AI tools like Cursor to write accurate SQL queries with minimal prompting, data teams must build a "semantic layer." This file, often a structured JSON, acts as a translation layer defining business logic, tables, and metrics, dramatically improving the AI's zero-shot query generation ability.

Databricks and Snowflake took opposite approaches. Snowflake optimized for fast queries on curated, proprietary "downstream" data. Databricks focused on large-scale, messy "upstream" data ingestion using open formats. Databricks found it easier to add speed than it was for Snowflake to move upstream and abandon its proprietary lock-in.

Specialized knowledge that takes humans weeks to learn can be codified into compact 'skill files' for AI agents. dbt Labs condensed its training curriculum into a small file that 'teaches' an agent to perform complex tasks, like a data migration, in a fraction of the time and cost.