Treat your data pipelines like code, because your agents already do | MotherDuck's Jordan Tigani

The podcast discusses the evolving landscape of data engineering and data science in the context of AI advancements, focusing on how new technologies are reshaping workflows and tools. A key theme is the shift from large-scale distributed systems like BigQuery - designed for massive datasets - to more efficient, single-node systems that leverage modern hardware capabilities such as high-core-count processors and abundant RAM. This transition enables faster, lower-latency processing ideal for the majority of real-world queries, which tend to be small in scale. The discussion highlights how this architectural evolution supports emerging paradigms like serverless, scale-to-zero instances and better aligns with the needs of AI agents, which benefit from simpler, more responsive environments.

AI and agent-driven workflows are transforming traditional data roles, automating tasks such as SQL generation, data visualization, and pipeline creation. As AI models like Claude improve, they can generate accurate queries, create visualizations, and even handle data transformations, reducing reliance on manual coding. However, this shift introduces new challenges around governance, security, and oversight - particularly concerning unchecked AI-generated queries and the need for review processes similar to code reviews. To address these issues, approaches like natural language "guides" are proposed to provide business context to AI systems, replacing rigid semantic models with flexible, human-readable documentation. The conversation also emphasizes the importance of knowledge sharing, cost management due to expensive AI tokens, and building systems that are both atomic in operation and adaptable to rapid technological change.

• What if you treated your next data project as an AI agent onboarding process?

  • Move: Instead of writing SQL or building pipelines manually, define your data logic in a Markdown "Guide" (e.g., revenue.md) that explains business rules in plain English, includes example queries, and links to source tables in your MotherDuck/DuckDB instance.
  • Why Now?: AI models like Claude now understand natural language context well enough to generate accurate SQL from guides - avoiding rigid semantic layers that slow down solo developers. Tools like GitHub + commit hooks make syncing guides to live systems trivial.
  • Expected Upside: Cut query development time by 50 - 70% over the next month; reuse the same guide for dashboards, alerts, and agent tasks without re-engineering logic.

• What if you replaced your scheduled ETL scripts with reactive AI-driven flights?

  • Move: Replace one cron-based Python script (e.g., hourly data backfill) with a lightweight Flight triggered by a GitHub commit or file drop, using a simple LLM prompt to infer schema changes and adjust transformations automatically.
  • Why Now?: Modern single-node engines like DuckDB can handle most pipeline workloads efficiently, and AI agents can now detect field renames or null spikes better than static code. Over-engineered Airflow-style orchestration is no longer needed for 90% of use cases.
  • Expected Upside: Reduce maintenance overhead by eliminating brittle schedules; achieve faster iteration cycles with atomic, version-controlled flights that adapt to input changes without manual updates.

• What if you outsourced your dashboard creation to an AI agent - but kept control via code review?

  • Move: Use a prompt with your data guide and sample dataset to generate a visualization (Dive) via Claude or Gemini, then commit the output code (TypeScript/Python) to GitHub where you manually review, tweak, and deploy it via CI.
  • Why Now?: LLMs now produce production-ready visualization code, but blind trust leads to errors. By treating AI output as a pull request - not a final product - you maintain safety while accelerating delivery.
  • Expected Upside: Launch new dashboards in hours instead of days; build a reusable library of vetted visualizations that evolve alongside your data, all tracked in Git for auditability and collaboration.

• Implement a GitHub-first workflow for data scripts and visualizations, using commit hooks to automatically sync changes to your data platform to ensure version control and collaboration.
• Build lightweight, single-node data solutions optimized for low-latency and cost-efficiency, especially for small to mid-sized datasets under 100MB, leveraging modern hardware capabilities.
• Create and maintain simple Markdown guides that document business logic, schema definitions, and key metrics to provide AI agents with consistent context and reduce token usage in prompts.
• Design AI agent workflows with built-in review and verification steps - either human or automated - to catch errors in generated SQL, visualizations, or data mutations before execution.
• Prioritize strategic use of high-performance AI models only for critical tasks, monitor token consumption closely, and avoid over-engineering solutions that may become obsolete as AI improves.