Modern Backups with Gilles Chehade

The conversation explores how data backup and protection have shifted from the era of physical tape libraries and dedicated servers to modern ephemeral, cloud-based workloads. The guest explains that traditional tools assumed stable hardware, gigabyte-scale data, and slower networks, but today's environment involves terabytes and petabytes of data on transient infrastructure that can spawn and despawn on demand, making old approaches physically infeasible within expected timeframes.

The threat landscape has changed substantially with the rise of ransomware, AI agents that can internally delete data, and climate-related infrastructure failures. The guest argues that the probability of data loss has shifted from a low-odds event to an expectation, requiring a move from "in case it happens" planning to "it will happen" recovery planning.

Solutions discussed include categorizing data sets by criticality, applying reconciliation loops similar to infrastructure-as-code practices, and treating backup as a continuous process with inventory, policies, and automated validation rather than a one-shot cron job. The guest describes an open-source backup tool using deduplication, compression, and encryption at the source, with connectors for various data types, plus a proprietary control plane for multi-tenant enterprise use.

• What if you grant your AI agent write access to a production database and it drops a table during a routine task?

  • Move: Before granting any AI agent write access, implement a reconciliation loop that compares your expected schema and data state against reality, with a CI check that breaks on mismatch.
  • Why Now? The guest explains that "the agent is running internally" and "you get the same threats, but you get it from inside", and cites the Facebook AI security head whose mailbox was wiped by an agent during lunch.
  • Expected Upside: As a possibility, you could detect and recover from an accidental schema change within minutes rather than discovering it days later, reducing downtime and data loss.

• What if you discover your cloud bucket - the one you assumed was your backup - has been corrupted or deleted?

  • Move: Implement a separate, encrypted, deduplicated backup at a different provider, driven by the same inventory that manages your infrastructure so there is no drift between what you think is backed up and what actually is.
  • Why Now? The guest states "a big part of the, the cloud era is that we have an assumption that cloud providers store backups, that your history buckets is a backup" and notes providers explicitly disclaim responsibility for your data.
  • Expected Upside: As a possibility, you would have a verified, restorable copy independent of your primary storage, turning a total-loss scenario into a recoverable incident.

• What if you need to restore a single user's accidentally deleted mailbox rather than the entire system?

  • Move: Design your backup strategy around data categorization, assigning restore-time objectives per data set, and use connectors that allow granular, non-file-system-level restores such as database-level or S3-object-level.
  • Why Now? The guest explains that backing up "your entire data store of your whole mail system, that didn't really solve for the kind of problem you would run into, which would be, oh man, a user accidentally deleted their mailbox".
  • Expected Upside: As a possibility, you could restore a single user's data in minutes rather than performing a full-system restore, minimizing disruption to other users.

• Categorize every data set by criticality and assign a restore-time objective to each, so you can back up the most sensitive data more frequently without spending equally on everything.
• Use the same inventory that drives your infrastructure (Ansible, Kube, AWS) to drive your backup policies, preventing drift between what you think is protected and what actually is.
• Implement a reconciliation loop that continuously compares your backup policies against reality, with automated alerts or CI failures when a resource evades coverage.
• Schedule regular restore validation - actually restore from backup and verify the data - because "the restore and validation that a restore would work is much more painful than the actual backup" and is therefore usually skipped.
• Do not treat cloud storage or history buckets as a backup; maintain a separate, encrypted, deduplicated copy at a different provider that you control and can restore independently.