Building resilient systems with Sam Newman

The podcast discusses core principles of building resilient distributed systems, emphasizing that microservices should be considered an architecture of last resort due to their complexity. The conversation explores the four dimensions of resilience - robustness, rebound, graceful extensibility, and sustained adaptability - and highlights the importance of learning from failures, both internally and across the industry. Observability, idempotency, and the trade-offs in system design (such as fail-open vs. fail-close strategies) are examined in the context of real-world operations, with examples from companies like Uber, Netflix, and Cloudflare.

The discussion extends to the role of AI in software development, cautioning against cognitive surrender and over-reliance on automation. While AI can assist with code generation and research, it introduces risks related to non-determinism, reduced critical thinking, and potential erosion of expertise. The importance of maintaining human oversight, modular system design, and shared mental models is stressed. The conversation also covers practical strategies for adopting AI incrementally, ensuring system observability, and aligning technical decisions with business context, particularly through the use of SLOs, threat modeling, and thoughtful architectural planning.

• What if you treated your next software project as a resilience experiment?

  • Move: Design the system with explicit idempotency keys for all external operations (e.g., payments, notifications) and implement observability around retry patterns using OpenTelemetry.
  • Why Now?: Distributed failures and retry storms are increasingly common in solo-run systems due to reliance on third-party APIs and cloud services; catching duplication early prevents costly bugs.
  • Expected Upside: Reduce data inconsistency and customer-facing errors by 70%+ during network hiccups, while building trust through predictable behavior under stress.

• What if you shifted from writing code to curating specs as your primary output?

  • Move: Use a spec-first workflow (e.g., with SpecKit or custom Markdown templates) to define inputs, outputs, and SLOs before generating any code - then validate implementation against the spec in CI.
  • Why Now?: AI tools make code generation easy but risky; anchoring development in verifiable specs ensures alignment with business outcomes and avoids cognitive surrender.
  • Expected Upside: Cut debugging time by 50% and increase feature delivery speed by reducing rework, while maintaining clarity on "what good looks like" in production.

• What if you modularized your existing codebase to isolate AI-generated components?

  • Move: Refactor one core module (e.g., email processing or data export) into a standalone, well-tested unit with strict boundaries, then experiment with AI-generated logic inside it, monitored via SLOs and traceability.
  • Why Now?: Blind adoption of AI code leads to technical and cognitive debt; modular isolation allows safe experimentation without compromising system integrity.
  • Expected Upside: Gain efficiency from automation while retaining control and observability, enabling faster iteration with lower risk of cascading failures.

• Prioritize building observability into systems from the start using tools like OpenTelemetry to gain actionable insights from production behavior, which serves as the ultimate source of truth.
• Implement idempotency using unique keys (e.g., UUIDs) in APIs to prevent unintended side effects from retries, especially in payment or booking workflows.
• Adopt a modular architecture by refactoring code incrementally with strong test coverage, focusing on encapsulation and information hiding to improve maintainability and enable safer automation.
• Define clear Service Level Objectives (SLOs) aligned with business needs - such as response time and error rates - before implementing resilience measures like retries or multi-region deployment.
• Hedge against AI unreliability by using deterministic code for critical logic, validating AI-generated outputs, and avoiding cognitive surrender through active human review and oversight.

Treat microservices as an architecture of last resort due to the significant complexity they introduce in state management and distribution. Their primary benefit is enabling higher team autonomy, not superior technical performance.

Resilience is comprised of four dimensions: robustness, rebound, graceful extensibility, and sustained adaptability. Do not focus solely on robustness; recovery and adapting to surprise are equally critical.

Treat the production environment as the ultimate source of truth for system behavior. Specifications and code are secondary artifacts compared to the actual running system.

Use idempotency keys (unique client-generated identifiers) for new systems to enable safe retry of operations. Use fingerprinting (hashing input payloads) as a fallback strategy for retrofitting idempotency into legacy systems.

Avoid cognitive surrender by critically evaluating all AI-generated outputs. Over-reliance on AI tools without human oversight can erode team expertise and understanding of the system.

Hedge against AI vendor lock-in by using models and services from multiple providers. Prioritize generating deterministic code over using non-deterministic AI workflows for critical business logic.

Base the decision to fail open or closed on the specific business context and feature. E-commerce systems can often fail open and reconcile later, while systems handling finite resources (like tickets) must fail closed.

Distinguish between types of thundering herds (self-inflicted, retry storms, success-driven) to apply the correct mitigation. Mitigations include sensible retry policies with backoff, gradual rollouts, and caching strategies.

Internalize the three rules of distributed systems: latency is inevitable, components will be unavailable, and resource pools are finite. Resource saturation is the root cause of most system outages.

Prioritize modular architecture with strict encapsulation and information hiding to manage complexity effectively. This principle applies whether using a monolith or microservices.

Invest in observability (traces, spans, and metrics) from the project's inception. It is impossible to understand and debug distributed system behavior without robust telemetry.

Recognize software development as a collective, relational endeavor requiring shared mental models. Preventing cognitive debt in a team is just as important as managing technical debt.

Explore spec-driven development and clearly define "what good looks like" before automating. Specifications may become the new source of truth as AI abstracts away low-level coding.

Align resilience strategies and technical trade-offs directly with business priorities and revenue drivers. Engineers must understand the business impact of their technical decisions.

Microservices are an opinionated evolution of SOA defined by independent deployability and business-driven boundaries. Rebadging existing layered architectures as microservices fails to deliver the intended benefits.