Published 23 Jan 2026
Duration: 2479
Apache Spark's development and significance in data processing is analyzed, focusing on its efficiency, flexibility, and challenges in AI and machine learning tasks.
This interview was recorded for GOTO Unscripted. https://gotopia.tech Check out more here: https://gotopia.tech/articles/410 Holden Karau - Co-Founder...
The podcast explores the development and current role of Apache Spark in modern data processing, emphasizing its efficiency over older technologies like MapReduce. It discusses how Spark has evolved to support AI and machine learning workloads, addressing challenges such as data skew, fault tolerance in ML training, and the integration of AI tools with data processing frameworks. The discussion also highlights Spark's flexibility and distributed computing features, along with strategies for optimizing performance.
Additionally, the podcast touches on a real-world project that uses AI to help individuals appeal denied health insurance claims. This involves analyzing reasons for denials and generating appeals, while addressing challenges related to data quality, the complexity of insurance systems, and the sustainability of open source projects. The conversation also covers technical aspects such as GPU utilization, checkpointing, and resource management in the context of training machine learning models.
14 Aug 2026 The Platform Engineers Handbook Ajay Chankramath & Kaspar von Grunberg
"Explores platform engineering's evolution, practical applications, and challenges, emphasizing DevEx, AI integration, and long-term infrastructure investment to enhance software development efficiency and scalability."
7 Aug 2026 State of the Art of Platform Engineering Abby Bangser & Charles Humble
"Explores generative AI's impact on software engineering, platform evolution, and best practices for scalable, maintainable platforms while addressing AI's role in automation, reliability, and trust."
31 Jul 2026 The C4 Model: Visualizing Software Architecture Simon Brown & Susanne Kaiser
The C4 model is a lightweight, structured approach to visualizing software architecture with four levels (System Context, Container, Component, and Code) that prioritizes simplicity, traceability, and effective communication among teams.
17 Jul 2026 Signals & Levers Elisabeth Hendrickson, Joel Tosi & Charles Humble
"Modern software development faces persistent challenges, requiring systems thinking to address dynamic problems, human factors, and cultural shifts beyond Agile and DevOps limitations."
10 Jul 2026 The Architect's Guide to the AI Era Luca Mezzalira & Teena Idnani
"AI accelerates coding and research but requires human expertise for system design, governance, and business alignment, with architects balancing innovation, ethics, and practical constraints to ensure AI augmentsnot replacestraditional engineering."