12 Nov 2026 Marian Kamenistak on Product Management for Software Engineers
As AI automates coding, engineers must master product management, focusing on customer value, rigorous validation, and business outcomes to remain valuable.
More Software Engineering Radio episodes
Published 2 Jul 2026
Duration: 01:02:16
The *polars* library offers a high-performance, multi-language (Python, R, Node.js) data frame tool leveraging Rust, columnar storage, out-of-core processing, and GPU acceleration for fast data manipulation, optimized for large datasets and scalable data engineering workflows.
Jeroen Janssens, a senior developer relations engineer at Posit, and Thijs Nieuwdorp, a developer relations engineer at Polars, speak with host Gregor...
The podcast discusses the polars library, a high-performance data frame implementation written in Rust with language bindings for Python, R, and Node.js. Designed for speed, it outperforms alternatives like pandas and has evolved over five years through iterative improvements. The library emphasizes a columnar storage model based on Apache Arrow, which enhances analytical efficiency by allowing selective column access, unlike traditional row-based storage. Key features include out-of-core processing for large datasets via streaming and spill-to-disk mechanisms, along with a declarative API that aligns with SQL-like logic, avoiding procedural row-wise operations. Polars supports advanced data transformations such as joins, aggregations, and pivot/unpivot operations, tailored for tidy data workflows. Its declarative approach reduces complexity, with expressions forming the foundation for data manipulation, enabling scalable and readable code through left-to-right operation chaining.
The discussion also highlights polars' use cases in data engineering and science, driven by its performance and intuitive design, with examples like aggregating sales data using group-by and sum operations. It contrasts with pandas in programming style, favoring declarative definitions over procedural step-by-step commands. Polars supports both lazy and eager execution modes, with lazy mode optimizing query plans for large-scale data. The library's ecosystem includes a dedicated book authored by industry practitioners and ongoing development of cloud-native, distributed computing capabilities. Performance benchmarks demonstrate significant reductions in computation time and cost, as seen in a case study where migrating from pandas to polars cut processing costs by 98%. Additionally, polars handles specialized tasks like time-series analysis, vector similarity searches, and GPU acceleration for join-heavy queries, though row-wise operations are discouraged in favor of columnar processing for optimal efficiency.
What if you rewrote your current data pipelines using Polars' declarative API for faster processing?
What if you tested Polars' out-of-core processing with spill-to-disk on a dataset exceeding your systems RAM?
scan_parquet method, enable spill-to-disk, and perform transformations like filtering, grouping, or pivoting. Monitor memory usage in task manager or htop.What if you migrated join-heavy queries to a GPU environment to exploit Polars vectorization and parallelism?
PL.lazy() for query planning.Leverage Polars' Lazy API for Query Optimization
Use the lazy execution mode to delay computations until the end of a workflow, allowing the internal query optimizer to generate an efficient execution plan. This reduces memory usage and improves performance for large datasets, especially when working with formats like Parquet.
Adopt Columnar Storage Formats (e.g., Parquet)
Prioritize Parquet or other columnar formats for large datasets to take advantage of Polars' columnar processing capabilities. This reduces memory overhead and accelerates analytical queries by enabling selective column access and compression.
Use Declarative Expressions for Data Manipulation
Replace row-wise operations with Polars' declarative API, which uses expressions (e.g., PL.call("X") + PL.call("Y")) to define transformations. This improves readability, scalability, and performance by avoiding temporary variables and enabling efficient column-wise operations.
Enable Spill-to-Disk for Memory-Intensive Workloads
Configure Polars to spill intermediate results to disk when working with datasets larger than available memory. This allows processing of larger data volumes, though it may slightly increase runtime due to disk I/O overhead.
Benchmark and Migrate from Pandas for Cost Efficiency
Identify performance-critical workflows and benchmark them using Polars against Pandas. Migrate to Polars for significant speedups (e.g., 30 seconds to 1.2 seconds) and cost reductions (e.g., 150k to 3.5k), especially for large-scale data processing and transformations.
12 Nov 2026 Marian Kamenistak on Product Management for Software Engineers
As AI automates coding, engineers must master product management, focusing on customer value, rigorous validation, and business outcomes to remain valuable.
29 Oct 2026 Danilo Krummrich on Rust in the Linux Kernel
"Nova GPU driver project integrates Rust into the Linux kernel to replace the aging Novo driver, leveraging Rust's memory safety for better reliability and performance while addressing firmware and architectural challenges through incremental, collaborative updates."
7 Oct 2026 Sahil Walia on Schema Changes and Data Migrations
Recommended: Schema as source of truth
"Schemas define data structure, apply across systems, and require careful evolution to avoid breaking changes, with best practices including additive updates, blue-green deployments, and stable identifiers."
30 Sept 2026 Raju Dandigam on Building Production AI Agents
"Explores production AI agents, their autonomy, safety, and engineering best practices, comparing workflows, tool design, orchestration, and memory systems while addressing challenges like cost, latency, and reliability."
16 Sept 2026 Milan Milanovic on the Laws of Software Engineering
"Explores key software engineering principles (like Conway's Law, Brooks' Law) and their impact on systems, teams, and decision-making, emphasizing context-dependent trade-offs, AI's role, and practical applications like measuring technical debt."