KubeFM

Latest Episodes:

Performance testing Kubernetes workloads, with Stephan Schwarz May 27, 2025

Show notes

If you're tasked with performance testing Kubernetes workloads without much guidance, this episode offers clear, experience-based strategies that go beyond theory.

Stephan Schwarz, a DevOps engineer at iits-consulting, walks through his systematic approach to performance testing Kubernetes applications. He covers everything from defining what performance actually means, to the practical methodology of breaking individual pods to understand their limits, and navigating the complexities of Kubernetes-specific components that affect test results.

You will learn:

How to establish baseline performance metrics by systematically testing individual pods, disabling autoscaling features, and documenting each incremental change to understand real application limits
Why shared Kubernetes components skew results and how ingress controllers, service meshes, and monitoring stacks create testing challenges that require careful consideration of the entire request chain
Practical approaches to HPA configuration, including how to account for scaling latency, the time delays inherent in Kubernetes scaling operations, and planning for spare capacity based on your SLA requirements
The role of observability tools like OpenTelemetry in production environments where load testing isn't feasible, and how distributed tracing helps isolate performance bottlenecks across interdependent services

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/yY-FnmGfH
Interested in sponsoring an episode? Learn more.

Managing 100s of Kubernetes Clusters using Cluster API, with Zain Malik May 20, 2025

Show notes

Discover how to manage Kubernetes at scale with declarative infrastructure and automation principles.

Zain Malik shares his experience managing multi-tenant Kubernetes clusters with up to 30,000 pods across clusters capped at 950 nodes. He explains how his team transitioned from Terraform to Cluster API for declarative cluster lifecycle management, contributing upstream to improve AKS support while implementing GitOps workflows.

You will learn:

How to address challenges in large-scale Kubernetes operations, including node pool management inconsistencies and lengthy provisioning times
Why Cluster API provides a powerful foundation for multi-cloud cluster management, and how to extend it with custom operators for production-specific needs
How implementing GitOps principles eliminates manual intervention in critical operations like cluster upgrades
Strategies for handling production incidents and bugs when adopting emerging technologies like Cluster API

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/5PLksqVlk
Interested in sponsoring an episode? Learn more.

Super-Scaling Open Policy Agent with Batch Queries, with Nicholaos Mouzourakis May 13, 2025

Show notes

Dive into the technical challenges of scaling authorization in Kubernetes with this in-depth conversation about Open Policy Agent (OPA).

Nicholaos Mouzourakis, Staff Product Security Engineer at Gusto, explains how his team re-architected Kubernetes native authorization using OPA to support scale, latency guarantees, and audit requirements across services. He shares detailed insights about their journey optimizing OPA performance through batch queries and solving unexpected interactions between Kubernetes resource limits and Go's runtime behavior.

You will learn:

Why traditional authorization approaches (code-driven and data-driven) fall short in microservice architectures, and how OPA provides a more flexible, decoupled solution
How batch authorization can improve performance by up to 18x by reducing network round-trips
The unexpected interaction between Kubernetes CPU limits and Go's thread management (GOMAXPROCS) that can severely impact OPA performance
Practical deployment strategies for OPA in production environments, including considerations for sidecars, daemon sets, and WASM modules

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/S-2vQ_j-4
Interested in sponsoring an episode? Learn more.

Kubernetes upgrades: beyond the one-click update, with Tanat Lokejaroenlarb May 06, 2025

Show notes

Discover how Adevinta manages Kubernetes upgrades at scale in this episode with Tanat Lokejaroenlarb. Tanat shares his team's journey from time-consuming blue-green deployments to efficient in-place upgrades for their multi-tenant Kubernetes platform SHIP, detailing the engineering decisions and operational challenges they overcame.

You will learn:

How to transition from blue-green to in-place Kubernetes upgrades while maintaining service reliability
Techniques for tracking and addressing API deprecations using tools like Pluto and Kube-no-trouble
Strategies for minimizing SLO impact during node rebuilds through serialized approaches and proper PDB configuration
Why a phased upgrade approach with "cluster waves" provides safer production deployments even with thorough testing

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/VVHFfXGl_
Interested in sponsoring an episode? Learn more.

From Fragile to Faultless: Kubernetes Self-Healing In Practice, with Grzegorz Głąb Apr 29, 2025

Show notes

Discover how to build resilient Kubernetes environments at scale with practical automation strategies from an engineer who's tackled complex production challenges.

Grzegorz Głąb, Kubernetes Engineer at Cloud Kitchens, shares his team's journey developing a comprehensive self-healing framework. He explains how they addressed issues ranging from spot node preemptions to network packet drops caused by unbalanced IRQs, providing concrete examples of automation that prevents downtime and improves reliability.

You will learn:

How managed Kubernetes services like AKS provide benefits but require customization for specific use cases
The architecture of an effective self-healing framework using DaemonSets and deployments with Kubernetes-native components
Practical solutions for common challenges like StatefulSet pods stuck on unreachable nodes and cleaning up orphaned pods
Techniques for workload-level automation, including throttling CPU-hungry pods and automating diagnostic data collection

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/yg_fkP0LN
Interested in sponsoring an episode? Learn more.

Replacing StatefulSets with a custom Kubernetes operator in our Postgres cloud platform, with Andrew Charlton Apr 22, 2025

Show notes

Discover why standard Kubernetes StatefulSets might not be sufficient for your database workloads and how custom operators can provide better solutions for stateful applications.

Andrew Charlton, Staff Software Engineer at Timescale, explains how they replaced Kubernetes StatefulSets with a custom operator called Popper for their PostgreSQL Cloud Platform. He details the technical limitations they encountered with StatefulSets and how their custom approach provides more intelligent management of database clusters.

You will learn:

Why StatefulSets fall short for managing high-availability PostgreSQL clusters, particularly around pod ordering and volume management
How Timescale's instance matching approach solves complex reconciliation challenges when managing heterogeneous database workloads
The benefits of implementing discrete, idempotent actions rather than workflows in Kubernetes operators
Real-world examples of operations that became possible with their custom operator, including volume downsizing and availability zone consolidation

Sponsor

This episode is brought to you by mirrord — run local code like in your Kubernetes cluster without deploying first.

More info

Find all the links and info for this episode here: https://ku.bz/fhZ_pNXM3
Interested in sponsoring an episode? Learn more.

Saving 10s of thousands of dollars deploying AI at scale with Kubernetes, with John McBride Mar 18, 2025

Show notes

Curious about running AI models on Kubernetes without breaking the bank? This episode delivers practical insights from someone who's done it successfully at scale.

John McBride, VP of Infrastructure and AI Engineering at the Linux Foundation shares how his team at OpenSauced built StarSearch, an AI feature that uses natural language processing to analyze GitHub contributions and provide insights through semantic queries. By using open-source models instead of commercial APIs, the team saved tens of thousands of dollars.

You will learn:

How to deploy VLLM on Kubernetes to serve open-source LLMs like Mistral and Llama, including configuration challenges with GPU drivers and daemon sets
Why smaller models (7-14B parameters) can achieve 95% effectiveness for many tasks compared to larger commercial models, with proper prompt engineering
How running inference workloads on your own infrastructure with T4 GPUs can reduce costs from tens of thousands to just a couple thousand dollars monthly
Practical approaches to monitoring GPU workloads in production, including handling unpredictable failures and VRAM consumption issues

Sponsor

This episode is brought to you by StackGen! Don't let infrastructure block your teams. StackGen deterministically generates secure cloud infrastructure from any input - existing cloud environments, IaC or application code.

More info

Find all the links and info for this episode here: https://ku.bz/wP6bTlrFs
Interested in sponsoring an episode? Learn more.

I just want mTLS on Kubernetes, with John Howard Mar 04, 2025

Show notes

Dive into the world of Kubernetes security with this insightful conversation about securing cluster traffic through encryption.

John Howard, Senior Software Engineer at Solo.io, explains the complexities of implementing Mutual TLS (mTLS) in Kubernetes. He discusses the evolution from DIY approaches to Service Mesh solutions, focusing on Istio's Ambient Mesh as a simplified path to workload encryption.

You will learn:

Why DIY mTLS implementation in Kubernetes is challenging at scale, requiring certificate management, application updates, and careful transition planning
How Service Mesh solutions offload security concerns from applications, allowing developers to focus on business logic while infrastructure handles encryption
The advantages of Ambient Mesh's approach to simplifying mTLS implementation with its node proxy and waypoint proxy architecture

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/sk-ZF1PG9
Interested in sponsoring an episode? Learn more.

Learned it the hard way: don't use Cilium's default Pod CIDR, with Isala Piyarisi Feb 25, 2025

Show notes

This episode examines how a default configuration in Cilium CNI led to silent packet drops in production after 8 months of stable operations.

Isala Piyarisi, Senior Software Engineer at WSO2, shares how his team discovered that Cilium's default Pod CIDR (10.0.0.0/8) was conflicting with their Azure Firewall subnet assignments, causing traffic disruptions in their staging environment.

You will learn:

How Cilium's default CIDR allocation can create routing conflicts with existing infrastructure
A methodical process for debugging network issues using packet tracing, routing table analysis, and firewall logs
The procedure for safely changing Pod CIDR ranges in production clusters

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/kJjXQlmTw
Interested in sponsoring an episode? Learn more.

Simplifying Kubernetes deployments with a unified Helm chart, with Calin Florescu Feb 18, 2025

Show notes

Managing microservices in Kubernetes at scale often leads to inconsistent deployments and maintenance overhead. This episode explores a practical solution that standardizes service deployments while maintaining team autonomy.

Calin Florescu discusses how a unified Helm chart approach can help platform teams support multiple development teams efficiently while maintaining consistent standards across services.

You will learn:

Why inconsistent Helm chart configurations across teams create maintenance challenges and slow down deployments
How to implement a unified Helm chart that balances standardization with flexibility through override functions
How to maintain quality through automated documentation and testing with tools like Helm Docs and Helm unittest

Sponsor

This episode is sponsored by LearnKube — get started on your Kubernetes journey through comprehensive online, in-person or remote training.

More info

Find all the links and info for this episode here: https://ku.bz/mcPtH5395
Interested in sponsoring an episode? Learn more.

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected