Rebuilding a Small k3s Host Without Making It Precious
Notes from wiping an old single-node k3s setup and rebuilding it with a smaller, cleaner baseline for ingress, certificates, and persistent storage.
Tag: kubernetes
Rails Performance from the Metal Up: A Field Guide
A practical Rails performance field guide for diagnosing CPU, IO, GVL, GC, memory, Puma, and container-runtime bottlenecks before choosing a fix.
Giving Hermes Read-Only Access to GCP, AWS, and Kubernetes
A practical setup giving an AI agent read-only access to debug QA and production on GCP, AWS, GKE, and EKS without touching workloads or secrets.
Zombie PostgreSQL replicas: the WAL segment removed deadlock
PostgreSQL replicas on Kubernetes can get stuck in a WAL segment removed loop that passes liveness checks, and three fixes stop it for good.
Production Ruby Debugging: A Progressive Escalation Guide
A progressive escalation guide to debugging Ruby applications in Kubernetes production without restarting: rbspy, kubectl debug, signals, rbtrace, and gdb.
Redis High Availability: Shared Sessions and Fault Tolerance
How to split Redis into isolated session, cache, and Sidekiq instances so a cache blip never logs users out or stops background jobs.
Redis Cleanup and Memory Management in Kubernetes
A production-tested guide to keeping Redis healthy in Kubernetes: memory control, safe cleanup patterns, Sidekiq pitfalls, backups, and troubleshooting.
Zero-Downtime GKE Upgrades with PostgreSQL in the Cluster
A blue/green node pool runbook for upgrading GKE clusters with PostgreSQL running as a StatefulSet, without a maintenance window.
Zero-downtime GKE upgrades with PostgreSQL failover
Blue/green node pools plus a Kubegres replica promotion let you upgrade a GKE cluster, PostgreSQL included, without a maintenance window.
PostgreSQL MD5 to SCRAM-SHA-256: a zero-downtime migration
PostgreSQL 18 deprecates MD5 authentication, and migrating to SCRAM-SHA-256 in mixed mode lets you switch pg_hba.conf without downtime or forced app changes.
Enabling Local HTTPS for Development and Debugging
Local HTTPS matters because browser APIs like Service Workers require a secure context, and this covers Caddy, Nginx with mkcert, Puma-dev, and Cert-Manager.
Troubleshooting pgpool Connection Termination in Kubernetes
Intermittent pgpool connection drops in Kubernetes usually trace to max_connections, idle timeouts, or NetworkPolicy issues, found layer by layer.
GKE's Two Health Check Systems and the Premature 502
GKE deployments can pass every readiness probe and still serve 502s, because Kubernetes probes and the Cloud Load Balancer run two separate health checks.
Redis Sentinel Split-Brain: Fixing It on Kubernetes
How Redis Sentinel split-brain happens during Kubernetes node maintenance, and the sentinel tuning, anti-affinity, and PDB changes that stopped it.
Kubernetes Node Upgrades: Cordon, Drain, and Failover
A cordon-drain-upgrade runbook for Kubernetes nodes, plus the PodDisruptionBudgets, priority classes, and lifecycle hooks that prevent outages.
Upgrading Bitnami Redis Helm Charts Without Downtime
A runbook for upgrading Bitnami's Redis Helm chart in Kubernetes, covering node migration, debug dry runs, and a tested rollback plan.
Kaniko: building container images without a Docker daemon
Kaniko builds and pushes container images without a Docker daemon, and the setup for it differs across GitHub Actions, Google Cloud Build, and CircleCI.
Linux kill signals: graceful shutdown in containerized workers
SIGTERM, SIGKILL, and grace periods control whether a containerized worker finishes its job before Kubernetes or Supervisord kills it outright.
Kubectl aliases and functions that actually save time
A set of kubectl aliases and shell functions for pod access, JSONPath queries, and log searching that cut real typing out of day-to-day Kubernetes work.
k3s on Ubuntu: Install, kubectl Access, and Unauthorized Errors
How to install k3s on Ubuntu, connect with kubectl, and fix the 'You must be logged in to the server (Unauthorized)' error caused by kubeconfig or expired certs.
Self-Hosted Whisper: A Speech-to-Text API with Ruby and Kubernetes
How to run OpenAI's Whisper as a self-hosted speech-to-text REST API with Docker, a Ruby client, and a Kubernetes deployment for transcribing audio.
How to Rotate GKE Cluster Credentials
The gcloud commands for checking certificate expiry, starting rotation, upgrading nodes to the new certificate, and completing GKE credential rotation.
Container Security with Fine-Grained Permission Control
How Linux file permissions and Kubernetes securityContext settings enforce least-privilege access for containerized workloads, with a worked example.
Rails in Kubernetes: Migrations, Jobs, and Puma
Running a Ruby on Rails app in Kubernetes means containerizing it correctly and handling migrations, background jobs, and Puma as separate concerns.
Production Debugging Techniques for Rails Apps
Practical production debugging techniques for Rails: log analysis, replicating production data, rbtrace live injection, feature-flagged debug modes, and replica apps.
Postgres 'Too Many Connections': Debugging Pgpool-II Limits
Debugging a Postgres 'too many connections' error through Pgpool-II's num_init_children and max_pool settings, and the fix that stopped clients queueing.
Scaling PostgreSQL on Kubernetes with Kubegres
How Kubegres brings primary/standby failover, automatic backups, and PITR to a self-hosted PostgreSQL cluster on Kubernetes without a managed database bill.
GCP 502 Errors from GKE Readiness Probe Failures
A GKE 502 with 'failed_to_pick_backend' traced back to readiness probes failing on every pod at once, and the liveness probe and anti-affinity fixes that resolved it.
Pgpool-II: Middleware for PostgreSQL
Pgpool-II sits between PostgreSQL and its clients as a connection pooler and load balancer, and this covers the Kubernetes setup that keeps writes consistent.
When the OOM Killer Silently Kills a Background Job
A missed notification email traced back to the Linux OOM killer terminating a Sidekiq job mid-run, leaving a stuck Redis lock and no error logged anywhere.