When Ubuntu Builds Fail on libvips but macOS Does Not
Why a Rails test suite can fail on Ubuntu because of an older native libvips, and how to upgrade it safely without confusing it with the ruby-vips gem.
DevOps and Reliability
Systems that stay up, deploy safely, and can be explained when they fail.
The problem
Delivery has become slow or risky. Deployments need a hero, incidents repeat, and the database is the part nobody wants to touch.
What I offer
- Kubernetes, PostgreSQL, and cloud infrastructure that is documented, tested, and reversible
- CI pipelines that fail for real reasons and pass deterministically
- Observability, backups, and recovery paths that have actually been exercised
- Security hardening grounded in audits, not checklists
Start here
- Zombie PostgreSQL replicas: the WAL segment removed deadlock · PostgreSQL replicas on Kubernetes can get stuck in a WAL segment removed loop that passes liveness checks, and three fixes stop it for good.
- I Audited the OpenVPN Server I Wrote a Guide For · Two years after publishing a guide to restricting OpenVPN clients to specific IPs, I audited the server it built, and most of the findings trace back to the guide.
- Kaniko: building container images without a Docker daemon · Kaniko builds and pushes container images without a Docker daemon, and the setup for it differs across GitHub Actions, Google Cloud Build, and CircleCI.
All writing in this area (97)
Tightening a dev VM that was quietly living without a firewall
What a routine security audit found on a small always-on dev VM: passwordless databases, no host firewall, over-privileged jobs, and how it got fixed.
When an Always Free server gets smaller
What Oracle's smaller Ampere A1 Always Free limits mean for a self-hosted ARM server, and how I redesigned mine around a 12 GB, 2-core budget.
Rebuilding a Small k3s Host Without Making It Precious
Notes from wiping an old single-node k3s setup and rebuilding it with a smaller, cleaner baseline for ingress, certificates, and persistent storage.
Tightening Hermes Secrets with Bitwarden
A practical walkthrough for moving Hermes API keys out of local env files and into Bitwarden Secrets Manager, with verification and cleanup steps.
Shared AI Artifacts Aren't Private From a Hard-to-Guess URL
A security note on the Claude shared-chat and Artifacts leak, and why AI workspaces need clear boundaries between private, unlisted, and public content.
Voice Control for Hermes Agent over Telegram
How I wired Hermes Agent to accept Telegram voice notes, transcribe them locally, and reply with local text-to-speech without adding another paid speech API.
Tightening Hermes Agent Security for Local Sensitive Work
How I tightened my Hermes agent setup after adding a local model for PII and credential-sensitive work: tool limits, manual approvals, and a quiet watchdog.
Securing Hermes Agent with OpenShell
Why I run Hermes Agent inside OpenShell when it reads untrusted content, and how filesystem, network, process, and credential policy change the security model.
Giving Hermes Read-Only Access to GCP, AWS, and Kubernetes
A practical setup giving an AI agent read-only access to debug QA and production on GCP, AWS, GKE, and EKS without touching workloads or secrets.
Cloud Run as a Gemini proxy for an unsupported region
How I used Google Cloud Run as a small AI gateway so clients in an unsupported region can still call Gemini through a supported GCP region.
Zombie PostgreSQL replicas: the WAL segment removed deadlock
PostgreSQL replicas on Kubernetes can get stuck in a WAL segment removed loop that passes liveness checks, and three fixes stop it for good.
Why pg_dump backups on a PostgreSQL replica fail intermittently
Running pg_dump on a PostgreSQL replica can fail with conflict with recovery errors because the backup is a long read transaction colliding with WAL replay.
Postgres conflict with recovery: why replicas cancel queries
A PostgreSQL replica cancels long queries with conflict with recovery errors during VACUUM replication, and hot_standby_feedback fixes it without downtime.
Production Ruby Debugging: A Progressive Escalation Guide
A progressive escalation guide to debugging Ruby applications in Kubernetes production without restarting: rbspy, kubectl debug, signals, rbtrace, and gdb.
Incus vs. Docker: System Containers vs. App Containers
A practical comparison of Incus and Docker: when system containers suit development environments better than single-process application containers.
Redis High Availability: Shared Sessions and Fault Tolerance
How to split Redis into isolated session, cache, and Sidekiq instances so a cache blip never logs users out or stops background jobs.
Identity-Aware GCS Access Instead of Hashed Secret URLs
Replacing hashed 'secret' URLs in Google Cloud Storage with IAM Conditions and domain-restricted access that verifies identity instead of hiding a path.
How I Finally Got Sentry to Create GitHub Issues (After 3 Hours of Clicking Around)
How to fix Sentry's GitHub integration when it shows connected but issues won't create, plus why linking Sentry errors to GitHub issues is worth the setup.
Faster Rails Docker Feedback Loop: Runtime Deps and HMR
How I reduced Docker development friction across three Rails apps by removing rebuild-heavy steps, adding runtime dependency sync, and enabling webpack HMR.
Polymorphic Indexes in MySQL: Type-First or ID-First?
A practical comparison of [type, id] versus [id, type] composite index order for polymorphic associations in Rails and MySQL, and how to choose.
Redis Cleanup and Memory Management in Kubernetes
A production-tested guide to keeping Redis healthy in Kubernetes: memory control, safe cleanup patterns, Sidekiq pitfalls, backups, and troubleshooting.
Fixing the Asia/Rangoon Postgres Error on Alpine Linux
If you recently updated your Docker images and were greeted by a PG::InvalidParameterValue: ERROR: time zone "Asia/Rangoon" not recognized, you aren't alone.
Sending Email from an Ubuntu VM on Oracle Cloud (OCI)
A Postfix and OCI Email Delivery setup guide covering SMTP auth, sender approval, SPF/DKIM/DMARC, and the errors that block delivery.
Optimizing a 1C1G Ubuntu Cloud Server
A production-tested checklist for making a 1 vCPU, 1GB RAM Ubuntu server stable: swap, kernel tuning, service removal, and MySQL/PHP-FPM limits.
Zero-downtime GKE upgrades with PostgreSQL failover
Blue/green node pools plus a Kubegres replica promotion let you upgrade a GKE cluster, PostgreSQL included, without a maintenance window.
PostgreSQL on Ubuntu: install, user setup, and auth debugging
Install PostgreSQL from the PGDG repo on Ubuntu, set up a matching development role, and debug the password authentication failures Rails hits most.
PostgreSQL MD5 to SCRAM-SHA-256: a zero-downtime migration
PostgreSQL 18 deprecates MD5 authentication, and migrating to SCRAM-SHA-256 in mixed mode lets you switch pg_hba.conf without downtime or forced app changes.
OpenSSH on WSL: enabling remote SSH access on Windows 10
Install and configure OpenSSH inside WSL2, enable systemd, and forward a Windows port so you can SSH into your WSL environment from anywhere on your LAN.
From 499 to 200: Fixing Nginx Timeouts in a Rails API
How Puma worker timeouts and Nginx 499 errors expose slow Rails requests, and how to fix them with aligned timeouts, background jobs, and caching.
localhost vs 127.0.0.1 vs 0.0.0.0: Network Binding Explained
Why localhost, 127.0.0.1, 0.0.0.0, and * behave differently for MySQL clients and services, and how to find and fix a local port conflict between Docker and SSH.
Securing AWS Credentials: aws-vault vs 1Password
Storing long-lived AWS access keys in ~/.aws/credentials is a common anti-pattern; aws-vault and the 1Password AWS Shell Plugin both fix it with short-lived STS tokens.
Signed React Native Android APKs from GitHub Actions
A GitHub Actions pipeline that decodes a base64 keystore from repository secrets and produces a signed React Native Android APK on every push.
Optimizing Android Builds in GitHub Actions
Android builds in GitHub Actions run 10-30 minutes unoptimized; layered Gradle and Yarn caching plus scoped environment secrets get that down to a few minutes.
PostgreSQL 18: What's New and How to Upgrade
PostgreSQL 18 adds better query parallelism, parallel logical replication apply, and more flexible partitioning, plus the pg_upgrade and dump/restore paths to get there.
Migrating a GCP Classic Load Balancer to Global External
GCP lets you migrate a Classic Application Load Balancer to the Global External Load Balancer with a staged, four-state rollout that avoids downtime.
Enabling Local HTTPS for Development and Debugging
Local HTTPS matters because browser APIs like Service Workers require a secure context, and this covers Caddy, Nginx with mkcert, Puma-dev, and Cert-Manager.
Designing a Schema for an Optional Attribute
A one-line feature request, recording why a student failed a course, walks through three schema options: a nullable column, a normalized table, and jsonb.
Troubleshooting pgpool Connection Termination in Kubernetes
Intermittent pgpool connection drops in Kubernetes usually trace to max_connections, idle timeouts, or NetworkPolicy issues, found layer by layer.
User Impersonation in Rails with the Pretender Gem
User impersonation lets an admin temporarily act as another user for debugging and support, and the pretender gem handles the current_user switch and session state.
GKE's Two Health Check Systems and the Premature 502
GKE deployments can pass every readiness probe and still serve 502s, because Kubernetes probes and the Cloud Load Balancer run two separate health checks.
Speeding Up Docker: Fixing Slow Host-to-Container File Sync
Bind-mounted host directories make Docker Desktop file access painfully slow on macOS and Windows, and this covers volumes, docker-sync, Mutagen, and WSL 2 fixes.
Enabling Gzip and Brotli Compression on Google Cloud CDN
Google Cloud CDN can compress responses with Brotli or Gzip automatically, covering the compressionMode setting, cache invalidation, and verifying it works.
Managing Temporary Data in Multi-Step Rails Forms
Multi-step Rails forms need somewhere to hold data between steps, and this compares session storage, direct Redis caching, and temporary DB rows for that job.
PostgreSQL JSONB vs Linking Tables for Relationships
Comparing PostgreSQL jsonb columns against a traditional linking table for modeling course prerequisites, and where a hybrid of both makes more sense.
MySQL vs PostgreSQL: Foreign Key Deadlock Contention
How foreign key checks cause deadlock contention in MySQL InnoDB versus PostgreSQL, and why PostgreSQL's FOR KEY SHARE lock reduces the risk considerably.
Idempotent Rails Requests with a Redis Semaphore Lock
How to stop duplicate Rails requests from creating the same record twice, using a Redis-backed semaphore lock instead of chasing a race condition.
Redis Sentinel Split-Brain: Fixing It on Kubernetes
How Redis Sentinel split-brain happens during Kubernetes node maintenance, and the sentinel tuning, anti-affinity, and PDB changes that stopped it.
Build and Push Docker Images to Quay.io with GitHub Actions
A working GitHub Actions workflow that builds a Docker image and pushes it to Quay.io using a scoped robot account and repository secrets.
Why npm ci Runs prepare Scripts for Git Dependencies
npm ci is not the script-free install it looks like; it still runs prepare scripts for Git-based dependencies, which broke Husky in CI.
Deadlocks in MySQL and PostgreSQL: Foreign Keys vs Performance
How deadlocks form in MySQL and PostgreSQL under concurrent updates, and the trade-off between foreign key integrity and write throughput.
Catching Database Configuration Drift Between Microservices
Two services sharing a database can pass every test and still fail in production if their DB configuration silently drifts apart.
Detecting Cloudflare Zero Trust Beyond the WARP Flag
WARP can report itself as off while Zero Trust still routes your Mac's traffic; here is how to check DNS, ASN, and DoH to find out for sure.
Monitor Cloudflare WARP Status on macOS with launchd
A launchd-scheduled shell script that detects when Cloudflare WARP reconnects on a company-managed Mac and sends a desktop notification.
Kubernetes Node Upgrades: Cordon, Drain, and Failover
A cordon-drain-upgrade runbook for Kubernetes nodes, plus the PodDisruptionBudgets, priority classes, and lifecycle hooks that prevent outages.
When AWS Lambda Makes Sense for a Single API Endpoint
A decision framework for when AWS Lambda fits a single API endpoint, and when a container or dedicated server is the better call instead.
Ordering by an Associated Column Without Breaking includes
Why ordering ActiveRecord results by an associated table's column breaks with includes, and how references, Arel, and left_joins fix it for good.
Upgrading Bitnami Redis Helm Charts Without Downtime
A runbook for upgrading Bitnami's Redis Helm chart in Kubernetes, covering node migration, debug dry runs, and a tested rollback plan.
Migrating OTP Secrets Out of Google Authenticator
How to export TOTP secrets from Google Authenticator with the otpauth tool and import them into Authy, Aegis, or 1Password without losing account access.
Kaniko: building container images without a Docker daemon
Kaniko builds and pushes container images without a Docker daemon, and the setup for it differs across GitHub Actions, Google Cloud Build, and CircleCI.
ARG, ENV, and BuildKit secrets: how each behaves in Docker
ARG, ENV, exported shell variables, and BuildKit secret mounts each persist differently in a Docker image, and only one of them is actually safe for tokens.
Linux kill signals: graceful shutdown in containerized workers
SIGTERM, SIGKILL, and grace periods control whether a containerized worker finishes its job before Kubernetes or Supervisord kills it outright.
Retry mechanisms for flaky CI and Docker builds
Wrapping curl, bundle install, and Docker builds in retry logic turns transient network failures in CI into automatic retries instead of full pipeline restarts.
jq and yq: transforming embedded key-value strings in YAML
jq and yq combine to turn a YAML file's embedded key=value strings into properly structured, typed fields, using a single scan and from_entries pipeline.
Kubectl aliases and functions that actually save time
A set of kubectl aliases and shell functions for pod access, JSONPath queries, and log searching that cut real typing out of day-to-day Kubernetes work.
Docker secrets: how layers leak them and how BuildKit fixes it
Docker's layer history persists secrets baked into ARG and ENV instructions, and BuildKit secret mounts are the practical fix that keeps them out of the image.
Docker Buildx --on-error: drop into a shell when a build fails
The --on-error flag in Docker Buildx drops you into a shell inside the failing build step so you can inspect state instead of guessing from logs.
How to Debug Docker Image Builds
Practical techniques for debugging Docker image builds: BuildKit output flags, inspecting intermediate layers, nsenter tricks, and multi-stage build targets.
Leapp vs aws-vault: Comparing AWS CLI Credential Tools
A practical comparison of Leapp and aws-vault for managing AWS CLI credentials, covering SSO and IAM setup, session tokens, and when each tool fits better.
Running Multiple Processes in a Single Container
Six ways to run multiple processes in one Docker container, from GNU Parallel and Foreman to a custom entrypoint script, Supervisord, and tmux, with trade-offs.
OpenVPN: Restrict Client Access to Specific IPs Only
How to restrict OpenVPN clients to a limited set of destination IPs, using client-config-dir routes, a client-connect script, and iptables rules.
Fixing Multi-Tab Session Bugs with JWT and sessionStorage
How cookie-based session state breaks when a user opens multiple tabs in a single-page app, and how splitting auth into JWT plus sessionStorage fixes it.
k3s on Ubuntu: Install, kubectl Access, and Unauthorized Errors
How to install k3s on Ubuntu, connect with kubectl, and fix the 'You must be logged in to the server (Unauthorized)' error caused by kubeconfig or expired certs.
How to Rotate GKE Cluster Credentials
The gcloud commands for checking certificate expiry, starting rotation, upgrading nodes to the new certificate, and completing GKE credential rotation.
Container Security with Fine-Grained Permission Control
How Linux file permissions and Kubernetes securityContext settings enforce least-privilege access for containerized workloads, with a worked example.
Building and Deploying pgpool Images with GitHub Actions
A GitHub Actions workflow that builds multi-architecture pgpool images for several versions in parallel and pushes them to GitHub Container Registry.
Supervisord: Process Management for Containers and VMs
How supervisord monitors and restarts UNIX processes through simple INI-style config, with a working example for MySQL, PHP, and an init script.
Common MySQL Commands: Databases, Users, and Grants
A quick-reference list of MySQL and MariaDB commands for databases, tables, users, and grants, kept for copy-pasting from the shell.
Rails in Kubernetes: Migrations, Jobs, and Puma
Running a Ruby on Rails app in Kubernetes means containerizing it correctly and handling migrations, background jobs, and Puma as separate concerns.
Materialized Views: Speeding Up Queries with a Tradeoff
Materialized views cache expensive PostgreSQL queries as physical tables, trading storage and refresh cost for fast reads, plus how pgpool fits in.
GitHub Actions Variables: env, Contexts, Secrets, and Matrix
How GitHub Actions variables work in practice: env blocks, context variables, secrets, matrix strategies, and passing step outputs between steps.
PostgreSQL Sequence Conflicts After an Upgrade: How to Fix Them
Upgrading PostgreSQL can leave sequences out of sync with table max values, causing duplicate-key errors and blocked sign-ins, fixed here with SQL and bash.
Zero-Downtime PostgreSQL Upgrades With Logical Replication
A bash script that sets up PostgreSQL logical replication, publications, and subscriptions to perform a major-version upgrade without downtime.
Redis Sentinel: How Automatic Failover Actually Works
How Redis Sentinel provides high availability: quorum-based monitoring, master election, automatic failover, and what your client applications must do to follow along.
Rails Secrets: Credentials, 1Password, and Cloud Vaults
A rundown of Rails credentials, environment variables, 1Password CLI, and cloud secret managers like AWS Secrets Manager and Vault, and when to use each.
Config Approaches: Files, Env Vars, and Config Servers
A comparison of YAML config files, Rails encrypted credentials, environment variables, and centralized config servers, with the trade-offs of each approach.
Postgres 'Too Many Connections': Debugging Pgpool-II Limits
Debugging a Postgres 'too many connections' error through Pgpool-II's num_init_children and max_pool settings, and the fix that stopped clients queueing.
Redis in Ruby on Rails: Cache Server and Job Queue
Redis serves two very different roles in a Rails app, a disposable read cache and a durable job queue, and conflating the two causes real outages.
Scaling PostgreSQL on Kubernetes with Kubegres
How Kubegres brings primary/standby failover, automatic backups, and PITR to a self-hosted PostgreSQL cluster on Kubernetes without a managed database bill.
GCP 502 Errors from GKE Readiness Probe Failures
A GKE 502 with 'failed_to_pick_backend' traced back to readiness probes failing on every pod at once, and the liveness probe and anti-affinity fixes that resolved it.
Pgpool-II: Middleware for PostgreSQL
Pgpool-II sits between PostgreSQL and its clients as a connection pooler and load balancer, and this covers the Kubernetes setup that keeps writes consistent.
When the OOM Killer Silently Kills a Background Job
A missed notification email traced back to the Linux OOM killer terminating a Sidekiq job mid-run, leaving a stuck Redis lock and no error logged anywhere.
Soft Delete in Rails: Discard vs ActsAsParanoid
A comparison of Discard and ActsAsParanoid, the two common Rails soft-delete gems, and the query and unscope pitfalls each one introduces.
Database Auditing: pg_audit vs audited vs paper_trail
A comparison of database auditing options, pg_audit, audited, paper_trail, Hibernate Envers, and SQL Server temporal tables, and when each one fits.
Defending Against DoS Attacks
How to defend against DoS and DDoS attacks with request throttling, input validation, firewall rules, load balancers, and cloud-based mitigation.
count file size from json
A bash one-liner that uses jq to pull resource IDs out of lesson JSON files, then du to total their file sizes on disk.
NETSCREEN 5GT密码破解方法
恢复 NetScreen 5GT 密码的两种方法：通过 Console 输入序列号初始化，或用重置针孔硬复位，以及导入旧配置文件前的编辑注意事项。