Open a pull request, get a website

Hello! This follows on from my homelab post, where I set up a small K3s cluster on two Raspberry Pis. It had been sitting there being tidy and useless, so I gave it a real job.

The job is this. You open a pull request. A minute or two later, a bot comments a link on it. You click the link and there is your branch, running as a real website, with its own database. You push another commit and the link updates itself, keeping the database. You merge the pull request and the whole thing quietly disappears.

Vercel and Netlify give you this for free, and Heroku called them Review Apps. I wanted to know what it takes to build one, so I built it on two Raspberry Pis sitting next to my router.

Why bother

The usual way to check a change is either "it works on my laptop" or a single shared staging server that somebody else is always using. A screenshot in a pull request is not the same as clicking around the real thing, and the shared server is permanently occupied by whoever asked first.

The catch is that "temporary" is the hard part. Creating things is easy. Making sure they reliably delete themselves, and that they cannot damage anything while they exist, is where the actual work is. That is really a question about multi-tenancy, resource limits, secret handling and garbage collection — all of which are much more interesting on 8 GB of RAM than they are on someone else's cloud account.

What it does

Underneath the pull request bot is a single command I can also run by hand:

k8i-env create --image ghcr.io/kaikiat/api:abc123 \
               --port 8080 --with postgres,redis --seed-sql ./seed.sql

  namespace : env-brave-otter-7f3a
  url       : http://env-brave-otter-7f3a.192-168-1-174.sslip.io
  expires   : 2026-08-25 14:02 SGT  (in 24h0m)
  logs      : https://logs.tail2990f9.ts.net/explore  ->  {namespace="env-brave-otter-7f3a"}
  ready in  : 41s
  http      : 200

You hand it a container image. It hands back a running app with a Postgres and a Redis next to it, generated credentials, seeded data, a URL, and an expiry.

Measured on the real cluster: 76s for a cold create including the image pull, 43s for a full set of extras once layers are cached, 8s for a bare image. Then it deletes itself 24 hours later.

The CLI itself is a POSIX shell script. It renders a kustomize overlay into a temp directory and runs kubectl apply -k, so the dependency list is kubectl and openssl and the templates stay valid YAML you can apply by hand when something misbehaves.

A namespace per environment

A namespace is Kubernetes' way of putting a fence around a group of objects, and here it does double duty as the record of the environment itself. There is no database and no custom resource — everything worth knowing is a label or an annotation on the namespace:

metadata:
  name: env-brave-otter-7f3a
  labels:
    k8i.dev/ephemeral: "true"                  # the only thing the cleanup job selects on
  annotations:
    k8i.dev/expires-at-epoch: "1787654400"
    k8i.dev/image: ghcr.io/kaikiat/api:abc123
    k8i.dev/addons: postgres,redis

Two small decisions here paid off. Expiry is stored as epoch seconds, so the cleanup job is an integer comparison in sh rather than date -d parsing that busybox and GNU coreutils disagree about. And the name stays under 30 characters, because it also has to work as a DNS label in the URL.

Deleting the namespace is the entire teardown. Deployments, Services, Ingresses, PVCs, Secrets and Jobs are all namespaced and garbage-collect with it. That is worth designing for deliberately — it means an environment must never create a cluster-scoped object, or teardown stops being one API call, and a cleanup job with ten steps is one that will eventually only do nine.

Why this part is deliberately not GitOps

Everything else in that repo is ArgoCD watching git and syncing. I wrote a whole post about why that is good, and then made these opt out of it.

selfHeal: true means ArgoCD recreates whatever you deleted. The cleanup job deletes a namespace; ArgoCD puts it straight back. A garbage collector and a self-healing controller cannot both own the same object.

There are lesser reasons too — three environments a day is roughly a thousand commits a year of pure churn, and generated database passwords would land in git in plaintext — but the TTL conflict alone settles it.

So the line is drawn between the platform and the tenants. ArgoCD owns the long-lived things: the cleanup CronJob, its RBAC, the PriorityClass, the shared registry pull secret. The environments themselves are cattle, created by a CLI and reaped by a job, with the namespace as the source of truth. Working out which resources belong in git and which must not is the more useful half of understanding GitOps.

The free URL

Every environment needs its own hostname and I did not want to buy or configure one.

sslip.io is a public DNS service that resolves any hostname containing an IP address to that address — a-b-c-d.sslip.io becomes a.b.c.d. So env-brave-otter-7f3a.192-168-1-174.sslip.io points at my Pi with no DNS setup anywhere, and the whole "dynamic URL per environment" requirement costs one Traefik Ingress object.

The alternative was wildcard entries in /etc/hosts, which has no wildcard support, so every environment would need a manual edit on every machine. The Pi's IP is read out of the kubeconfig rather than hardcoded, so a DHCP change does not silently break every future URL.

There is a --tailnet flag that instead exposes it through the Tailscale operator with a real Let's Encrypt certificate, reachable from my phone anywhere. It costs a proxy pod and a tailnet device per environment, which is the wrong default for "spin up a branch and poke at it" but exactly right occasionally.

Backing services with zero config

The app needs a database and should not have to be told where it is. Each extra — postgres, redis, minio — is a self-contained kustomize directory, so --with postgres,redis just appends those directories to the generated overlay. Adding a new one is adding a directory.

At create time the CLI generates credentials with openssl rand -base64 24 and writes a single Secret that every workload picks up with envFrom:

DATABASE_URL      postgres://app:<generated>@postgres:5432/app
REDIS_URL         redis://redis:6379/0
S3_ENDPOINT       http://minio:9000

The services keep their plain names because the namespace already scopes them, and postgres:5432 is exactly what a developer's docker-compose.yml looks like. The passwords live only in that namespace, never touch my laptop's disk, and die with the environment — which is the fourth reason this is not GitOps.

Seeding runs as a Job whose init container polls pg_isready rather than trusting startup order, and the CLI fails the create if seeding fails. A preview environment with a half-seeded database is worse than no environment at all.

Logs came free: Alloy already runs as a DaemonSet tailing /var/log/pods, so pods appear in Loki the moment they start. The only work was per-stream retention, since every environment mints a new value for the indexed namespace label — preview logs expire after 48h instead of the platform's 14 days.

Keeping one environment from ruining the cluster

This is the part I found genuinely interesting. The Pis run things I care about — ArgoCD, Loki, Tempo — and now they also run code from a pull request I have not read.

Four rules bound the blast radius:

• A ResourceQuota per namespace. My first limits.cpu: 2 was wrong: every extra plus a seed Job at once is 2450m, so seeding a full-stack environment was silently blocked at exactly the moment it started. A real one measures 1750m / 1408Mi.
• A LimitRange, which is not optional. More on that below.
• A PriorityClass at value: -10 with preemptionPolicy: Never. The kubelet evicts by QoS and priority, so preview environments are the first thing killed under memory pressure and ArgoCD is the last.
• Node affinity away from the control plane, preferred rather than required so a single-node cluster still works, just less politely.

That last one is the most valuable of the four, and it only became possible when a second Pi joined. Being out of the blast radius entirely beats being ranked first within it.

The LimitRange is the one that will catch you out:

Once a ResourceQuota constrains limits.memory, the admission controller rejects every pod that does not set memory limits — and an arbitrary user image does not.

Without default and defaultRequest values, --image nginx fails with must specify limits.memory and you spend twenty minutes wondering what nginx has to do with anything. It also has to be applied before the Deployment, or the first ReplicaSet's pods lose the race.

Isolation is a default-deny NetworkPolicy in both directions, opened up for DNS, Traefik, Tempo and the public internet — but explicitly not the rest of the cluster or my LAN:

// platform/ephemeral/base/networkpolicy.yaml
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8          # K3s pod + service CIDRs
              - 172.16.0.0/12
              - 192.168.0.0/16      # the home LAN, including the Pi itself
              - 169.254.0.0/16      # link-local / cloud metadata

K3s enforces NetworkPolicy out of the box, but a silently unenforced policy is worse than none because you stop thinking about the problem. So I sat inside an environment with nc and checked:

Add automountServiceAccountToken: false and Pod Security Admission at baseline, and a hostile image gets a fairly boring cage. Half an hour with nc was the best-spent half hour of the project.

The cleanup job itself is the most dangerous thing in the repo, since it holds namespace-delete rights. Three layers guard it: it only ever lists namespaces matching k8i.dev/ephemeral=true, it checks an explicit protected-namespace denylist before every delete, and its RBAC covers namespaces: get/list/watch/delete and nothing else. One honest gap: RBAC cannot be scoped by label, so that ServiceAccount can technically delete any namespace. The label selector and the denylist are what actually enforce the boundary, and they are lines in a shell script. The real fix is a ValidatingAdmissionPolicy; it is on the list, not in the build.

Making it automatic

A command I have to type is not a platform. The version that matters is the one where the environment appears without anyone asking.

The problem is that my cluster is behind NAT on a home LAN. GitHub cannot reach into it. The obvious answer is to expose a webhook receiver — Tailscale Funnel would do it — so GitHub can push events in.

I did the opposite. A CronJob inside the cluster asks GitHub every 60 seconds which pull requests are open and what each one's head SHA is, then converges the cluster on that answer.

A webhook is edge-triggered. Drop the event — receiver restarting, cluster rebooting, GitHub's retries exhausted — and it is gone forever, and the cluster is silently wrong until a human notices. A poller is level-triggered: it re-derives desired state from scratch every cycle, so any outage self-corrects on the next tick.

It also needs no new infrastructure — no public endpoint, no HMAC validation to get right forever, and no self-hosted Actions runner holding cluster credentials where a pull request author could reach them. The cost is up to 60 seconds of latency, which is nothing when the arm64 build takes minutes.

Some details that matter more than they look:

• The reconciler shells out to the same CLI rather than reimplementing it. One implementation, two callers: my laptop and a CronJob.
• A push does kubectl set image and waits for the rollout. It does not recreate the namespace, because the database is the whole point of clicking around a preview. Migrations run as a Job named after the SHA, so they are idempotent per commit.
• Environments are opt-in via a preview label on the PR, which doubles as the kill switch.
• Namespaces are env-pr-<number>-<repo>. Pull request numbers are per-repo, so api#42 and blog#42 both exist, and env-pr-42 would have the second repo silently redeploy over the first one's environment.

It went live on a real pull request: labelled, built, and the environment was serving HTTP 200 with the URL commented back on the PR. Under 2 minutes from poll to environment once the image lands.

Things I would tell myself before starting

• kubectl does not fall back to in-cluster config. Unlike client-go, it dials http://localhost:8080 with no kubeconfig — so my CLI reported "cluster is not reachable. Is the Pi on?" from a pod with a valid token and the API server three hops away.
• Anything wrapped in || true will eventually lie to you. That is how a read-only rehearsal passed while every kubectl call in it failed: a total failure returned an empty list, which looks exactly like "nothing to do". Same shape as stamping the namespace with the commit SHA before applying the workloads — a half-finished create then reads as "up to date" forever. Derive readiness from the thing you need, not a marker you wrote down beforehand.
• Silence is the worst error message. Twice I labelled a pull request and nothing happened, with no error anywhere: once because .github/workflows/preview.yaml was missing from that branch (pull_request workflows run from the PR's own branch, not main), once because merge conflicts made GitHub skip the workflow entirely. Meanwhile my reconciler logged "not in the registry yet" forever, which is what a build still running looks like too.
• concurrencyPolicy: Forbid needs a deadline. Both nodes were left cordoned after a shutdown, the cleanup job's pod could not be scheduled, and one stuck run blocked every later one for 2 days and 20 hours with no alarm. activeDeadlineSeconds: 240 fixed it, and proved itself when the same thing happened again.
• Check that your recovery command has ever worked. make uncordon ran kubectl uncordon --all. That flag belongs to cordon and drain, not uncordon, so it had exited with unknown flag and changed nothing, every single time.
• When several things break at once, look for the shared substrate. The symptom was kubectl spamming metrics.k8s.io errors, which points at metrics-server — innocent, just the loudest complainer. kubectl get nodes would have shown SchedulingDisabled in one line.
• Emulate only the stage whose output is architecture-specific. My first arm64 build took 20 minutes under QEMU. FROM --platform=$BUILDPLATFORM on the build stage runs the install and bundle natively on x86, while the runtime stage stays linux/arm64 for its native addons.

What is next

• Read the PR's check-run conclusion, so a failed build says so instead of waiting forever for an image that is never coming. Also skip PRs with merge conflicts.
• Hibernation instead of deletion. A preview environment is idle about 95% of the time, so scaling to zero on no traffic would turn a cap of 4 into a cap of 10 on the same hardware.
• A wildcard certificate via cert-manager DNS-01, so the default URL gets real HTTPS without a Tailscale proxy pod per environment.
• A registry pull-through cache, to make the 76s cold create look more like the 8s one.

If you have a homelab gathering dust, this is a good thing to point it at. Every hard part is hard for a real reason, and none of them show up when a cloud provider is quietly absorbing your mistakes for you.

That's all for now. See you in the next one !

sslip.io: https://sslip.io/

Docker multi-platform builds: https://docs.docker.com/build/building/multi-platform/

Kubernetes Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/