forge-runner-k8s

A sidecar that lets forgejo-runner run CI jobs as Kubernetes Pods instead of Docker containers — without modifying the runner.

It implements a subset of the Docker Engine API (DOCKER_HOST=tcp://127.0.0.1:2375) and translates each call into Kubernetes API operations:

Docker concept	Kubernetes equivalent
Long-lived container	Pod with the container's Entrypoint/Cmd (or sleep infinity when both are empty)
docker exec per step	pods/exec via SPDY remotecommand
Bind mounts	volumeMounts with subPath on a shared RWX PVC
docker build	BuildKit daemon sidecar via gRPC Solve()

┌──────────────────────────────────────────────────────────┐
│  runner Pod                                              │
│  ┌──────────────────┐  ┌──────────────────┐  ┌────────┐ │
│  │  forgejo-runner  │  │   dockerk8s      │  │buildkit│ │
│  │  (unmodified)    │─▶│   sidecar        │─▶│  d     │ │
│  │  DOCKER_HOST=    │  │   :2375          │  │sidecar │ │
│  │  tcp://127.0.0.1 │  │                  │  │        │ │
│  └──────────────────┘  └────────┬─────────┘  └────────┘ │
│          │                      │  unix socket /run/bk   │
│          └──────────────────────┘                        │
│                   │ shared RWX PVC /data                 │
└───────────────────┼──────────────────────────────────────┘
                    │
  ┌─────────────────▼──────────┐
  │  job Pod                   │
  │  image: ubuntu:22.04       │
  │  /data/workspace ◀── PVC  │
  └────────────────────────────┘

Prerequisites

Requirement	Notes
Kubernetes cluster	v1.25+ recommended
RWX-capable StorageClass	NFS, CephFS, AWS EFS, Azure Files, etc.
Forgejo instance	v7+ with Actions enabled ([actions] ENABLED = true in app.ini)
Container registry	For docker build steps (any OCI-compatible registry)
seccompProfile: Unconfined	Required on worker nodes for rootless buildkitd
kubectl + kustomize	For deployment

1 — Enable Forgejo Actions

In your Forgejo instance app.ini:

[actions]
ENABLED = true

Restart Forgejo after changing this.

2 — Build and push the sidecar image

# Clone this repo and build
git clone https://your-forgejo/forge-runner-k8s.git
cd forge-runner-k8s

# Auto-detects podman or docker
make image IMAGE=registry.example.com/ci/dockerk8s TAG=v0.1.0
make push  IMAGE=registry.example.com/ci/dockerk8s TAG=v0.1.0

# Or explicitly
make image-podman IMAGE=registry.example.com/ci/dockerk8s TAG=v0.1.0
make image-docker IMAGE=registry.example.com/ci/dockerk8s TAG=v0.1.0

3 — Configure the deployment

Edit the files in deploy/ to match your environment before applying:

deploy/pvc.yaml — set your StorageClass

spec:
  storageClassName: "nfs-client"   # ← your RWX StorageClass name
  resources:
    requests:
      storage: 20Gi

deploy/runner-deployment.yaml — set your image and registry

Update the dockerk8s container image and environment variables:

- name: dockerk8s
  image: registry.example.com/ci/dockerk8s:v0.1.0   # ← your pushed image

  env:
    - name: FORGEJO_INSTANCE_URL
      value: "https://forgejo.example.com"           # ← your Forgejo URL
    - name: REGISTRY
      value: "registry.example.com/ci"               # ← for image builds
    - name: REGISTRY_PUSH_SECRET
      value: "registry-creds"
    - name: IMAGE_PULL_SECRETS
      value: "registry-creds"
    - name: BUILDKIT_ADDR
      value: "unix:///run/buildkit/buildkitd.sock"   # default; matches buildkitd sidecar

Update the runner container to point at your Forgejo instance:

- name: runner
  image: data.forgejo.org/forgejo/runner:6.2.2
  env:
    - name: FORGEJO_INSTANCE_URL
      value: "https://forgejo.example.com"           # ← your Forgejo URL

deploy/forgejo-runner-config.yaml — set runner labels

runner:
  labels:
    - "k8s:docker://ubuntu:22.04"      # jobs with `runs-on: k8s`

The label format is <runs-on-value>:docker://<image>. You can add multiple labels for different base images.

deploy/registry-secret.yaml — create the image pull secret

kubectl create namespace ci

kubectl create secret docker-registry registry-creds \
  --docker-server=registry.example.com \
  --docker-username=YOUR_USER \
  --docker-password=YOUR_PASSWORD \
  -n ci

This secret is used for IMAGE_PULL_SECRETS (pulling private job-image references). It is not used for build push credentials — those come per-request from the caller's docker login (see "Image builds" under Architecture notes below).

4 — Get a runner registration token

In your Forgejo instance, navigate to one of:

• Instance-level (admin only): https://forgejo.example.com/admin/actions/runners
• Organisation-level: https://forgejo.example.com/org/MY_ORG/settings/actions/runners
• Repository-level: https://forgejo.example.com/MY_USER/MY_REPO/settings/actions/runners

Click "Create new runner". Copy the displayed token — it is shown only once.

Store it as a Kubernetes Secret:

kubectl create secret generic forgejo-runner-token \
  --from-literal=token=<PASTE_TOKEN_HERE> \
  -n ci

5 — Deploy

kubectl apply -k deploy/

Check that the runner pod comes up:

kubectl get pods -n ci
# NAME                            READY   STATUS    RESTARTS   AGE
# forge-runner-xxxxxxxxx-xxxxx    2/2     Running   0          30s

Check the runner registered itself:

kubectl logs -n ci deploy/forge-runner -c runner | grep "Runner registered"

Within a minute the runner should appear in the Forgejo runners list (green dot).

6 — Register the runner (first run only)

The runner registers itself automatically on first start using the token from the Secret. If it doesn't (e.g. the token was wrong), you can register manually:

kubectl exec -n ci deploy/forge-runner -c runner -- \
  forgejo-runner register \
    --instance https://forgejo.example.com \
    --secret <TOKEN> \
    --name my-k8s-runner \
    --labels k8s \
    --no-interactive

The registration writes a .runner file to /home/runner/.runner (the runner-state emptyDir volume). Re-deploying the pod will re-register automatically; to preserve registration across restarts, back this volume up to a PVC or use the connections: block in config.yaml (see Forgejo runner configuration docs).

7 — Run a workflow

Create .forgejo/workflows/hello.yaml in any repository on your Forgejo instance:

on: [push]

jobs:
  hello:
    runs-on: k8s          # must match a label configured in step 3
    steps:
      - run: echo "Hello from Kubernetes!"
      - run: echo world > /github/workspace/f.txt
      - run: cat /github/workspace/f.txt

Push the file. The job should appear green in the repository's Actions tab.

Configuration reference

All dockerk8s sidecar settings are environment variables:

Variable	Default	Description
LISTEN_ADDR	127.0.0.1:2375	Address the Docker API proxy listens on
POD_NAMESPACE	default	Namespace for job pods (injected by downward API)
POD_NAME	—	Runner pod name (downward API) — used for ownerRefs
POD_UID	—	Runner pod UID (downward API) — used for ownerRefs
SHARED_PVC_NAME	(required)	Name of the RWX PVC
SHARED_MOUNT_PATH	/data	Mount path of the PVC in both containers
REGISTRY	—	Registry prefix for image builds (e.g. registry.io/ci)
REGISTRY_PUSH_SECRET	—	Unused by the build path (kept for compatibility); see "Image builds" below — push credentials come per-request from the job's docker login, not a static secret
IMAGE_PULL_SECRETS	—	Comma-separated pull secret names injected into job pods
BUILDKIT_ADDR	unix:///run/buildkit/buildkitd.sock	Address of buildkitd sidecar
DEFAULT_JOB_IMAGE	ubuntu:22.04	Fallback image if none specified

Development

make help          # list all targets
make build         # compile the binary
make test          # run unit tests
make test-race     # run with -race detector
make lint          # go vet

Architecture notes

• Workspace sharing: act writes the checked-out code, action binaries, and tool cache under SHARED_MOUNT_PATH. The job pod mounts the same PVC at the same path with subPath entries derived from the runner's bind-mount requests. The paths must match exactly — both containers declare mountPath: /data.

• Exec streaming: POST /exec/{id}/start hijacks the HTTP connection and bridges to the Kubernetes SPDY remotecommand protocol. Output is framed with the 8-byte Docker stdcopy multiplexed stream header so act's stdcopy.StdCopy reader works unmodified. Exit codes are extracted from exec.CodeExitError returned by StreamWithContext.

• Image builds: POST /build extracts the tar context to the shared PVC, then calls buildkitd (a third sidecar in the runner Pod) via gRPC using the dockerfile.v0 frontend. BuildKit reads the context from the PVC and pushes the built image to your registry. The resulting fully-qualified tag is registered in the proxy's state store so a subsequent ContainerCreate with the local tag resolves to the pushed image. BuildKit's layer cache persists for the lifetime of the runner Pod — rebuilds with unchanged layers are significantly faster than one-shot build Pod approaches.

• Push credentials are per-request, not a shared secret. BuildKit only resolves registry credentials from its client session — a statically mounted dockerconfigjson in the buildkitd container is never consulted for push auth. Instead, when a caller runs docker login before docker build, the classic Docker client sends the resulting credentials on POST /build as the base64-encoded X-Registry-Config header. handleBuild decodes it (parseRegistryConfig in internal/dockerapi/handlers_image.go) and buildSolveOpt attaches it to the BuildKit solve as a session auth provider (authAttachable in internal/build/buildkit.go), so each caller can push to its own registry/account with its own credentials. docker login itself succeeds against dockerk8s because POST /auth is a stub that accepts any credentials unconditionally (handleAuth in internal/dockerapi/handlers_stub.go) — real registry authentication happens later, at push time, inside BuildKit. If no X-Registry-Config header is sent, the push is attempted anonymously, matching prior behavior.

• Garbage collection: every job pod carries an ownerReference pointing to the runner pod. If the runner pod is deleted, Kubernetes GC reaps orphan job pods automatically.

Limitations (v1)

• No services: container support (no pod-to-pod networking between job containers).
• No Docker-in-Docker or privileged containers.
• In-memory state only — a sidecar restart mid-job will cause the job to fail (retryable).
• Tar copy fallback requires tar to be present in the job container image.