fix: move CI workspaces off the NVMe and alert on I/O stalls #60

Merged

morten-olsen merged 3 commits from fix/573f2ae1-ci-disk into main

2026-10-12 21:41:44 +02:00

morten-olsen commented

2026-10-12 21:36:13 +02:00

Owner

Summary

CI workspace writes shared the NVMe with k3s and application data, causing the October 10 disk stall. Keep local-path there and add woodpecker-ci at /mnt/ssd/woodpecker-ci on the general SATA SSD. Other workloads can use sibling folders; Woodpecker gets no exclusive allocation.

local-path     → NVMe /var/lib/rancher/k3s/storage
woodpecker-ci  → sda2 /mnt/ssd/woodpecker-ci

Provisioning and deletion require the host filesystem marker and reject unexpected paths and symlinks. Add NodeIOPressure (>10% full I/O stalls over 1m, sustained 30s) and extend disk-space alerts to /mnt/ssd. Persist executable storage and alert checks in PR/push CI. Deploy this before the apps workspace change.

Evidence

Before: only local-path existed; four installs caused ~40% I/O stalls.
After locally: rendered class/script tests and promtool incident/recovery tests pass. Shared/monitor Helm lint and Kubernetes server dry-runs pass. Every existing local-path PV matches the preserved NVMe path convention.
Host: approved sda2 preparation is live as ext4 UUID b8566ede-b661-4074-8b5a-f2dc743d6aee, mounted by UUID at /mnt/ssd. Task 573f2ae1 records approval and commands. No CI workload has moved yet.

Merge danger

Door: two-way for charts. The separately approved destruction of obsolete sda2 ZFS contents cannot be reverted. GPT/sda1 preserved.

Blast radius: cluster. The shared provisioner's scripts serve all local-path PVCs. CI remains at two workflows during the move.

Rollback and verification

Roll back: first drain CI and return the idle agent to local-path at two via apps. Keep the new class/config while any CI PV exists; never prune it out from under pending deletions. The host mount can stay.
Verify after rollout: mirror, shared/monitor expected revision, live class/config/rule; real workflow PVC/PV/mount on sda2 and measured writes, I/O pressure, services and restarts under concurrent installs.
Reviewer: Security Engineer, focusing on host-path scope and cleanup guards. Synthetic tests do not replace load acceptance. Reboot persistence is configured but not reboot-tested. Alerts appear in Alertmanager/Grafana; the existing receiver is null.

## Summary CI workspace writes shared the NVMe with k3s and application data, causing the October 10 disk stall. Keep `local-path` there and add `woodpecker-ci` at `/mnt/ssd/woodpecker-ci` on the general SATA SSD. Other workloads can use sibling folders; Woodpecker gets no exclusive allocation. ```text local-path → NVMe /var/lib/rancher/k3s/storage woodpecker-ci → sda2 /mnt/ssd/woodpecker-ci ``` Provisioning and deletion require the host filesystem marker and reject unexpected paths and symlinks. Add NodeIOPressure (>10% full I/O stalls over 1m, sustained 30s) and extend disk-space alerts to `/mnt/ssd`. Persist executable storage and alert checks in PR/push CI. Deploy this before the apps workspace change. ## Evidence - Before: only local-path existed; four installs caused ~40% I/O stalls. - After locally: rendered class/script tests and promtool incident/recovery tests pass. Shared/monitor Helm lint and Kubernetes server dry-runs pass. Every existing local-path PV matches the preserved NVMe path convention. - Host: approved sda2 preparation is live as ext4 UUID b8566ede-b661-4074-8b5a-f2dc743d6aee, mounted by UUID at `/mnt/ssd`. Task 573f2ae1 records approval and commands. No CI workload has moved yet. ## Merge danger **Door:** two-way for charts. The separately approved destruction of obsolete sda2 ZFS contents cannot be reverted. GPT/sda1 preserved. **Blast radius:** cluster. The shared provisioner's scripts serve all local-path PVCs. CI remains at two workflows during the move. ## Rollback and verification - **Roll back:** first drain CI and return the idle agent to local-path at two via apps. Keep the new class/config while any CI PV exists; never prune it out from under pending deletions. The host mount can stay. - **Verify after rollout:** mirror, shared/monitor expected revision, live class/config/rule; real workflow PVC/PV/mount on sda2 and measured writes, I/O pressure, services and restarts under concurrent installs. - **Reviewer:** Security Engineer, focusing on host-path scope and cleanup guards. Synthetic tests do not replace load acceptance. Reboot persistence is configured but not reboot-tested. Alerts appear in Alertmanager/Grafana; the existing receiver is null.

morten-olsen added 3 commits

2026-10-12 21:36:13 +02:00

fix(storage): provision CI in a subfolder on the shared SSD be3037c7e8

feat(monitor): alert on node I/O stalls and shared SSD capacity 464df97e5c

ci: exercise storage scripts and pressure alerts on every change

ci/woodpecker/push/charts Pipeline was successful

Details

ci/woodpecker/pr/charts Pipeline was successful

Details

5a8c2e11db

morten-olsen reviewed

2026-10-12 21:40:13 +02:00

morten-olsen left a comment

Reviewed core #60 head 5a8c2e11db against 6049d67. ## Standards No blocking findings. Diff and rendered manifests add no RBAC, authentication, public exposure, privileged flags or secret payloads. Existing helper image is unchanged; new CI uses digest-pinned ci/mise, version-pinned tools and checksum-verified promtool. Provisioner v0.0.37 mounts only the volume parent /mnt/ssd/woodpecker-ci, not the general /mnt/ssd disk; ordinary workflow PVC mounts expose only their workspace. Scripts require the UUID marker for CI setup and teardown, reject workspace/marker symlinks, traversal and unexpected names, and retain NVMe symlink/direct-directory handling. Root-owned parent/marker integrity is a trusted host prerequisite, not a defense against a compromised host or provisioner. Source: https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go#L666-L728 ## Spec Non-blocking MEDIUM availability risk: arbitrary CI writes can fill the shared SSD and affect future sibling workloads. PVC capacity is not enforced; the docs correctly state no quota/exclusive allocation, consistent with Morten's correction. Capacity alerts mitigate detection, not consumption; no additional risk waiver is implied by this review. Non-blocking UNKNOWN acceptance: actual SSD workflow mount, live cleanup, concurrent-load service health and reboot persistence remain unverified. This is a pre-rollout review, not acceptance. Evidence: scripts/ci/validate-charts.sh passed independently (archive checksum, Helm lint, rendered storage tests, promtool rule/incident/recovery tests). kubectl get deployment local-path-provisioner -n kube-system confirms v0.0.37. kubectl get pv -o json with path-only filtering: 129 local-path PVs, zero outside retained NVMe convention. kubectl get storageclass: only local-path currently. Forgejo PR checks show push #102 and PR #103 green; woodpecker-cli is installed but unconfigured here. No host SSH, ConfigMaps, Secrets, logs or exec accessed. After rollout, owner must verify expected GitOps revision, actual PVC/PV path and filesystem, representative writes/load, cleanup leaving marker/siblings intact, and original service symptom. Rollback: drain CI, return idle agent to local-path at two; retain CI class/config/mount until its PVs drain. Never erase sibling data or prune the class prematurely. Standards: 0 findings, none blocking. Spec: 2 non-blocking findings, worst MEDIUM.

Reviewed core #60 head 5a8c2e11db3a8aa5c1c86d81224ab08fd0874e46 against 6049d67. ## Standards No blocking findings. Diff and rendered manifests add no RBAC, authentication, public exposure, privileged flags or secret payloads. Existing helper image is unchanged; new CI uses digest-pinned ci/mise, version-pinned tools and checksum-verified promtool. Provisioner v0.0.37 mounts only the volume parent /mnt/ssd/woodpecker-ci, not the general /mnt/ssd disk; ordinary workflow PVC mounts expose only their workspace. Scripts require the UUID marker for CI setup and teardown, reject workspace/marker symlinks, traversal and unexpected names, and retain NVMe symlink/direct-directory handling. Root-owned parent/marker integrity is a trusted host prerequisite, not a defense against a compromised host or provisioner. Source: https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go#L666-L728 ## Spec Non-blocking MEDIUM availability risk: arbitrary CI writes can fill the shared SSD and affect future sibling workloads. PVC capacity is not enforced; the docs correctly state no quota/exclusive allocation, consistent with Morten's correction. Capacity alerts mitigate detection, not consumption; no additional risk waiver is implied by this review. Non-blocking UNKNOWN acceptance: actual SSD workflow mount, live cleanup, concurrent-load service health and reboot persistence remain unverified. This is a pre-rollout review, not acceptance. Evidence: scripts/ci/validate-charts.sh passed independently (archive checksum, Helm lint, rendered storage tests, promtool rule/incident/recovery tests). kubectl get deployment local-path-provisioner -n kube-system confirms v0.0.37. kubectl get pv -o json with path-only filtering: 129 local-path PVs, zero outside retained NVMe convention. kubectl get storageclass: only local-path currently. Forgejo PR checks show push #102 and PR #103 green; woodpecker-cli is installed but unconfigured here. No host SSH, ConfigMaps, Secrets, logs or exec accessed. After rollout, owner must verify expected GitOps revision, actual PVC/PV path and filesystem, representative writes/load, cleanup leaving marker/siblings intact, and original service symptom. Rollback: drain CI, return idle agent to local-path at two; retain CI class/config/mount until its PVs drain. Never erase sibling data or prune the class prematurely. Standards: 0 findings, none blocking. Spec: 2 non-blocking findings, worst MEDIUM.

morten-olsen merged commit 1f252d8d1c into main

2026-10-12 21:41:44 +02:00

morten-olsen referenced this pull request from a commit

2026-10-12 21:41:46 +02:00

fix: move CI workspaces off the NVMe and alert on I/O stalls (#60)

morten-olsen referenced this pull request

2026-10-12 21:47:25 +02:00

fix(storage): follow the existing local PV convention for CI #61