fix: keep backup credentials out of agent namespace #59

Merged

morten-olsen merged 1 commit from fix/c794e94b-exclude-agent-backup-secret into main

2026-10-10 09:39:59 +02:00

morten-olsen commented

2026-10-10 09:35:36 +02:00

Owner

Summary

Exclude exactly mission-control-agents from automatic and manually requested volsync-restic reflection; retain every other namespace.

- reflection-auto-namespaces: ""
+ reflection-allowed-namespaces: "^(?!mission-control-agents$).+$"
+ reflection-auto-namespaces: "^(?!mission-control-agents$).+$"

Evidence

Before: live reflected Secret exists in agents namespace; all 41 scheduled backups completed today.
After (local): Helm lint passes; rendered changed ExternalSecret and Password pass server dry-run. Namespace expression excludes only agents among 30 live namespaces, permits similar prefixes.
ESO v2.11.0 controller shouldRefreshPeriodic skips refresh for interval zero when previously synced; isSecretValid hashes data only. Therefore template update does not rotate password but requires one metadata-only source annotation patch after GitOps reconciliation. No force refresh or data write.
Repo scan workflow runs only on main push/manual; no PR build configured. Owner will watch main scan after merge.

Merge danger

Door: two-way. Revert template and restore source reflection annotations; Reflector recreates agents copy from unchanged source.

Blast radius: namespace. Only agents copy is removed; no workload or backup schedule changes.

Rollback and verification

Roll back: revert Git change, then remove source allowed-namespaces annotation and restore auto-namespaces to empty. Never delete source or force-sync generator.
Verify: shared Application Synced/Healthy at merged SHA; ExternalSecret template matches; patch only source annotations to match Git. Confirm source UID and data unchanged (boolean comparison only), agents Secret absent repeatedly, all other copies remain, backup conditions and lastSyncTime do not regress.
Reviewers: Security (exact exclusion, manual reflection blocked, password preservation); SRE (backup availability and rollout/rollback).

## Summary Exclude exactly `mission-control-agents` from automatic and manually requested `volsync-restic` reflection; retain every other namespace. ```diff - reflection-auto-namespaces: "" + reflection-allowed-namespaces: "^(?!mission-control-agents$).+$" + reflection-auto-namespaces: "^(?!mission-control-agents$).+$" ``` ## Evidence - Before: live reflected Secret exists in agents namespace; all 41 scheduled backups completed today. - After (local): Helm lint passes; rendered changed ExternalSecret and Password pass server dry-run. Namespace expression excludes only agents among 30 live namespaces, permits similar prefixes. - ESO v2.11.0 controller `shouldRefreshPeriodic` skips refresh for interval zero when previously synced; `isSecretValid` hashes data only. Therefore template update does not rotate password but requires one metadata-only source annotation patch after GitOps reconciliation. No force refresh or data write. - Repo scan workflow runs only on main push/manual; no PR build configured. Owner will watch main scan after merge. ## Merge danger **Door:** two-way. Revert template and restore source reflection annotations; Reflector recreates agents copy from unchanged source. **Blast radius:** namespace. Only agents copy is removed; no workload or backup schedule changes. ## Rollback and verification - Roll back: revert Git change, then remove source allowed-namespaces annotation and restore auto-namespaces to empty. Never delete source or force-sync generator. - Verify: shared Application Synced/Healthy at merged SHA; ExternalSecret template matches; patch only source annotations to match Git. Confirm source UID and data unchanged (boolean comparison only), agents Secret absent repeatedly, all other copies remain, backup conditions and lastSyncTime do not regress. - Reviewers: Security (exact exclusion, manual reflection blocked, password preservation); SRE (backup availability and rollout/rollback).

morten-olsen added 1 commit

2026-10-10 09:35:36 +02:00

fix: exclude agent namespace from backup secret reflection c1983b7971

morten-olsen reviewed

2026-10-10 09:38:47 +02:00

morten-olsen left a comment

Security review of PR 59, c1983b7971 against authoritative Forgejo main (one commit, three files). ## Standards No blocking findings. Diff and locally rendered backups.yaml narrow both reflection-allowed-namespaces and reflection-auto-namespaces at ExternalSecret metadata and target.template.metadata to ^(?!mission-control-agents$).+$. No credentials committed, data changes, generator changes, refresh changes, RBAC, exposure, image sources or host access changes. Docs explain the imperative metadata-only step and forbid source deletion/force-sync. Non-blocking verification limitation (UNKNOWN pending rollout): static review plus permitted live backup metadata; PR build not seen. scan.yaml runs only main push/manual, so there is no configured PR build. Owner reports baseline pipeline 99 success; my woodpecker-cli is unconfigured and cannot independently confirm it. Software Engineer must confirm merged pipeline green before parent closure. ## Spec No blocking findings. kubectl get namespaces plus expression check: 30 namespaces, excluded only mission-control-agents; similar prefix allowed. kubectl get replicationsources -A shows all 41 lastSyncTime values today, consumers in monitoring/prod/shared, Mission Control 04:28:20Z. Rendered template preserves refreshInterval=0 and existing generator/data expressions. Reflector documentation confirms both allowed and automatic namespace matching and removal of invalid automatic mirrors: https://github.com/emberstack/kubernetes-reflector#usage . ESO v2.11.0 shouldRefreshPeriodic skips previously synced interval-zero resources before comparing ExternalSecret version; isSecretValid also checks UID and managed label, then data hash (not reflection annotation equality): https://github.com/external-secrets/external-secrets/blob/v2.11.0/pkg/controllers/externalsecret/externalsecret_controller.go . Thus changing only reflection annotations preserves validity if source is already valid. Owner should confirm source is valid before patch; do not change managed labels/data-hash, data, UID, or force-sync. No password rotation is authorized. Non-blocking verification limitation (UNKNOWN): Secret access is intentionally unavailable; ExternalSecret get also returned Forbidden. Current source validity/annotations, copy identity, and eventual mirror removal rely on owner verification. No secret values read or printed. Rollout acceptance: after required SRE review, confirm shared Synced/Healthy at merged revision and rendered template; patch only existing source reflection annotations. Check no namespace-selector annotation supplies an alternative allowance. Privately compare source UID/data and report booleans only; repeatedly verify agents copy absent, other copies retained, and backup health/scheduled syncs unchanged. This prevents agents pods mounting the reflected password; it does not remove other authorized namespaces or revoke any previously obtained credential. Rollback: revert Git and restore previous source reflection annotations only; no source deletion, generator refresh or data write. Restoring agents access would need the applicable Morten decision. Standards: 0 blocking, 1 non-blocking verification limitation (UNKNOWN). Spec: 0 blocking, 1 non-blocking verification limitation (UNKNOWN).

Security review of PR 59, c1983b79716bb06a4d6e2f77239a8b4c5bee452a against authoritative Forgejo main (one commit, three files). ## Standards No blocking findings. Diff and locally rendered backups.yaml narrow both reflection-allowed-namespaces and reflection-auto-namespaces at ExternalSecret metadata and target.template.metadata to ^(?!mission-control-agents$).+$. No credentials committed, data changes, generator changes, refresh changes, RBAC, exposure, image sources or host access changes. Docs explain the imperative metadata-only step and forbid source deletion/force-sync. Non-blocking verification limitation (UNKNOWN pending rollout): static review plus permitted live backup metadata; PR build not seen. scan.yaml runs only main push/manual, so there is no configured PR build. Owner reports baseline pipeline 99 success; my woodpecker-cli is unconfigured and cannot independently confirm it. Software Engineer must confirm merged pipeline green before parent closure. ## Spec No blocking findings. kubectl get namespaces plus expression check: 30 namespaces, excluded only mission-control-agents; similar prefix allowed. kubectl get replicationsources -A shows all 41 lastSyncTime values today, consumers in monitoring/prod/shared, Mission Control 04:28:20Z. Rendered template preserves refreshInterval=0 and existing generator/data expressions. Reflector documentation confirms both allowed and automatic namespace matching and removal of invalid automatic mirrors: https://github.com/emberstack/kubernetes-reflector#usage . ESO v2.11.0 shouldRefreshPeriodic skips previously synced interval-zero resources before comparing ExternalSecret version; isSecretValid also checks UID and managed label, then data hash (not reflection annotation equality): https://github.com/external-secrets/external-secrets/blob/v2.11.0/pkg/controllers/externalsecret/externalsecret_controller.go . Thus changing only reflection annotations preserves validity if source is already valid. Owner should confirm source is valid before patch; do not change managed labels/data-hash, data, UID, or force-sync. No password rotation is authorized. Non-blocking verification limitation (UNKNOWN): Secret access is intentionally unavailable; ExternalSecret get also returned Forbidden. Current source validity/annotations, copy identity, and eventual mirror removal rely on owner verification. No secret values read or printed. Rollout acceptance: after required SRE review, confirm shared Synced/Healthy at merged revision and rendered template; patch only existing source reflection annotations. Check no namespace-selector annotation supplies an alternative allowance. Privately compare source UID/data and report booleans only; repeatedly verify agents copy absent, other copies retained, and backup health/scheduled syncs unchanged. This prevents agents pods mounting the reflected password; it does not remove other authorized namespaces or revoke any previously obtained credential. Rollback: revert Git and restore previous source reflection annotations only; no source deletion, generator refresh or data write. Restoring agents access would need the applicable Morten decision. Standards: 0 blocking, 1 non-blocking verification limitation (UNKNOWN). Spec: 0 blocking, 1 non-blocking verification limitation (UNKNOWN).

morten-olsen reviewed

2026-10-10 09:39:23 +02:00

morten-olsen left a comment

SRE review c1983b7: no blocking findings. Standards: Helm lint and rendered template pass; only reflection annotations/docs change, no resources, PVCs, schedules, generator, creationPolicy, waves or workload changes. Non-blocking CI limit: main-only advisory scan configured, no PR pipeline; baseline 99 success owner-reported, my Woodpecker CLI is unconfigured. Owner must confirm merged scan. Spec: expression excludes only mission-control-agents among all 30 live namespaces and retains similar prefixes. Independent live baseline: all 41 ReplicationSources Successful today and WaitingForSchedule (prod=38/shared=2/monitoring=1), all use volsync-restic; 149 PVCs Bound, compute Ready without pressure, 81/81 Applications Synced/Healthy, ESO/Reflector/Volsync ready. Shared remains at 341d6f7. Non-blocking rollout verification limit: Secrets/logs inaccessible; ExternalSecret read Forbidden. Owner must verify UID/data preservation privately (booleans only), repeated agents absence, every other copy retained, backup conditions retained and next natural scheduled runs successful. ESO v2.11.0 source supports interval-zero skipping template refresh with valid target; metadata-only source annotation patch after Git reconciliation is appropriate. No force-sync, source deletion, generator/data change or backup Job. Rollback: revert Git and restore source annotations (remove allowed restriction, auto namespaces empty), recreating only agents mirror from unchanged source. task health/check unavailable: task executable absent, no complete alert assessment. Full command evidence and sources recorded on subtask 2295935a-9755-412f-a2d2-5e90018e5f97. Standards: 1 non-blocking verification limitation, 0 blocking; Spec: 1 non-blocking rollout verification limitation, 0 blocking.

SRE review c1983b7: no blocking findings. Standards: Helm lint and rendered template pass; only reflection annotations/docs change, no resources, PVCs, schedules, generator, creationPolicy, waves or workload changes. Non-blocking CI limit: main-only advisory scan configured, no PR pipeline; baseline 99 success owner-reported, my Woodpecker CLI is unconfigured. Owner must confirm merged scan. Spec: expression excludes only mission-control-agents among all 30 live namespaces and retains similar prefixes. Independent live baseline: all 41 ReplicationSources Successful today and WaitingForSchedule (prod=38/shared=2/monitoring=1), all use volsync-restic; 149 PVCs Bound, compute Ready without pressure, 81/81 Applications Synced/Healthy, ESO/Reflector/Volsync ready. Shared remains at 341d6f7. Non-blocking rollout verification limit: Secrets/logs inaccessible; ExternalSecret read Forbidden. Owner must verify UID/data preservation privately (booleans only), repeated agents absence, every other copy retained, backup conditions retained and next natural scheduled runs successful. ESO v2.11.0 source supports interval-zero skipping template refresh with valid target; metadata-only source annotation patch after Git reconciliation is appropriate. No force-sync, source deletion, generator/data change or backup Job. Rollback: revert Git and restore source annotations (remove allowed restriction, auto namespaces empty), recreating only agents mirror from unchanged source. task health/check unavailable: task executable absent, no complete alert assessment. Full command evidence and sources recorded on subtask 2295935a-9755-412f-a2d2-5e90018e5f97. Standards: 1 non-blocking verification limitation, 0 blocking; Spec: 1 non-blocking rollout verification limitation, 0 blocking.

morten-olsen merged commit 6049d6767b into main

2026-10-10 09:39:59 +02:00

morten-olsen referenced this pull request from a commit

2026-10-10 09:40:01 +02:00

fix: keep backup credentials out of agent namespace (#59)

morten-olsen referenced this pull request

2026-10-25 09:45:11 +01:00

fix: deploy safe postgres credential diagnostics #69