fix: remove credential SQL from operator diagnostics #2

Merged

morten-olsen merged 1 commit from fix/20ca7f9d-password-logs into main

2026-10-24 22:36:39 +02:00

morten-olsen commented

2026-10-24 21:35:29 +02:00

Owner

Summary

Credential CREATE/ALTER USER failures now expose a fixed operation, username and validated SQLSTATE. Password-bearing SQL and raw driver causes no longer enter operator logs, DatabaseReady conditions or returned reconcile errors. SQL execution/retry behavior is unchanged.

- credential query + raw driver diagnostics
+ safe operation + username + SQLSTATE

Evidence

Before: synthetic CREATE/ALTER failure cases reproduced SQL/password leaks in logs, returned errors and conditions.
After: quoted-password, SQL-echoing driver errors, wrapped pq diagnostics and invalid SQLSTATE tests pass; full tests/build/lint and chart/server dry-runs pass.
Main Tests 40130556194, Lint 40130556259, Kind E2E 40130556224 and Release 40130556184 succeeded at 90fae7b. No Woodpecker registration.
Approved 0.0.20 publication verified: amd64/arm64 revision labels match 90fae7b; release/Helm chart contents match. Image index sha256:4a1f2b026b92c3f322b14b987cc69ab0395a986efa5031702c1d61eb3db785c8 is now running via core #69, chart 0.1.9. Version tags retained in Forgejo and mirrored.
ArgoCD Synced/Healthy; actual operator one ready replica/zero restarts, all 15 databases Ready and same PVC Bound. Current operator logs/status contain zero credential SQL. Failure behavior verified synthetically; no production failure provoked. Final strict health check HEALTHY.

Merge danger

Door: one-way
Published images/releases/manifests/charts persist; code/image can be reverted. Releases were explicitly approved.

Blast radius: app
Operator diagnostics only. Confirmed PostgreSQL server-log credential exposure remains HIGH in HOMEIT-433. This fix does not stop that separate leak or authorize rotation/risk acceptance.

Rollback and verification

Roll back: restore the previously verified 0.0.19 digest on HOMEIT-427; restored version tags may rebuild images. Unsafe diagnostics return; historical exposure remains.
Verify after rollout: published image digest, ArgoCD revision, replicas/restarts, database/PVC state and count-only log/status scan. Completed; evidence retained on HOMEIT-427 and base #59.
Reviewers: Security Engineer completed HOMEIT-435, no blocking code finding. Server-log fix/approved rotation stays HOMEIT-433; durable release-tag correction stays HOMEIT-434.

## Summary Credential CREATE/ALTER USER failures now expose a fixed operation, username and validated SQLSTATE. Password-bearing SQL and raw driver causes no longer enter operator logs, DatabaseReady conditions or returned reconcile errors. SQL execution/retry behavior is unchanged. ```diff - credential query + raw driver diagnostics + safe operation + username + SQLSTATE ``` ## Evidence - **Before:** synthetic CREATE/ALTER failure cases reproduced SQL/password leaks in logs, returned errors and conditions. **After:** quoted-password, SQL-echoing driver errors, wrapped pq diagnostics and invalid SQLSTATE tests pass; full tests/build/lint and chart/server dry-runs pass. - Main [Tests 40130556194](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40130556194), [Lint 40130556259](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40130556259), [Kind E2E 40130556224](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40130556224) and [Release 40130556184](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40130556184) succeeded at 90fae7b. No Woodpecker registration. - Approved 0.0.20 publication verified: amd64/arm64 revision labels match 90fae7b; release/Helm chart contents match. Image index `sha256:4a1f2b026b92c3f322b14b987cc69ab0395a986efa5031702c1d61eb3db785c8` is now running via [core #69](https://code.olsen.cloud/homelab/core/pulls/69), chart 0.1.9. Version tags retained in Forgejo and mirrored. - ArgoCD Synced/Healthy; actual operator one ready replica/zero restarts, all 15 databases Ready and same PVC Bound. Current operator logs/status contain zero credential SQL. Failure behavior verified synthetically; no production failure provoked. Final strict health check HEALTHY. ## Merge danger **Door:** one-way Published images/releases/manifests/charts persist; code/image can be reverted. Releases were explicitly approved. **Blast radius:** app Operator diagnostics only. Confirmed PostgreSQL server-log credential exposure remains HIGH in HOMEIT-433. This fix does not stop that separate leak or authorize rotation/risk acceptance. ## Rollback and verification - **Roll back:** restore the previously verified 0.0.19 digest on HOMEIT-427; restored version tags may rebuild images. Unsafe diagnostics return; historical exposure remains. - **Verify after rollout:** published image digest, ArgoCD revision, replicas/restarts, database/PVC state and count-only log/status scan. Completed; evidence retained on HOMEIT-427 and base #59. - **Reviewers:** Security Engineer completed HOMEIT-435, no blocking code finding. Server-log fix/approved rotation stays HOMEIT-433; durable release-tag correction stays HOMEIT-434.

morten-olsen added 1 commit

2026-10-24 21:35:29 +02:00

fix: redact credential statement failures from operator diagnostics 11820dc3df

morten-olsen referenced this pull request from homelab/core

2026-10-24 21:36:07 +02:00

fix: deploy safe postgres credential diagnostics #69

morten-olsen reviewed

2026-10-24 21:42:05 +02:00

morten-olsen left a comment

Security review pinned to operator 11820dc3df (PR #2), core 51d8f61 (#69), base 7ed2287 (#59). Forgejo source/three-dot diffs reviewed.

Standards

NON-BLOCKING: no security boundary regression found in the diff. No new exposure, auth, RBAC, host access, image source or secret payload. Core changes only chart/operator versions; base adds response documentation and pointer. Sanitized errors deliberately discard the driver cause, consistent with the no-password-in-status rule.
BLOCKING RELEASE GATE (already recorded; not a code defect): owner must obtain Morten's explicit approval before the automatic release merge, source tag restoration/retention and core publication/deployment. HOMEIT-434 must remain tracked; confirm the approved source tags and intended immutable artifact digests before rollout. This review grants neither release approval nor risk acceptance.

Spec

NON-BLOCKING: operator-output contract passes source review. CREATE failure uses one sanitized error for logs, DatabaseReady and return; ALTER uses the sanitizer and preserves continuation. errors.As handles wrapped pq errors; only five ASCII uppercase letters/digits survive as SQLSTATE; no %w/cause is retained. Control behavior is unchanged.
NON-BLOCKING validation limitation: synthetic tests exercise CREATE/ALTER generic SQL-echoing and wrapped pq errors with quoted synthetic credentials; assertions cover logs, stored status, returned text, CREATE cause removal, safe user context and SQLSTATE validation. ALTER test confirms continuation to CREATE DATABASE. Local focused go test could not run: go is absent. Owner reports exact-head Lint 40125972927, Tests 40125972961 and Kind E2E 40125972942 green, core Woodpecker #137/#138 green; base task check green/no pipeline. Forgejo commit status API has no checks and local Forgejo Actions query returned no runs; remote conclusions were not independently verified here. Owner must retain these exact-head green run links as merge evidence.
NON-BLOCKING HIGH residual exposure: supplied count-only summaries confirm 975 current + 136 previous server-log occurrences across all 15 managed credentials, Oct 19–24. This is credential exposure, not proof of unauthorized use. Log/node/backup readers could gain database access as those roles; public application compromise is a possible pivot, not verified exploitation. Live kubectl checks show postgres operator 0.0.19/one ready replica, ArgoCD Synced/Healthy at 0.0.19, PostgreSQL ClusterIP services, no VirtualService HTTP destination containing postgres, and no shared NetworkPolicy. Isolation/log-reader RBAC not proven.
NON-BLOCKING UNKNOWN historical extent: current operator's 374 clean lines and clean status summary do not clear prior history. Previous operator logs, older node archives/exports/backups and unauthorized readership remain unverified. I did not read logs or secrets; PostgresDatabase listing is forbidden for this role. Summaries are owner evidence.

Recommend promptly advancing HOMEIT-433 (currently triage): stop server-side disclosure, including genuine CREATE/ALTER failures, then obtain Morten-approved coordinated rotation of all 15 affected credentials and retention decision. Operator fix is available in this PR but does not stop server-side leakage; no deployed server-side fix verified. Keep these tasks open; do not interpret review completion as incident resolution.

After approved publication, owner verifies exact image digest, ArgoCD revision, replica/restarts, all 15 Ready resources and count-only logs/status; use synthetic tests for failure paths, never provoke production credential failures. Rollback to the prior pin restores unsafe logging and cannot revoke exposed credentials or remove published artifacts.

Findings: Standards 1 blocking release gate, 1 non-blocking; Spec 4 non-blocking, worst residual HIGH (historical extent UNKNOWN). No blocking implementation finding.

Security review pinned to operator 11820dc3dfabdeb34b75efaf9f994fa0b661733f (PR #2), core 51d8f61 (#69), base 7ed2287 (#59). Forgejo source/three-dot diffs reviewed. ## Standards - NON-BLOCKING: no security boundary regression found in the diff. No new exposure, auth, RBAC, host access, image source or secret payload. Core changes only chart/operator versions; base adds response documentation and pointer. Sanitized errors deliberately discard the driver cause, consistent with the no-password-in-status rule. - BLOCKING RELEASE GATE (already recorded; not a code defect): owner must obtain Morten's explicit approval before the automatic release merge, source tag restoration/retention and core publication/deployment. HOMEIT-434 must remain tracked; confirm the approved source tags and intended immutable artifact digests before rollout. This review grants neither release approval nor risk acceptance. ## Spec - NON-BLOCKING: operator-output contract passes source review. CREATE failure uses one sanitized error for logs, DatabaseReady and return; ALTER uses the sanitizer and preserves continuation. errors.As handles wrapped pq errors; only five ASCII uppercase letters/digits survive as SQLSTATE; no %w/cause is retained. Control behavior is unchanged. - NON-BLOCKING validation limitation: synthetic tests exercise CREATE/ALTER generic SQL-echoing and wrapped pq errors with quoted synthetic credentials; assertions cover logs, stored status, returned text, CREATE cause removal, safe user context and SQLSTATE validation. ALTER test confirms continuation to CREATE DATABASE. Local focused go test could not run: go is absent. Owner reports exact-head Lint 40125972927, Tests 40125972961 and Kind E2E 40125972942 green, core Woodpecker #137/#138 green; base task check green/no pipeline. Forgejo commit status API has no checks and local Forgejo Actions query returned no runs; remote conclusions were not independently verified here. Owner must retain these exact-head green run links as merge evidence. - NON-BLOCKING HIGH residual exposure: supplied count-only summaries confirm 975 current + 136 previous server-log occurrences across all 15 managed credentials, Oct 19–24. This is credential exposure, not proof of unauthorized use. Log/node/backup readers could gain database access as those roles; public application compromise is a possible pivot, not verified exploitation. Live kubectl checks show postgres operator 0.0.19/one ready replica, ArgoCD Synced/Healthy at 0.0.19, PostgreSQL ClusterIP services, no VirtualService HTTP destination containing postgres, and no shared NetworkPolicy. Isolation/log-reader RBAC not proven. - NON-BLOCKING UNKNOWN historical extent: current operator's 374 clean lines and clean status summary do not clear prior history. Previous operator logs, older node archives/exports/backups and unauthorized readership remain unverified. I did not read logs or secrets; PostgresDatabase listing is forbidden for this role. Summaries are owner evidence. Recommend promptly advancing HOMEIT-433 (currently triage): stop server-side disclosure, including genuine CREATE/ALTER failures, then obtain Morten-approved coordinated rotation of all 15 affected credentials and retention decision. Operator fix is available in this PR but does not stop server-side leakage; no deployed server-side fix verified. Keep these tasks open; do not interpret review completion as incident resolution. After approved publication, owner verifies exact image digest, ArgoCD revision, replica/restarts, all 15 Ready resources and count-only logs/status; use synthetic tests for failure paths, never provoke production credential failures. Rollback to the prior pin restores unsafe logging and cannot revoke exposed credentials or remove published artifacts. Findings: Standards 1 blocking release gate, 1 non-blocking; Spec 4 non-blocking, worst residual HIGH (historical extent UNKNOWN). No blocking implementation finding.

morten-olsen merged commit 90fae7bb52 into main

2026-10-24 22:36:39 +02:00

morten-olsen referenced this pull request from a commit

2026-10-24 22:36:40 +02:00

Merge pull request 'fix: remove credential SQL from operator diagnostics' (#2) from fix/20ca7f9d-password-logs into main