fix: prevent ownership reconciliation from blocking app startup #1

Merged

morten-olsen merged 2 commits from fix/3f88f077-ownership-locks into main

2026-10-24 20:58:06 +02:00

morten-olsen commented

2026-10-24 10:19:23 +02:00

Owner

Summary

Prevent ownership reconciliation from blocking Authentik startup while its migration worker holds table locks. Query catalogs for public objects with the wrong owner, repair each in its own transaction with a two-second local lock timeout, and return failures for controller retry. The complete repair has a 30-second deadline. Skip table row types and generated array types, whose owners follow the parent object.

catalog → wrong owners only → one transaction/object → commit → next object
                              ↳ lock wait >2s → rollback + retry

HOMEIT-368. Pair with Authentik startup protection.

Evidence

Before: real PostgreSQL contention tests fail: no-op repair waits behind a read lock, and a blocked ownership change waits five seconds and reports success. Several type owners also remain incorrect.
After: all contention cases pass in 2.20s: already owned table 0.00s; blocked wrong-owner repair errors in 2.06s and succeeds on retry. Initial and repeated repair cover tables, sequence, view, materialized view, function, enum, composite and domain.
Full make test build passes; Helm lint/render and API server dry-run pass. Compatible Go 1.25 lint passes with zero issues.
Test DB is isolated PostgreSQL 16.10. The catalog query also executes read-only on live PostgreSQL 18 and reports zero ownership repairs needed for Authentik. Read-only preflight across all 15 managed databases succeeds. The new repair selects 17 preexisting type-owner mismatches in agent (3), immich (8), metabase (1) and miniflux (5); Authentik requires zero changes. These are repairs under the existing ownership policy. Live contention/reboot testing is not claimed.
Mirror CI is green at d6ecb61: unit tests, lint, Kind E2E. The E2E run builds the operator image, starts it in Kind and verifies PostgresCluster/PostgresDatabase reconciliation.
The old E2E fixture crashlooped on its non-root PVC mount. It now uses a writable PGDATA child directory and pg_isready readiness probe; all three E2E specs pass. Fixture behavior is documented in the same commit.
No Woodpecker registration exists (repo show returns 404). Docker is unavailable locally; the existing mirror CI performed the image build and E2E run.

Merge danger

Door: one-way publication. The existing mirror main workflow automatically creates v0.0.19 and publishes the image, release and Helm chart; Morten must approve before merge. Restoring the previous image does not undo object ownership repairs already committed. Published artifacts remain.

Blast radius: household. The operator serves every managed PostgresDatabase. Ownership privileges are unchanged, but failures now report not ready and retry instead of being swallowed. No schema migration, credential rotation or deletion is introduced.

Rollback and verification

Roll back: restore core's prior operator chart/image (0.0.18), then revert this code for later releases. Published artifacts remain.
Verify after rollout: operator digest and replicas, PostgresDatabase conditions, no queued ownership DDL, Authentik live/ready/OIDC HTTP 200 and NetBird management/Coder availability. Deploy the database fix before the Authentik startup-probe rollout.
Reviewers: self-review; no expansion of auth, privileges, exposure, secrets or image sources.

## Summary Prevent ownership reconciliation from blocking Authentik startup while its migration worker holds table locks. Query catalogs for public objects with the wrong owner, repair each in its own transaction with a two-second local lock timeout, and return failures for controller retry. The complete repair has a 30-second deadline. Skip table row types and generated array types, whose owners follow the parent object. ```text catalog → wrong owners only → one transaction/object → commit → next object ↳ lock wait >2s → rollback + retry ``` HOMEIT-368. Pair with [Authentik startup protection](https://code.olsen.cloud/homelab/authentik-operator/pulls/1). ## Evidence - Before: real PostgreSQL contention tests fail: no-op repair waits behind a read lock, and a blocked ownership change waits five seconds and reports success. Several type owners also remain incorrect. - After: all contention cases pass in 2.20s: already owned table 0.00s; blocked wrong-owner repair errors in 2.06s and succeeds on retry. Initial and repeated repair cover tables, sequence, view, materialized view, function, enum, composite and domain. - Full `make test build` passes; Helm lint/render and API server dry-run pass. Compatible Go 1.25 lint passes with zero issues. - Test DB is isolated PostgreSQL 16.10. The catalog query also executes read-only on live PostgreSQL 18 and reports zero ownership repairs needed for Authentik. Read-only preflight across all 15 managed databases succeeds. The new repair selects 17 preexisting type-owner mismatches in agent (3), immich (8), metabase (1) and miniflux (5); Authentik requires zero changes. These are repairs under the existing ownership policy. Live contention/reboot testing is not claimed. - Mirror CI is green at **d6ecb61**: [unit tests](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40117314535), [lint](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40117314632), [Kind E2E](https://github.com/morten-olsen/homelab-postgres-operator/actions/runs/40117314553). The E2E run builds the operator image, starts it in Kind and verifies PostgresCluster/PostgresDatabase reconciliation. - The old E2E fixture crashlooped on its non-root PVC mount. It now uses a writable PGDATA child directory and pg_isready readiness probe; all three E2E specs pass. Fixture behavior is documented in the same commit. - No Woodpecker registration exists (`repo show` returns 404). Docker is unavailable locally; the existing mirror CI performed the image build and E2E run. ## Merge danger **Door:** one-way publication. The existing mirror main workflow automatically creates v0.0.19 and publishes the image, release and Helm chart; Morten must approve before merge. Restoring the previous image does not undo object ownership repairs already committed. Published artifacts remain. **Blast radius:** household. The operator serves every managed PostgresDatabase. Ownership privileges are unchanged, but failures now report not ready and retry instead of being swallowed. No schema migration, credential rotation or deletion is introduced. ## Rollback and verification - **Roll back:** restore core's prior operator chart/image (0.0.18), then revert this code for later releases. Published artifacts remain. - **Verify after rollout:** operator digest and replicas, PostgresDatabase conditions, no queued ownership DDL, Authentik live/ready/OIDC HTTP 200 and NetBird management/Coder availability. Deploy the database fix before the Authentik startup-probe rollout. - **Reviewers:** self-review; no expansion of auth, privileges, exposure, secrets or image sources.

morten-olsen added 1 commit

2026-10-24 10:19:23 +02:00