fix: reuse Authentik API transport across reconciliations #13

Merged

morten-olsen merged 1 commit from fix/fe2472f2-reuse-http-transport into main

2026-10-30 08:51:26 +01:00

morten-olsen commented

2026-10-30 08:45:46 +01:00

Owner

Summary

Repeated Authentik client reconciliations leak idle HTTP connections. NewClient(..., true, ...) creates a fresh transport on every call, with no idle timeout. Its idle connection goroutines keep the transport alive after the API client is unreachable. The live 0.3.0 operator climbs from about 30Mi to its 128Mi limit in roughly an hour and OOMs repeatedly.

- each reconcile → new transport → retained idle connection
+ each reconcile → shared pool → reuse; at most 10 idle / 2 per host / 30s idle timeout

Keep URLs and authorization headers on each SDK client; preserve the explicit TLS mode. Correct the stale README release instructions to match the existing tag-only release runbook/workflow.

Evidence

Before: a local experiment against the actual deployed client's HealthCheck interface retained 100 connections and about 300 goroutines after 100 fresh clients and forced GC. Default-client control retained one connection; explicitly closing each client's idle connections retained zero. The new reuse regression failed on the original implementation (100 accepted connections, expected at most two).
After: regressions pass for successful responses and repeated HTTP 503s with changing synthetic tokens; TLS verification remains isolated from the explicit insecure mode. Full API suite under -race passes. task ci:quality passes locally: release safety tests, controller/envtest tests, lint, manager build, e2e compile, chart lint/render and generated manifest parity.
The approved memory mitigation is already deployed through core PR #82 and base PR #74. The new 256Mi pod has zero restarts so far but continued memory growth: 26.76Mi at 07:39:07Z → 35.64Mi at 07:44:07Z. It still runs 0.3.0. This source fix has not yet been published or verified live.

Merge danger

Door: two-way for source. Revert restores per-reconcile transport allocation and the leak. A separately approved version tag and published artifacts are lasting.

Blast radius: app. No permission, access policy, image source, credential, server version or TLS-verification change. HTTP pool reuse changes connection lifetime; per-client credentials and error responses are regression-tested.

Rollback and verification

Roll back: source revert, or separately revert the eventual GitOps operator-version pin to 0.3.0 while keeping the 256Mi mitigation.
Verify after rollout: approved tag/image/chart publication, mirror, core/Authentik revisions, exact image digest/resources, startup with all 15 clients/21 groups/5 proxies/one server, queue/concurrency and working-set/RSS over a window longer than the old roughly hourly OOM cycle. Error/retry cases are locally tested; live failure injection remains unverified.
Reviewers: owner review; this does not widen permissions, trust, exposure or execution and needs no role review under the current trigger table.
Main merges validate only. Publishing v0.3.1 and updating the installation require a separate explicit release approval after a read-only preflight at the merged commit.

## Summary Repeated Authentik client reconciliations leak idle HTTP connections. `NewClient(..., true, ...)` creates a fresh transport on every call, with no idle timeout. Its idle connection goroutines keep the transport alive after the API client is unreachable. The live 0.3.0 operator climbs from about 30Mi to its 128Mi limit in roughly an hour and OOMs repeatedly. ```diff - each reconcile → new transport → retained idle connection + each reconcile → shared pool → reuse; at most 10 idle / 2 per host / 30s idle timeout ``` Keep URLs and authorization headers on each SDK client; preserve the explicit TLS mode. Correct the stale README release instructions to match the existing tag-only release runbook/workflow. ## Evidence - **Before:** a local experiment against the actual deployed client's `HealthCheck` interface retained 100 connections and about 300 goroutines after 100 fresh clients and forced GC. Default-client control retained one connection; explicitly closing each client's idle connections retained zero. The new reuse regression failed on the original implementation (100 accepted connections, expected at most two). - **After:** regressions pass for successful responses and repeated HTTP 503s with changing synthetic tokens; TLS verification remains isolated from the explicit insecure mode. Full API suite under `-race` passes. `task ci:quality` passes locally: release safety tests, controller/envtest tests, lint, manager build, e2e compile, chart lint/render and generated manifest parity. - The approved memory mitigation is already deployed through [core PR #82](https://code.olsen.cloud/homelab/core/pulls/82) and [base PR #74](https://code.olsen.cloud/homelab/base/pulls/74). The new 256Mi pod has zero restarts so far but continued memory growth: 26.76Mi at 07:39:07Z → 35.64Mi at 07:44:07Z. It still runs 0.3.0. This source fix has not yet been published or verified live. ## Merge danger **Door:** two-way for source. Revert restores per-reconcile transport allocation and the leak. A separately approved version tag and published artifacts are lasting. **Blast radius:** app. No permission, access policy, image source, credential, server version or TLS-verification change. HTTP pool reuse changes connection lifetime; per-client credentials and error responses are regression-tested. ## Rollback and verification - **Roll back:** source revert, or separately revert the eventual GitOps operator-version pin to 0.3.0 while keeping the 256Mi mitigation. - **Verify after rollout:** approved tag/image/chart publication, mirror, core/Authentik revisions, exact image digest/resources, startup with all 15 clients/21 groups/5 proxies/one server, queue/concurrency and working-set/RSS over a window longer than the old roughly hourly OOM cycle. Error/retry cases are locally tested; live failure injection remains unverified. - **Reviewers:** owner review; this does not widen permissions, trust, exposure or execution and needs no role review under the current trigger table. - Main merges validate only. Publishing v0.3.1 and updating the installation require a separate explicit release approval after a read-only preflight at the merged commit.

morten-olsen added 1 commit

2026-10-30 08:45:46 +01:00