fix(cnpg): observe the per-statement export instead of its manifests #392

Merged

binjovi-bot merged 1 commit from fix/query-stats-runtime-guard into trunk 2026-09-06 23:33:10 +00:00

Conversation 0 Commits 1 Files changed 3 +141

sean commented 2026-09-06 23:26:28 +00:00

Owner

Copy link

What I found

While adding per-statement measurement for the Binjovi performance work, I found the pg_stat_statements export has been producing nothing since 2026-09-04, and every existing check passed the whole time.

Check	State
shared_preload_libraries	contains pg_stat_statements
postgres-query-stats ConfigMap	delivered, named in the Cluster
Database/monitoring spec.extensions	pg_stat_statements, ensure: present
Database/monitoring status	applied: true, extensions[0].applied: true
SELECT extversion FROM pg_extension in monitoring	returns nothing
cnpg_pg_stat_statements* series in VictoriaMetrics	zero

Every file in every database carries a creation time of 2026-09-04 12:49:52 and pg_stat_checkpointer.stats_reset is 2026-09-04 12:50:12, so the data plane was re-initialised that day. The Database object was not recreated with it: created 2026-08-07, still generation: 1.

That is documented CloudNativePG behaviour, not a bug. From the Database CRD documentation: "Once reconciled, a Database object will not be reapplied unless its metadata.generation changes, giving flexibility for direct PostgreSQL modifications." An initdb behind a surviving object is exactly that case, so the operator never looks again and the stale applied: true outlives the database it describes.

tests/cnpg-custom-queries.sh could not have caught this. That guard checks the delivery chain in the repository and every link was intact. Nothing in the repository was wrong. Only the running database disagreed.

Change

Repair, in cnpg-instance/query-stats.yaml. Declare schema: public on the extension. It says where the objects go instead of leaving it to the control file, and editing the spec increments the generation, which is the only supported way to make the operator apply the extension again. kubectl apply --dry-run=server reports the object as configured, so the generation does move.

This repairs today and nothing more. The next re-initialisation loses the extension again and reports success again. make rebuild is routine here, so a one-time repair is not a fix.

The durable half, in alerting/rules.yaml. PostgresQueryStatsAbsent watches cnpg_pg_stat_statements_info_tracked, the exact series the pg_stat_statements_info block emits, and fires after an hour of absence. It follows the CnpgBackupMetricMissing pattern already in that file. One hour rather than five minutes because only the primary exports, so a failover or a rolling restart is a normal gap.

Expect it to fire on merge and go quiet once the extension is back. That is the alert telling the truth about the current state.

The guard, in tests/cnpg-custom-queries.sh. The manifest chain checks stay as they are. A new section requires the alert to exist and to name a series the ConfigMap actually produces, because a rule on a misspelled series is the same silence with extra YAML.

Tests

tests/cnpg-custom-queries.sh passes, including its existing damage cases. Three new damage cases against the runtime guard are each rejected: dropping the alert, misspelling the series, and changing the expression so it no longer tests absence.

kubectl apply --dry-run=server accepts both files.

Not done here

The binjovi database still has no per-statement visibility of its own. That was the original goal, and it is a separate decision: pg_stat_statements is one cluster-wide hash table, so the deliberate design is to read it from the single monitoring database rather than install it per tenant. Once this export is alive again, Binjovi statements are already visible there through the datname label.

https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

## What I found While adding per-statement measurement for the Binjovi performance work, I found the `pg_stat_statements` export has been producing nothing since 2026-09-04, and every existing check passed the whole time. | Check | State | |---|---| | `shared_preload_libraries` | contains `pg_stat_statements` | | `postgres-query-stats` ConfigMap | delivered, named in the Cluster | | `Database/monitoring` `spec.extensions` | `pg_stat_statements`, `ensure: present` | | `Database/monitoring` `status` | `applied: true`, `extensions[0].applied: true` | | `SELECT extversion FROM pg_extension` in `monitoring` | **returns nothing** | | `cnpg_pg_stat_statements*` series in VictoriaMetrics | **zero** | Every file in every database carries a creation time of 2026-09-04 12:49:52 and `pg_stat_checkpointer.stats_reset` is 2026-09-04 12:50:12, so the data plane was re-initialised that day. The Database object was not recreated with it: created 2026-08-07, still `generation: 1`. That is documented CloudNativePG behaviour, not a bug. From the Database CRD documentation: *"Once reconciled, a `Database` object will not be reapplied unless its `metadata.generation` changes, giving flexibility for direct PostgreSQL modifications."* An initdb behind a surviving object is exactly that case, so the operator never looks again and the stale `applied: true` outlives the database it describes. `tests/cnpg-custom-queries.sh` could not have caught this. That guard checks the delivery chain in the repository and every link was intact. Nothing in the repository was wrong. Only the running database disagreed. ## Change **Repair, in `cnpg-instance/query-stats.yaml`.** Declare `schema: public` on the extension. It says where the objects go instead of leaving it to the control file, and editing the spec increments the generation, which is the only supported way to make the operator apply the extension again. `kubectl apply --dry-run=server` reports the object as `configured`, so the generation does move. This repairs today and nothing more. The next re-initialisation loses the extension again and reports success again. `make rebuild` is routine here, so a one-time repair is not a fix. **The durable half, in `alerting/rules.yaml`.** `PostgresQueryStatsAbsent` watches `cnpg_pg_stat_statements_info_tracked`, the exact series the `pg_stat_statements_info` block emits, and fires after an hour of absence. It follows the `CnpgBackupMetricMissing` pattern already in that file. One hour rather than five minutes because only the primary exports, so a failover or a rolling restart is a normal gap. **Expect it to fire on merge** and go quiet once the extension is back. That is the alert telling the truth about the current state. **The guard, in `tests/cnpg-custom-queries.sh`.** The manifest chain checks stay as they are. A new section requires the alert to exist and to name a series the ConfigMap actually produces, because a rule on a misspelled series is the same silence with extra YAML. ## Tests `tests/cnpg-custom-queries.sh` passes, including its existing damage cases. Three new damage cases against the runtime guard are each rejected: dropping the alert, misspelling the series, and changing the expression so it no longer tests absence. `kubectl apply --dry-run=server` accepts both files. ## Not done here The `binjovi` database still has no per-statement visibility of its own. That was the original goal, and it is a separate decision: `pg_stat_statements` is one cluster-wide hash table, so the deliberate design is to read it from the single `monitoring` database rather than install it per tenant. Once this export is alive again, Binjovi statements are already visible there through the `datname` label. https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

sean added 1 commit 2026-09-06 23:26:29 +00:00

fix(cnpg): observe the per-statement export instead of its manifests ... ac54b3455a

The pg_stat_statements export has produced nothing since 2026-09-04 and
every check passed. The library was preloaded, the ConfigMap was
delivered and named in the Cluster, and the Database object declared the
extension and reported applied: true. The extension was not in the
database, and the metric store held zero cnpg_pg_stat_statements series.

The data plane was re-initialised that day while the Database object
survived. CNPG does not reapply a Database whose metadata.generation has
not changed, so the operator never looked again and its stale success
outlived the database it described.

Declare the extension schema. That says where the objects go and, because
editing the spec increments the generation, it is the only supported way
to make the operator apply the extension again. That repairs today only:
the next re-initialisation loses it again and reports success again.

So add PostgresQueryStatsAbsent, which watches the series the ConfigMap
emits rather than the manifests meant to emit it, and require that rule
from the existing guard. Expect it to fire until the extension is back.

Claude-Session: https://claude.ai/code/session_015dkyh6itUFos3aSZnomh2w

binjovi-bot merged commit dc15d4b09c into trunk 2026-09-06 23:33:10 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

sean/seanfarm!392

No description provided.