Add direct state v3 - opt in only for record_deployment_history preview

[shreyas-goenka]

Why

experimental.record_deployment_history is the opt-in preview of the deployment metadata service (DMS). Previewing DMS changes the on-disk state format — but most bundles haven't opted in, and we must not force that format change on them. Two things have to hold:

• A preview user's upgraded state must be safe to keep using.
• A CLI that predates the format (an older release, or one that knows an older DMS protocol) must refuse to operate on it — but only for bundles that are actually using DMS.

What

• Non-DMS deploys keep writing state_version: 2 — unchanged.
• When a bundle previews DMS, the deploy upgrades its state to state_version: 3 and stamps dms_version: 1. UpgradeToDMS runs before the WAL is started so the bumped version is captured in the WAL header (it panics if called after).
• On load, the CLI accepts both v2 and v3 and never downgrades a v3 state; a state newer than it understands is rejected (reporting the real supported version, 3).
• The DMS protocol version (dms_version) is enforced by Open's WithDMS option, passed by process.go only when the bundle has opted into DMS — a bundle that hasn't opted in does not act on the recorded version, so it isn't blocked by it.
• record_deployment_history is rejected under the terraform engine (it drives DMS, which only the direct engine supports) instead of being silently ignored.

dmsStateVersion/currentDmsVersion are deliberately separate from currentStateVersion; the constant comments record this and the bump-to-4 procedure (step-by-step in TestStateSchemaVersions).

Tests

• Unit (dstate): state-version gating, WithDMS rejection (newer rejected / gated off when not opted in), UpgradeToDMS round-trip + panic-after-WAL, the bump-procedure tripwire, and a migrations-completeness invariant.
• Acceptance (record-deployment-history/):
  • state-upgrade — baseline (v2) → opt-in upgrade (v3) → older CLI rejects v3 → newer dms_version rejected → without the flag the same state is accepted.
  • terraform-unsupported — the flag is rejected under the terraform engine.

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Commit: cf6ef38

Run: 26828525415

[shreyas-goenka]

if the config is not set - the deployment just falls back to direct deplioyment.

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

12 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

/bundle/ - needs approval

8 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

/cmd/bundle/ - needs approval

Files: cmd/bundle/generate/dashboard.go, cmd/bundle/utils/process.go
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

General files (require maintainer)

Files: acceptance/acceptance_test.go
Based on git history:

• @denik -- recent work in bundle/direct/dstate/, bundle/phases/, cmd/bundle/generate/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @simonfaltum, @renaudhartert-db) can approve all areas.
See OWNERS for ownership rules.}

[denik]

What does this test? We know constants in golang work.

[shreyas-goenka]

This is a tripwire test that will trigger anytime someone does a version bump. This forces the author and reviewer to read the instructions above and not miss them.

[denik]

What happens if experimental flag is removed? Do we stop calling DMS but keep using normal direct engine? Do we keep version 3 or 2 in that case?

[shreyas-goenka]

Do we stop calling DMS but keep using normal direct engine?

Yes, we fall back to normal direct engine. It works fine because we'll continue to maintain direct state with WAL and resources.json as we preview.

[shreyas-goenka]

Do we keep version 3 or 2 in that case?

We keep version 3. The configuration remains the source of truth for whether to use direct or DMS (provide easy fallback for customers).