Skip to content

refactor: meta: reorganize txn fetched records#20033

Merged
drmingdrmer merged 1 commit into
databendlabs:mainfrom
drmingdrmer:meta-txn-mgr
Jun 21, 2026
Merged

refactor: meta: reorganize txn fetched records#20033
drmingdrmer merged 1 commit into
databendlabs:mainfrom
drmingdrmer:meta-txn-mgr

Conversation

@drmingdrmer

@drmingdrmer drmingdrmer commented Jun 21, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

refactor: meta: reorganize txn fetched records

Summary

Transaction get results are now modeled as fetched records and the txn module layout groups related record handles and MetaTxn internals together.

Details

The former for-update handle names now use FetchedRecord, PresentRecord, and AbsentRecord, which describes the record returned by a get operation instead of implying update behavior. get_for_update still names the read mode that installs the CAS guard.

MetaTxn manager, read state, test helpers, and tests now live under the meta_txn module, while fetched record handles live under fetched_record. Public reexports remain available from the meta API crate root.

refactor: txn: encode for-update existence state

Summary

MetaTxn for-update reads now carry their asserted existence state in the handle type, so callers that require present or absent records can continue with a narrower API after the guard succeeds.

Details

some_or_unknown_error now returns a present-only handle with direct access to the stored SeqV, while none_or_exist_error returns an absent-only handle that only supports creating the guarded record. This removes optional-value checks from callers after existence has already been validated.

Table tag reference errors now report the tag name without exposing the table id, with the operation context appended to the message.

refactor: txn: generalize KV error type from MetaError to KV::Error

Summary

The transaction layer previously hard-coded MetaError as the KV error
type throughout send_txn, ForUpdate, MetaTxn, and MetaTxnManager.
This change propagates KV::Error (or bounded associated types) instead,
making the txn module usable with any KVApi implementation.

Details

  • send_txn is now generic over KV: kvapi::KVApi + ?Sized, returning
    KV::Error instead of MetaError.
  • ForUpdate is split into two impl blocks: a no-bound block for
    read-only accessors and delete, and a KV::Error: From<InvalidArgument>
    block for put.
  • MetaTxn gains a similar split: a no-bound block (new, delete),
    a From<PbDecodeError> block (get, get_for_update, read_key), and
    a From<InvalidArgument> block (put).
  • decode_raw becomes generic over error E: From<PbDecodeError> instead
    of returning a fixed MetaError.
  • MetaTxnManager::run narrows the caller error bound from
    From<MetaError> to From<KV::Error>.
feat(meta): add ForUpdate existence-guard methods

Summary

Add two guard methods to the ForUpdate transaction handle that turn a read
key's presence or absence into the key's own declared error, and adopt them in
rename_dictionary. They replace the presence checks a CAS step otherwise
open-codes, while keeping the error at the key's own layer.

Details

  • some_or_unknown returns the read value, or the key's unknown_error when the
    key is absent — the precondition for a key a write depends on.
  • none_or_exist returns unit, or the key's exist_error when the key is present
    — the precondition for a key that must be free before it is created.
  • Both yield exactly the error the key declares via MetaServiceKeyErrorBuilder,
    leaving conversion (e.g. to AppError) to the caller, so ForUpdate stays
    independent of the higher-level error enums.
  • rename_dictionary uses both in place of its inline unknown_error/exist_error
    checks.
  • Tests cover the present and absent path of each, asserting the exact error
    text; DatabaseId keys are used for their stable, concrete error Display.
feat(meta): add MetaTxn CAS transaction helper for KV API

Summary

Introduce MetaTxn, a reusable helper that models one optimistic-concurrency
transaction attempt against the KV backend, and MetaTxnManager, which drives a
closure as a compare-and-swap retry loop with a fresh transaction per attempt.
The first API, DictionaryApi::rename_dictionary, is migrated onto this model;
its observable behavior is unchanged.

Details

  • MetaTxn snapshots reads in a per-key cache and turns reads taken "for update"
    into eq_seq guards at commit, so a value a write depends on cannot be left
    unguarded. execute() consumes self, making "commit at most once" a
    compile-time guarantee. Its read and write sets sit behind Arc<Mutex<..>> so
    the manager and the closure it drives operate on one shared set.
  • MetaTxnManager binds a KV backend and a retry policy and runs a closure as a
    CAS loop, handing it a fresh MetaTxn each attempt; the closure takes the
    transaction by value to remain Send under #[async_trait].
  • ForUpdate is a handle to a key read for update that writes back to that same
    key and transaction, so a staged write always targets the guarded key.
  • rename_dictionary is reimplemented as two guards (source must exist, target
    must be free) followed by moving the dictionary id, replacing the previous
    hand-rolled transaction loop.
  • Bump databend-meta-client 260205.11.0 -> 260205.12.0 (workspace dependency and
    patch tag) for the KV transaction APIs the helper builds on.
  • Add an in-memory KVApi and unit tests covering read caching, guard arming,
    retry-on-conflict, and write staging.

The crate now re-exports MetaTxn, MetaTxnManager, and ForUpdate.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Refactoring

Related Issues


This change is Reviewable

@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis

Workflow: 27895039380

📊 Summary

  • Total Jobs: 87
  • Failed Jobs: 1
  • Retryable: 0
  • Code Issues: 1

NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • linux / PROXY routing benchmark (statistics): Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c18e3df5e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/meta/api/src/txn/meta_txn/mod.rs

@xp-trumpet xp-trumpet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xp-trumpet reviewed 16 files and all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on drmingdrmer).

# Summary

Add a typed transaction helper for meta KV operations that records reads, stages writes, and retries failed CAS commits through a small manager API.

# Details

`MetaTxnManager` now drives retryable transaction closures, while `MetaTxn` keeps the read and write sets for one attempt. Reads registered with `get_for_update` become commit-time `eq_seq` guards, and fetched records can assert present or absent state before staging writes back to the same key.

Dictionary rename and table-tag drop use the helper to replace hand-built transaction loops. The helper uses the KV API associated error type instead of forcing `MetaError`, and the databend-meta-client dependency is updated to the version needed by the generic transaction path.

@xp-trumpet xp-trumpet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xp-trumpet reviewed 3 files and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on drmingdrmer).

@drmingdrmer drmingdrmer merged commit 0d91d2b into databendlabs:main Jun 21, 2026
176 of 178 checks passed
@drmingdrmer drmingdrmer deleted the meta-txn-mgr branch June 21, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants