Skip to content

feat(copy): improve ColumnMissingError with contextual advice#20029

Open
youngsofun wants to merge 1 commit into
databendlabs:mainfrom
youngsofun:improve-column-missing-error-advice
Open

feat(copy): improve ColumnMissingError with contextual advice#20029
youngsofun wants to merge 1 commit into
databendlabs:mainfrom
youngsofun:improve-column-missing-error-advice

Conversation

@youngsofun

@youngsofun youngsofun commented Jun 20, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

fix: #18834

Enhance error messages for missing columns during COPY INTO to help users diagnose and resolve issues faster.

When a column is missing, the error now includes contextual advice:

  • Shows the current MISSING_FIELD_AS setting
  • Indicates when the column is not nullable (for MISSING_FIELD_AS=NULL case)
  • Suggests COLUMN_MATCH_MODE=CASE_INSENSITIVE when a case-different match exists in the source data
  • Suggests COPY INTO <table> FROM (SELECT $1 FROM @<stage>) when the target table has a single VARIANT column

Applied consistently across NDJSON, Parquet, and ORC formats.

Example before:

Missing value for column 2 (b Int32)

Example after:

Missing value for column 2 (b Int32). current FILE_FORMAT option: MISSING_FIELD_AS=NULL. the column is not nullable

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jun 20, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 144d6fc090

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/common/storage/src/copy.rs
@youngsofun youngsofun force-pushed the improve-column-missing-error-advice branch from 144d6fc to 14978ca Compare June 20, 2026 03:11
@youngsofun youngsofun marked this pull request as draft June 20, 2026 03:14
@youngsofun youngsofun marked this pull request as ready for review June 20, 2026 04:41

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 23e956fb0f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/storages/common/stage/src/read/columnar/projection.rs Outdated
Enhance error messages for missing columns during COPY INTO to help
users diagnose and resolve issues faster:

- Show the current MISSING_FIELD_AS setting
- Indicate when a column is not nullable (for MISSING_FIELD_AS=NULL)
- Suggest COLUMN_MATCH_MODE=CASE_INSENSITIVE when a case-different
  match exists in the source data
- Suggest SELECT $1 pattern when target table has a single VARIANT column

Applied consistently across NDJSON, Parquet, and ORC formats.
The new advice field uses #[serde(default)] for rolling upgrade compatibility.
@youngsofun youngsofun force-pushed the improve-column-missing-error-advice branch from 105859f to f65b090 Compare June 20, 2026 08:40
@youngsofun youngsofun requested review from bohutang and sundy-li June 22, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Missing value for column 0 (data Variant)

1 participant