VariantGet RFC#58
Merged
Merged
Conversation
AdamGS
force-pushed
the
adamg/update-variant-canonical
branch
from
c41bb3c to
09a29d1
Compare
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
AdamGS
force-pushed
the
adamg/update-variant-canonical
branch
from
09a29d1 to
26f1e73
Compare
|
I think that I am a bit dumb when reading this - what would be advantage of skipping shredded values in the canonical array? |
Collaborator
Author
|
That's the current thing, it makes the canonical array a really weird thing that is basically a pass-through to a bunch of things, with delicate rules around it to make sure everything is pushdown down. |
gatesn
reviewed
May 5, 2026
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
robert3005
approved these changes
May 12, 2026
AdamGS
added a commit
to vortex-data/vortex
that referenced
this pull request
May 15, 2026
## Summary This PR includes two big changes as Variant moves closer to readiness. 1. Potentially holding the `shredded` child of a variant array in the canonical VariantArray 2. A `VariantGet` expression that can pull extract data out of variant arrays, either in a typed way or as a more opaque `Variant`. For reviewers, some relevant context might be: 1. The [VariantGet](vortex-data/rfcs#58) RFC: this RFC takes some lessons I've learned working on this into account and reflects my updated view of this problem. 2. The original [Variant type](https://vortex-data.github.io/rfcs/rfc/0015.html) RFC I think the Parquet spec is also a pretty good read and a very heavy influence of this work - [`Shredding`](https://parquet.apache.org/docs/file-format/types/variantshredding/) and the [`Variant type`](https://parquet.apache.org/docs/file-format/types/variantencoding/). --------- Signed-off-by: "Adam Gutglick" <adam@spiraldb.com> Signed-off-by: Adam Gutglick <adam@spiraldb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RFC for the
VariantGetexpression, with lessons and thoughts learned through vortex-data/vortex#7494.