acc: Prefix unique test resource names with CI run id for attribution and sweeping#5586
Merged
Conversation
… and sweeping Acceptance tests name cloud resources with a bare random string ($UNIQUE_NAME, e.g. 'osr5mzrrvzb73juixjoviti24y'). When such resources leak into shared test workspaces, nothing ties them back to a repo or CI run, which made a recent quota-exhaustion incident hard to diagnose and makes targeted cleanup impossible. When running against a cloud env (CLOUD_ENV set) with GITHUB_RUN_ID present, the harness now generates names of the form 'ci-<GITHUB_RUN_ID>-<random>', truncated to the same 26-character length as before so that length-constrained names built from $UNIQUE_NAME (e.g. 'app-$UNIQUE_NAME', exactly 30 chars, the app name limit) keep fitting. The character set stays lowercase alphanumerics plus '-', which tests already use adjacent to $UNIQUE_NAME in every resource type. Local runs and testserver runs are unchanged, and output masking is unaffected because the whole generated name (prefix included) is still replaced with [UNIQUE_NAME]. Also adds tools/sweep_test_resources.py, a small helper that lists (and with --delete removes) warehouses, pipelines and jobs whose names start with a given prefix, to be wired into CI job-end cleanup later. Co-authored-by: Isaac
Co-authored-by: Isaac
Drop the CLOUD_ENV gate: ciUniqueName already no-ops without a valid run id, and prefixing testserver runs in CI exercises the masking path everywhere instead of only on cloud runs. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 57d761e
24 interesting tests: 13 SKIP, 6 RECOVERED, 5 flaky
Top 21 slowest tests (at least 2 minutes):
|
denik
approved these changes
Jun 17, 2026
Python project_name validation rejects hyphens ("must consist of
letters, numbers, and underscores"). The CI run-id prefix
introduced by ciUniqueName uses hyphens (ci-<runID>-<random>),
causing bundle init to fail in the two default-python test
templates that embed $UNIQUE_NAME directly in project_name.
Fix: expose UNIQUE_NAME_SAFE (hyphens replaced with underscores)
alongside UNIQUE_NAME, add masking for it, and use it in the two
input.json.tmpl files and the integration_classic script.
Resources that accept hyphens (jobs, pipelines, apps, warehouses)
continue using the original $UNIQUE_NAME format.
Co-authored-by: Isaac
The previous ci-<runID>-<random> format injected hyphens into $UNIQUE_NAME, which is embedded into ~hundreds of test resource names. Hyphens are rejected by Python project names, Unity Catalog identifiers, and vector-search index names, so local acceptance jobs failed (e.g. bundle/templates/default-python and bundle/deployment/bind/vector_search_index). Switch the format to ci<runID>x<random>, which is purely lowercase-alphanumeric like the base32 name it replaces and is therefore valid in every naming context (including app names, which forbid underscores and uppercase). The run id is delimited from the random tail by the letter "x", keeping the sweep prefix ci<runID>x collision-free. This needs no per-template changes, so the UNIQUE_NAME_SAFE approach from the prior commit is reverted. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 6f51f96
20 interesting tests: 13 SKIP, 7 RECOVERED
Top 20 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes acceptance-test resource names attributable to the CI run that created them, and sweepable:
GITHUB_RUN_IDis numeric (any run, cloud or testserver),$UNIQUE_NAMEbecomesci<run-id>x<random>, truncated to the same 26-char total length as today (≥8 random chars guaranteed; falls back to the unprefixed name otherwise). The name stays purely lowercase-alphanumeric like the base32 name it replaces, so it remains valid in every context$UNIQUE_NAMEis embedded in — app names (which forbid underscores and uppercase) as well as Python project names and Unity Catalog identifiers (which forbid hyphens). The run id is delimited from the random tail by the letterx, which keeps the sweep prefixci<run-id>xcollision-free between runs whose ids share a prefix. Length preservation matters:app-$UNIQUE_NAMEsits exactly at the 30-char app-name limit;[UNIQUE_NAME]via whole-string replacement;GITHUB_RUN_IDare byte-for-byte unchanged; in-CI testserver runs now exercise the prefixed-name masking path on every PR;tools/sweep_test_resources.py: given a name prefix and CLI auth env, lists (dry-run) or deletes (--delete) matching warehouses/pipelines/jobs — ready to wire into CI job-end;TestCIUniqueNameunit coverage (missing/short/malformed/too-long run ids).Why
Today leaked test resources carry bare random names (e.g.
osr5mzrrvzb73juixjoviti24y) — nothing ties them to a repo, run, or owner. During a recent shared-workspace quota incident, attributing 100+ leaked warehouses to their source took hours of archaeology, and targeted cleanup was impossible. Run-scoped prefixes make leaks attributable at a glance and enable prefix-scoped sweeping.Tests
go build ./...,go vet ./acceptance,TestCIUniqueNamepass; local acceptance tests that assert[UNIQUE_NAME]in outputs (cmd/fs/cp,bundle/run_as/job_default) pass withGITHUB_RUN_IDset, confirming replacement behavior is unchanged.This pull request and its description were written by Isaac.