ci(e2e): add a curated @smoke gating leg, make the full generic leg informational#8759
ci(e2e): add a curated @smoke gating leg, make the full generic leg informational#8759jacekradko wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: db5acc0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited) Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
jacekradko
force-pushed
the
jacek/staging-e2e-validate-gate
branch
from
07c335c to
0eb5396
Compare
…nformational The generic leg runs ~210 tests against one shared staging instance, so any single post-retry flake reds the whole matrix and the staging run has been ~100% red, giving no signal. Split the gate: tag a handful of stable core-auth happy paths @smoke and run them as their own small matrix leg against just the two core apps, and mark the full generic leg continue-on-error so it still runs and uploads results but no longer fails the run. Because only gating legs (smoke + the framework legs) can fail the matrix job, needs.integration-tests.result is 'failure' only on a real gating-leg failure, so the report job's Slack notification fires on meaningful failures instead of on generic-only flake. The initial @smoke set is intentionally conservative (sign-in with password / instant password / modal, sign-up with password, sign-out) and is just tags, so it can grow as flake is brought under control.
jacekradko
force-pushed
the
jacek/staging-e2e-smoke-leg
branch
from
3d2c6ea to
db5acc0
Compare
1 participant
The
genericleg runs ~210 tests against one shared staging instance, so any single post-retry flake reds the whole matrix. That is the main reason the staging run has been ~100% red and gives no signal.This splits the gate. A handful of stable core-auth happy paths are tagged
@smokeand run as their own small matrix leg against just the two core apps, and the fullgenericleg becomescontinue-on-errorso it still runs and uploads results but no longer fails the run. Because only gating legs (smoke plus the framework legs) can fail the matrix job,needs.integration-tests.resultisfailureonly on a real gating-leg failure, so the existing report job notifies on meaningful failures instead of on generic-only flake. The clerk_go commit-status block stays commented out; when it is wired up (follow-up) it inherits the same gating semantics.The load-bearing choice here is the
@smokeset, since it is what gates. I kept it deliberately conservative: five happy paths that never appeared in a failed/flaky list across the recent runs (sign-in with password, with instant password, the modal variant, sign-up with password, and a single-flow sign-out), run againstreact.vite.withEmailCodesandnext.appRouter.withEmailCodes. It is just tags, so it should grow as the rate-limit flake is brought under control. A few tests now run in both legs (smoke and the informational generic), which is cheap insurance and lets the smoke result come from the clean, low-load leg rather than the noisy 210-test one.Stacked on #8757.