Stand up dual stable-ABI build harness and port nms as the first op.

[adabeyta]

This PR stands up the stable-ABI build path and ports nms (CPU, CUDA, quantized-CPU, MPS) as the first op. The intent is to validate the harness/per-backend patterns on nms.

Structure: Dual Extension

• torchvision._C: the existing extension, now built from all not-yet-ported sources.
• torchvision._C_stable: the new extension, built only from migrated sources (currently nms).

Why two extensions?
For a per op rollout a separate compilation unit helps to scope the pin and keep the tree buildable at every step of an incremental port (torchcodec migrated in place because it converted everything in one shot). Once all ops are stable remove legacy unstable compilation.

Layout Changes
setup.py: splits sources between the two extensions.

• STABLE_SOURCES + _stable()/_not_stable() helpers split the source globs, so migrated files compile into _C_stable and everything else stays in _C.
• make_C_stable_extension(): builds the new target with the -DTORCH_TARGET_VERSION pin.
• ensure_hipified() hoisted to a single idempotent step both extensions call (runs once/with no build-order dependency between them).

Registration: schema declaration/per-backend kernel binding.

• Schema via STABLE_TORCH_LIBRARY_FRAGMENT, which coexists with the classic TORCH_LIBRARY_FRAGMENT still in _C both register into the same torchvision namespace across two .so.
• Per-backend kernels bound with STABLE_TORCH_LIBRARY_IMPL + TORCH_BOX.
• The nms() C++ entrypoint dispatches through torch_call_dispatcher.

StableABICompat.h: shared compat header one place for ATen ops not yet wrapped in stable ops.h (placeholder until upstream.

• Wraps sort, index_select, and masked_select through the dispatcher escape hatch, so each kernel doesn't re-roll the boilerplate.
• Mirrors torchcodec's header of the same name

vision_stable.cpp: extension-root file for _C_stable, the stable mirror of vision.cpp.

• Holds the Windows PyInit__C_stable stub. This is extension-level glue, not tied to any op, so it lives here rather than in a kernel file.
• Added to STABLE_SOURCES so it builds into _C_stable and stays out of _C.

Missing Pytorch NMS MPS Stable-ABI Port Support

1. MPS scalar arguments:

   The nms schema takes a float iou_threshold, which the Metal kernel needs as a scalar argument (constant float & iou_threshold [[buffer(3)]]). With no scalar-arg setter in the shim, that float is currently carried as a 1-element float32 tensor bound via set_arg_tensor. shim_mps.h exposes only aoti_torch_mps_set_arg_tensor and aoti_torch_mps_set_arg_int

   Missing:

   • aoti_torch_mps_set_arg_double
   • aoti_torch_mps_set_arg_float
   • aoti_torch_mps_set_arg_bytes
   • aoti_torch_mps_set_arg_scalar.

   TODO: Workaround (for now) packs iou_threshold into a 1-element float32 tensor (new_empty + fill_) and bind it at buffer(3) via set_arg_tensor. The kernel reads the same 4 bytes as constant float&, so it's behavior-identical. File PR for missing mps_set_args.

2. Metal source isolation.
   Made nms Metal source into a standalone nms_metal_shader.h. The shared mps_kernels.h can't be used under the pin because it pulls in ATen/native/mps/OperationUtils.h and instantiates at::native::mps::MetalShaderLibrary. Derives from same isolation rationale as the dual extension (legacy _C keeps using the full mps_kernels.h)

Stable-ABI audit (torch-abi-audit)

Ran torch-abi-audit against the built torchvision package to verify the migrated extension stays on the stable surface and never reaches into at:: / c10:: / torch::jit:: internals.

_C_stable.so audits as STABLE — 67 stable-shim symbols, 0 unstable. The legacy _C.so and the
unrelated image.so stay UNSTABLE, which is expected: only nms has migrated so far, so the
package-level verdict remains UNSTABLE until the remaining ops move into _C_stable (same
mid-migration shape as torchcodec).

Package: torchvision
  Torch ABI:   UNSTABLE
  CPython ABI: n/a
  Extensions:  0
  Bundled libs: 3
  -- bundled libs --
    [UNSTABLE] [not-abi3] _C.so         (stable_shim=0,  unstable=110)
    [STABLE  ] [not-abi3] _C_stable.so  (stable_shim=67, unstable=0)
    [UNSTABLE] [uses-private-api] image.so  (stable_shim=0,  unstable=36)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9524

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm] MI350 CI jobs will have longer queue times due to CI migration

✅ You can merge normally! (1 Unrelated Failure)

As of commit 8f2660f with merge base 5c6ea42 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Tests / unittests-macos (3.10, macos-m1-stable) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-cla]

Hi @adabeyta!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[NicolasHug]

Thanks a lot @adabeyta ! The Macos job failure is unrelated and I'm trying to address it in #9526, we can merge this regardless.

[github-actions]

Hey @NicolasHug!

You merged this PR, but no labels were added.
The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py