feat(ppo): Megatron value-model sequence packing + context parallelism

[bg51717]

What does this PR do ?

Adds sequence packing and context parallelism (CP) support to the Megatron PPO value model, building on the in-model value head from PR #2825. Also fixes a value loss-scaling bug (the per-microbatch value loss was averaged instead of summed).

Issues

Closes #2687 (PR 2 of 2). Stacked on #2825 — the base branch is ppo-mcore-value-head until #2825 merges, after which this retargets to main.

Usage

Context parallelism for the value model requires sequence packing (matching Megatron-Core). Both are configured like the policy:

value:
  sequence_packing:
    enabled: true                    # now supported
  megatron_cfg:
    context_parallel_size: 2         # now supported (requires sequence_packing)
  make_sequence_length_divisible_by: 4   # must be a multiple of 2*CP

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Test

Tests run (all pass):

• Unit (tests/unit/models/value/test_megatron_value_worker.py, mcore, 2 GPU): packed-unpack (variable-length, CPU), packed train step (incl. PP2), and get_values equivalence across SP / dynamic batching / PP / sequence packing / CP (CP=1 vs CP=2, bf16), plus DP/TP regression. 19 passed.
• Functional (tests/functional/ppo_megatron.sh): passes.
• Convergence (Qwen2.5-1.5B, GSM8K, 1n8g, 40 steps), train reward / val accuracy — the two nightly value-parallelism recipes:

valuetp2sp-dynbatch (TP2 + SP + dynamic batching; re-validates the loss-scaling fix does not regress convergence): 0.85 / 0.76

valuetp2sp-pp2cp2-pack (TP2 + SP + PP2 + CP2 + sequence packing): 0.84 / 0.72

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[bg51717]

/ok to test 6cfe536

[bg51717]

/ok to test 6cfe536