test: add nanov3 prefill decode test

[ZhiyuLi-Nvidia]

What does this PR do ?

Background add nanov3 prefill decode test.

Vllm prefill and decode should generate consistent logprob. It was an issue before vllm<0.17.0 and fixed in vllm==0.17.0. Add the test to guard vllm prefill and decode logprob consistency.

Issues

It was an issue as per #2100

Fixed after vllm bump up.

Background:
I’d expect bump up vllm version to vLLM 0.17.0 would resolve the issue:
Some findings are as follows:
old vllm version is self-conflicting while megatron is good

vLLM decode  vs vLLM prefill:     TME = 322.490753   DIVERGED   <== vllm is self-conflicting 
vLLM decode  vs Megatron prefill: TME = 313.127106   DIVERGED   <== what we saw
vLLM prefill vs Megatron prefill: TME = 1.030557     HEALTHY   <==  megatron is good and aligns well with vllm 

prefillvllm prefill is like pass all prompt generated tokens to vllm and let it calculate logprobs, it is similar as a single forward  pass in training.
after bumping up vllm to 0.17.0 using the container /lustre/fsw/portfolios/coreai/users/terryk/enroot-images/gitlab-master.nvidia.com/terryk/images/nemo-rl:hemil-automodel-transformers-v5-9db945aa4.squashfs

  vLLM decode  vs vLLM prefill:     TME = 1.032336
  vLLM decode  vs Megatron prefill: TME = 1.032111
  vLLM prefill vs Megatron prefill: TME = 1.031372all 64 healthy

I think the root cause should be relevant to the prefill/decode kernel with mamba, kv cache and they were fixed with vllm bump up.

Usage

See tests.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• [x ] Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ZhiyuLi-Nvidia]

/ok to test 30f2253

[ZhiyuLi-Nvidia]

/ok to test d255ec9

[terrykong]

approved, but let's see where the v5 PR is. if it's almost done by the time this CI finishes, let's just do this in a following up PR to not slow down that PR. check with @yuki-97 on her preference on merging this one

[yuki-97]

thanks @ZhiyuLi-Nvidia @terrykong !
since CI passes and the PR is independent with other codes, it should be safe. I'll directly merge it to the bump PR.