fix: llama 3.1 405b fp8

[zhyncs]

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
3. Modify documentation as needed, such as docstrings or example tutorials.

[zhyncs]

# server
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-FP8 --disable-radix-cache --tp 8

# client
curl http://127.0.0.1:30000/v1/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Meta-Llama-3.1-405B-FP8", "prompt": "What the weather is like today?", "max_tokens": 100, "temperature": 0.7}'

before

{"id":"84a8d16010dd44e2b60f6423fe6624d0","object":"text_completion","created":1721835240,"model":"meta-llama/Meta-Llama-3.1-405B-FP8","choices":[{"index":0,"text":"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!","logprobs":null,"finish_reason":"FINISH_LENGTH: 100"}],"usage":{"prompt_tokens":8,"total_tokens":108,"completion_tokens":100}}

after

{"id":"9257b9d53660410a8ce2f7ffe2c60478","object":"text_completion","created":1721835356,"model":"meta-llama/Meta-Llama-3.1-405B-FP8","choices":[{"index":0,"text":" If we want to know it, we have to look at the sky. If the sky is blue and the sun is shining, we can say that it is fine weather. If the sky is cloudy and it is raining, we can say that it is bad weather.\nIt is not always easy to say what the weather is like. Sometimes the sky is clear and the sun is shining, but the wind is blowing and it is cold. Then we can say that it is windy weather.\nIt is","logprobs":null,"finish_reason":"FINISH_LENGTH: 100"}],"usage":{"prompt_tokens":8,"total_tokens":108,"completion_tokens":100}}

[zhyncs]

This fix will not affect the output of previously normal models.

python3 playground/reference_hf.py --model  meta-llama/Meta-Llama-3.1-8B-Instruct

python3 -m sglang.bench_latency --model  meta-llama/Meta-Llama-3.1-8B-Instruct --correct --output-len 16 --trust-remote-code

prefill logits tensor([ 5.2266,  2.0352,  1.0605,  ..., -5.2344, -5.2344, -5.2344],
       device='cuda:0')
prefill logits tensor([ 4.9609,  3.5312,  2.2852,  ..., -3.7344, -3.7344, -3.7344],
       device='cuda:0')
prefill logits tensor([10.2500,  3.1348,  2.1699,  ..., -2.6797, -2.6797, -2.6797],
       device='cuda:0')

prefill logits (final) tensor([[ 5.2188,  2.0312,  1.0781,  ..., -5.2188, -5.2188, -5.2188],
        [ 5.0000,  3.5625,  2.3125,  ..., -3.7344, -3.7344, -3.7344],
        [10.1875,  3.1406,  2.1719,  ..., -2.6875, -2.6875, -2.6875]],
       device='cuda:0', dtype=torch.bfloat16)