Simple and stable Inference APIs

docs/llama_mistral.md

@@ -37,12 +37,10 @@ Architecturally Llama-2, Llama-3 and Mistral-7b are very similar. As such Megatr
   - [Download Huggingface checkpoints](#download-huggingface-checkpoints)
   - [Convert checkpoint format](#convert-checkpoint-format)
     - [Huggingface format](#huggingface-format)
-  - [(Optional) Validate checkpoints](#optional-validate-checkpoints)
   - [Launch model](#launch-model)
 - [Mistral-7b](#mistral-7b)
   - [Download Huggingface checkpoints](#download-huggingface-checkpoints)
   - [Convert checkpoint format](#convert-checkpoint-format)
-  - [(Optional) Validate checkpoints](#optional-validate-checkpoints)
   - [Launch model](#launch-model)
 - [Other Llama-like model support](#other-llama-like-model-support)
 - [Known numerical differences](#known-numerical-differences)
@@ -210,14 +208,6 @@ python Megatron-Bridge/examples/conversion/convert_checkpoints.py import \
 
 After this conversion, we are ready to load the checkpoints into a Megatron GPT model.
 
-## (Optional) Validate checkpoints
-
-A Megatron-LM text generation server for Llama3 can be launched using the script `examples/inference/llama_mistral/run_text_generation_llama3.sh <PATH_TO_CONVERTED_CORE_CHECKPOINT> <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT>`. For Llama3.1, please use `examples/inference/llama_mistral/run_text_generation_llama3.1.sh`.
-
-Once running, query the server with `curl 'http://<TEXT_GENERATION_SERVER_IP>:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8'  -d '{"prompts":["<SOME_PROMPT>"], "tokens_to_generate":100, "top_k":1}'`.
-
-A reference generation for comparison can be obtained from the Huggingface transformers library by running `python examples/llama_mistral/huggingface_reference.py --model_path <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT> --prompt <SOME_PROMPT>`.
-
 ## Launch model
 
 If loading for either inference or finetuning, use the following arguments for Llama 3.0:
@@ -314,14 +304,6 @@ python Megatron-Bridge/examples/conversion/convert_checkpoints.py import \
 
 After this conversion, we are ready to load the checkpoints into a Megatron GPT model.
 
-## (Optional) Validate checkpoints
-
-A Megatron-LM text generation server for Mistral-7B can be launched using the script `examples/inference/llama_mistral/run_text_generation_mistral.sh <PATH_TO_CONVERTED_MCORE_CHECKPOINT> <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT>`.
-
-Once running, query the server with `curl 'http://<TEXT_GENERATION_SERVER_IP>:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8'  -d '{"prompts":["<SOME_PROMPT>"], "tokens_to_generate":100, "top_k":1}'`.
-
-A reference generation for comparison can be obtained from the Huggingface transformers library by running `python examples/inference/llama_mistral/huggingface_reference.py --model_path <PATH_TO_DOWNLOADED_HUGGINGFACE_CHECKPOINT> --prompt <SOME_PROMPT>`.
-
 ## Launch model
 
 If loading for either inference or finetuning, use the following arguments:

examples/inference/gpt/gpt_dynamic_inference.py → examples/inference/advanced/gpt_dynamic_inference.py

@@ -19,7 +19,7 @@
     os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
 )
 
-from examples.inference.gpt.utils import (
+from examples.inference.utils import (
     Request,
     build_dynamic_engine_setup_prefix,
     build_requests,

examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py → examples/inference/advanced/gpt_dynamic_inference_with_coordinator.py

@@ -13,7 +13,7 @@
 import torch
 import torch.distributed as dist
 
-from examples.inference.gpt.utils import Request, build_dynamic_engine_setup_prefix, build_requests
+from examples.inference.utils import Request, build_dynamic_engine_setup_prefix, build_requests
 from megatron.core.inference.engines import DynamicInferenceEngine
 from megatron.core.inference.engines.dynamic_engine import EngineState
 from megatron.core.inference.inference_client import InferenceClient

examples/inference/gpt/gpt_static_inference.py → examples/inference/advanced/gpt_static_inference.py

@@ -29,7 +29,7 @@
 import json
 from typing import List
 
-from examples.inference.gpt.utils import build_requests
+from examples.inference.utils import build_requests
 from megatron.inference.utils import add_inference_args, get_model_for_inference
 from megatron.training import get_args, get_tokenizer, print_rank_0
 from megatron.training.initialize import initialize_megatron