Chat tokenization fixes in generate.py & API

[vmpuri]

Currently, only the chat() function would encode chat-style messages into the correct format during an interactive session.

I adapted the ChatFormat class to create a generic _ChatFormatter base class - each one includes the function encode_dialog_prompt which can take a series of message objects and encode the system prompt & user/assistant messages correctly.

Allows us to load in a whole conversation (i.e. message objects in a completion request) & keep the API/server stateless.

Test:

Initially, I prompted with the following cURL request:

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama2",
    "stream": "true",
    "seed": "123",
    "max_tokens": "200",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant. Be as brief as possible, no yapping. Do not include additional details unless asked."
      },
      {
        "role": "user",
        "content": "List 3 early jet fighters."
      }
    ]}'

Then, I added the model's response and prompted it again with the following:

curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama2",
    "stream": "true",
    "seed": "123",
    "max_tokens": "200",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant. Be as brief as possible, no yapping. Do not include additional details unless asked."
      },
      {
        "role": "user",
        "content": "List 3 early jet fighters."
      },
      {
        "role": "system",
        "content": "Certainly! Here are three early jet fighters:\n\n1. Gloster Meteor (UK, 1943)\n2. Messerschmitt Me 262 (Germany, 1944)\n3. Lockheed P-80 Shooting Star (US, 1948)"
      },
      {
        "role": "user",
        "content": "Were there any notable ones from the Soviet Union or Japan?"
      }
    ]
  }'

And got the following response:

Yes, here are a few notable early jet fighters from the Soviet Union and Japan:

Soviet Union:

1. MiG-15 (1947)
2. Lavochkin La-150 (1948)
3. Yakovlev Yak-15 (1946)

Japan:

1. Nakajima J7W (1944)
2. Mitsubishi J8M (1945)
3. Kawasaki Ki-200 (1945)

These early jet fighters represented significant advancements in aeronautical technology and played important roles in their respective countries' militaries during the Cold War era.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1035

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d90e33b with merge base 147c292 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Jack-Khuu]

lgtm, remember to verify that python3 torchchat.py generate and chat still works properly for llama2/3 with these changes as well?

[Jack-Khuu]

Just checking that this is an intentional print

[vmpuri]

Yes - this prints out the prompt on the server side so that it's easy to track the full prompt solely from the server side.

However, this raises a larger issue in the generate/API stack - we need to replace print statements with a logger so that users can choose not to print these debug messages.

[Jack-Khuu]

Why is this is a pass again?

[vmpuri]

The callback function is only used in generate() for the CLI interactive chat to print results to stdout. I initially copied this code naively when refactoring the original generate.py and copied it over to openaiapi where it isn't used.

[prideout]

I believe this PR introduced a regression because encode_header now takes a string, but there is a spot in the Generator::chat method that is still passing a dictionary.

Fix proposal in #1061