Describe the bug
For phi3.4-mini-instruct model (presumably other models as well), the generated responses for a multi-prompt chat sequence appear to be sensitive to the order of EOS tokens listed within the genai_config.json
To Reproduce
Steps to reproduce the behavior:
- Prepare Phi-3.5 mini model, as per the instruction here: https://github.com/microsoft/onnxruntime-genai/tree/main/examples/c#phi-35-mini
- Run phi3.exe sample, with first prompt as
what color are apples?, and second prompt as what about grapes?
You'll find that for the second prompt, there is a 'good' response, in that it realizes you are asking what the color of grapes are (based on context around the first prompt / response).
This is what I get:
>phi3.exe Phi-3.5-mini-instruct-onnx\cpu_and_mobile\cpu-int4-awq-block-128-acc-level-4 cpu
-------------
Hello, Phi-3!
-------------
C++ API
Creating config...
Creating model...
Creating tokenizer...
Prompt: (Use quit() to exit) Or (To terminate current output generation, press Ctrl+C)
what color are apples?
Generating response...
Apples come in a variety of colors, depending on the species and cultivar. Here are some common apple colors you might encounter:
1. Red: This is the most iconic apple color, often associated with apples in general. Red apples can range from a light pinkish-red to a deep, vibrant red. Examples include Red Delicious, Gala, and Fuji apples.
2. Yellow: Yellow apples are less common than red ones but still quite popular. They can be found in varieties like Golden Delicious, Golden Harvest, and Yellow Transparent.
3. Green: Green apples are often tart and crisp, with a bright, vibrant green color. Examples include Granny Smith and Fuji apples.
4. Pink: Some apple varieties have a pinkish hue, such as the Pink Lady (also known as Envy) and the Pink Pearl.
5. Purple: While not as common, there are purple apple varieties like the Purple Haze.
6. Multicolored: Some apples have a mix of colors, such as the Rainbow Crisp, which has streaks of red, yellow, and green.
It's important to note that the color of an apple can also be influenced by factors like ripeness and growing conditions. For example, a green apple might appear more yellowish when fully ripe.
In summary, apples come in a wide range of colors, including red, yellow, green, pink, and purple, with some varieties even displaying a mix of colors. The specific color of an apple can vary depending on the species, cultivar, and ripeness.
-------------
Prompt length: 22, New tokens: 376, Time to first: 0.31s, Prompt tokens per second: 71.62 tps, New tokens per second: 17.11 tps
-------------
Prompt: (Use quit() to exit) Or (To terminate current output generation, press Ctrl+C)
what about grapes?
Generating response...
Grapes, like apples, come in a variety of colors, which can vary depending on the grape variety and its ripeness. Here are some common grape colors and their corresponding varieties:
1. Green:
- Green table grapes: These grapes are typically seedless and have a sweet, juicy flavor. Examples include the Thompson Seedless and Flame seedless.
- Green wine grapes: These grapes are used to make white wine. Examples include Sauvignon Blanc and Chardonnay.
2. Red:
- Red table grapes: These grapes have a deep, rich color and a sweet, juicy flavor. Examples include the Concord and Niagara varieties.
- Red wine grapes: These grapes are used to make red wine. Examples include Cabernet Sauvignon, Merlot, and Pinot Noir.
3. Black:
- Black currant grapes: These grapes have a dark, almost black color and a sweet, tangy flavor. They are often used in jams, jellies, and desserts.
- Black Muscat grapes: These grapes have a deep, dark color and a sweet, musky flavor. They are often used in winemaking and desserts.
4. Purple:
- Purple table grapes: These grapes have a deep, purple color and a sweet, juicy flavor. Examples include the Concord and Niagara varieties.
- Purple wine grapes: These grapes are used to make red wine. Examples include Cabernet Sauvignon, Merlot, and Pinot Noir.
5. Blue:
- Blue grape varieties: While not as common as other colors, there are grape varieties with a blueish hue. Examples include the Avalon and Black Desert varieties.
The color of grapes can also be influenced by factors such as ripeness, growing conditions, and grape variety. For example, green grapes are typically less ripe and have a higher acidity, while red grapes are usually riper and have a sweeter flavor.
In summary, grapes come in a range of colors, including green, red, black, purple, and even blue. The specific color of a grape can vary depending on the grape variety and its ripeness. Green grapes are typically less ripe and have a higher acidity, while red grapes are usually riper and have a sweeter flavor. Blue grape varieties are less common but still exist.
-------------
Prompt length: 9, New tokens: 999, Time to first: 0.21s, Prompt tokens per second: 43.81 tps, New tokens per second: 23.11 tps
-------------
- Now, within the
genai_config.json, swap the order of the EOS tokens, making them:
"eos_token_id": [
32000,
32001,
32007
],
When you run the same test with this configuration, you'll find that the response for the 2nd prompt appears to have lost some of the contextual awareness:
>phi3.exe Phi-3.5-mini-instruct-onnx\cpu_and_mobile\cpu-int4-awq-block-128-acc-level-4 cpu
-------------
Hello, Phi-3!
-------------
C++ API
Creating config...
Creating model...
Creating tokenizer...
Prompt: (Use quit() to exit) Or (To terminate current output generation, press Ctrl+C)
what color are apples?
Generating response...
Apples come in a variety of colors, depending on the species and cultivar. Here are some common apple colors you might encounter:
1. Red: This is the most iconic apple color, often associated with apples in general. Red apples can range from a light pinkish-red to a deep, vibrant red. Examples include Red Delicious, Gala, and Fuji apples.
2. Yellow: Yellow apples are less common than red ones but still quite popular. They can be found in varieties like Golden Delicious, Golden Harvest, and Yellow Transparent.
3. Green: Green apples are often tart and crisp, with a bright, vibrant green color. Examples include Granny Smith and Fuji apples.
4. Pink: Some apple varieties have a pinkish hue, such as the Pink Lady (also known as Envy) and the Pink Pearl.
5. Purple: While not as common, there are purple apple varieties like the Purple Haze.
6. Multicolored: Some apples have a mix of colors, such as the Rainbow Crisp, which has streaks of red, yellow, and green.
It's important to note that the color of an apple can also be influenced by factors like ripeness and growing conditions. For example, a green apple might appear more yellowish when fully ripe.
In summary, apples come in a wide range of colors, including red, yellow, green, pink, and purple, with some varieties even displaying a mix of colors. The specific color of an apple can vary depending on the species, cultivar, and ripeness.
-------------
Prompt length: 22, New tokens: 376, Time to first: 0.29s, Prompt tokens per second: 75.83 tps, New tokens per second: 17.02 tps
-------------
Prompt: (Use quit() to exit) Or (To terminate current output generation, press Ctrl+C)
what about grapes?
Generating response...
Grapes are delicious, nutritious fruits that come in various colors, sizes, and flavors. Here are some interesting facts about grapes:
1. **Varieties**: There are many different types of grapes, including red, green, black, and white varieties. Some popular ones include Concord, Thompson Seedless, and Flame Seedless.
2. **Nutritional Value**: Grapes are low in calories and rich in vitamins, minerals, and antioxidants. They are a good source of vitamins K and C, and also contain resveratrol, a compound that may have health benefits.
3. **Health Benefits**: Eating grapes may contribute to heart health, thanks to their antioxidant properties. They are also believed to help with weight management, blood sugar control, and may have anti-inflammatory effects.
4. **Growing Conditions**: Grapes thrive in warm climates and require a lot of sunlight. They are often grown in vineyards where they are carefully cultivated and harvested.
5. **Wine Production**: While grapes are commonly eaten fresh, they are also the primary ingredient in wine production. The process of fermenting grapes to make wine is a centuries-old tradition that varies by region and grape variety.
6. **Seedless Varieties**: Many grapes available in grocery stores are seedless, which means they have been bred to not contain seeds. This is convenient for snacking and cooking.
7. **Allergies**: Some people may be allergic to grapes, although this is relatively rare. If you suspect an allergy, it's best to consult with a healthcare provider.
8. **Culinary Uses**: Grapes can be eaten fresh, dried into raisins, or used in cooking and baking. They can also be made into jams, jellies, and juices.
9. **Cultural Significance**: Grapes have been cultivated for thousands of years and hold cultural significance in many parts of the world. They are often associated with festivals and celebrations.
10. **Pests and Diseases**: Like all crops, grapes can be affected by pests and diseases. Vineyard management practices are important to ensure healthy grape production.
Whether you enjoy them fresh, dried, or as part of a wine, grapes are a versatile and enjoyable fruit.
-------------
Prompt length: 9, New tokens: 992, Time to first: 0.22s, Prompt tokens per second: 40.14 tps, New tokens per second: 24.48 tps
-------------
- It's of course not limited to these specific prompts. In general, when the EOS array is listed as:
"eos_token_id": [
32000,
32001,
32007
],
It's not able to easily recall information from the previous conversation. So asking things like "Can you give a summary of our conversation so far", etc. will give poor responses.
Expected behavior
I wouldn't expect the order of listed EOS tokens to change the responses given for a multi-prompt chat sequence.
Additional context
I believe this has to do with how EOS tokens are being handled here and here.
Basically, EOS logits are 'tweaked' in HandleEOSArray to force max to the 'first' EOS token. This will then be the one implicitly appended to the front of the next prompt within AppendTokens(), and in this case sort of messes up the chat template.
Describe the bug
For phi3.4-mini-instruct model (presumably other models as well), the generated responses for a multi-prompt chat sequence appear to be sensitive to the order of EOS tokens listed within the
genai_config.jsonTo Reproduce
Steps to reproduce the behavior:
what color are apples?, and second prompt aswhat about grapes?You'll find that for the second prompt, there is a 'good' response, in that it realizes you are asking what the color of grapes are (based on context around the first prompt / response).
This is what I get:
genai_config.json, swap the order of the EOS tokens, making them:When you run the same test with this configuration, you'll find that the response for the 2nd prompt appears to have lost some of the contextual awareness:
It's not able to easily recall information from the previous conversation. So asking things like "Can you give a summary of our conversation so far", etc. will give poor responses.
Expected behavior
I wouldn't expect the order of listed EOS tokens to change the responses given for a multi-prompt chat sequence.
Additional context
I believe this has to do with how EOS tokens are being handled here and here.
Basically, EOS logits are 'tweaked' in HandleEOSArray to force max to the 'first' EOS token. This will then be the one implicitly appended to the front of the next prompt within
AppendTokens(), and in this case sort of messes up the chat template.