-
-
Notifications
You must be signed in to change notification settings - Fork 529
Tokenizer
Tolga Kayhan edited this page May 11, 2023
·
1 revision
At present, we only offer limited functionality with the GPT-3 tokenizer. However, we plan to introduce more support in the future.
https://platform.openai.com/tokenizer Visit OpenAI website to learn more about tokenizer.
var encodedList = TokenizerGpt3.Encode(input)
var encodedList = TokenizerGpt3.TokenCount("My Sample String");
This will help you clean up end-of-line characters, which will make your token results more accurate.
const string fileName = "TokenizerSample.txt";
var input = await FileExtensions.ReadAllTextAsync($"SampleData/{fileName}");
var encodedList = TokenizerGpt3.Encode(input, true);