Skip to content

Commit a8777ad

Browse files
pudepiedjggerganov
pudepiedj
and
ggerganov
authored
parallel : add option to load external prompt file (#3416)
* Enable external file and add datestamp * Add name of external file at end * Upload ToK2024 * Delete ToK2024.txt * Experiments with jeopardy * Move ParallelQuestions to /proimpts and rename * Interim commit * Interim commit * Final revision * Remove trailing whitespace * remove cmake_all.sh * Remove cmake_all.sh * Changed .gitignore * Improved reporting and new question files. * Corrected typo * More LLM questions * Update LLM-questions.txt * Yet more LLM-questions * Remove jeopardy results file * Reinstate original jeopardy.sh * Update examples/parallel/parallel.cpp --------- Co-authored-by: Georgi Gerganov <[email protected]>
1 parent 97af49f commit a8777ad

File tree

7 files changed

+151
-11
lines changed

7 files changed

+151
-11
lines changed

‎common/common.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,8 @@ bool gpt_params_parse(int argc, char ** argv, gpt_params & params) {
167167
invalid_param = true;
168168
break;
169169
}
170+
// store the external file name in params
171+
params.prompt_file = argv[i];
170172
std::copy(std::istreambuf_iterator<char>(file), std::istreambuf_iterator<char>(), back_inserter(params.prompt));
171173
if (params.prompt.back() == '\n') {
172174
params.prompt.pop_back();

‎common/common.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ struct gpt_params {
7979
std::string model_draft = ""; // draft model for speculative decoding
8080
std::string model_alias = "unknown"; // model alias
8181
std::string prompt = "";
82+
std::string prompt_file = ""; // store the external prompt file name
8283
std::string path_prompt_cache = ""; // path to file for saving/loading prompt eval state
8384
std::string input_prefix = ""; // string to prefix user inputs with
8485
std::string input_suffix = ""; // string to suffix user inputs with

‎examples/jeopardy/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This is pretty much just a straight port of aigoopy/llm-jeopardy/ with an added graph viewer.
44

5-
The jeopardy test can be used to compare the fact knowledge of different models and compare them to eachother. This is in contrast to some other tests, which test logical deduction, creativity, writing skills, etc.
5+
The jeopardy test can be used to compare the fact knowledge of different models and compare them to each other. This is in contrast to some other tests, which test logical deduction, creativity, writing skills, etc.
66

77

88
Step 1: Open jeopardy.sh and modify the following:

‎examples/parallel/parallel.cpp‎

Lines changed: 51 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
#include <cstdio>
1111
#include <string>
1212
#include <vector>
13+
#include <ctime>
1314

1415
// trim whitespace from the beginning and end of a string
1516
static std::string trim(const std::string & str) {
@@ -70,6 +71,26 @@ struct client {
7071
std::vector<llama_token> tokens_prev;
7172
};
7273

74+
static void print_date_time() {
75+
std::time_t current_time = std::time(nullptr);
76+
std::tm* local_time = std::localtime(&current_time);
77+
char buffer[80];
78+
strftime(buffer, sizeof(buffer), "%Y-%m-%d %H:%M:%S", local_time);
79+
80+
printf("\n\033[35mrun parameters as at %s\033[0m\n", buffer);
81+
}
82+
83+
// Define a split string function to ...
84+
static std::vector<std::string> split_string(const std::string& input, char delimiter) {
85+
std::vector<std::string> tokens;
86+
std::istringstream stream(input);
87+
std::string token;
88+
while (std::getline(stream, token, delimiter)) {
89+
tokens.push_back(token);
90+
}
91+
return tokens;
92+
}
93+
7394
int main(int argc, char ** argv) {
7495
srand(1234);
7596

@@ -104,6 +125,23 @@ int main(int argc, char ** argv) {
104125
params.logits_all = true;
105126
std::tie(model, ctx) = llama_init_from_gpt_params(params);
106127

128+
// load the prompts from an external file if there are any
129+
if (params.prompt.empty()) {
130+
printf("\n\033[32mNo new questions so proceed with build-in defaults.\033[0m\n");
131+
} else {
132+
// Output each line of the input params.prompts vector and copy to k_prompts
133+
int index = 0;
134+
printf("\n\033[32mNow printing the external prompt file %s\033[0m\n\n", params.prompt_file.c_str());
135+
136+
std::vector<std::string> prompts = split_string(params.prompt, '\n');
137+
for (const auto& prompt : prompts) {
138+
k_prompts.resize(index + 1);
139+
k_prompts[index] = prompt;
140+
index++;
141+
printf("%3d prompt: %s\n", index, prompt.c_str());
142+
}
143+
}
144+
107145
fprintf(stderr, "\n\n");
108146
fflush(stderr);
109147

@@ -233,7 +271,7 @@ int main(int argc, char ** argv) {
233271
client.n_decoded = 0;
234272
client.i_batch = batch.n_tokens - 1;
235273

236-
LOG_TEE("\033[1mClient %3d, seq %4d, started decoding ...\033[0m\n", client.id, client.seq_id);
274+
LOG_TEE("\033[31mClient %3d, seq %4d, started decoding ...\033[0m\n", client.id, client.seq_id);
237275

238276
g_seq_id += 1;
239277

@@ -336,8 +374,8 @@ int main(int argc, char ** argv) {
336374

337375
const auto t_main_end = ggml_time_us();
338376

339-
LOG_TEE("\033[1mClient %3d, seq %4d, prompt %4d t, response %4d t, time %5.2f s, speed %5.2f t/s, cache miss %d \033[0m \n\nInput: %s\nResponse: %s\n\n",
340-
client.id, client.seq_id, client.n_prompt, client.n_decoded,
377+
LOG_TEE("\033[31mClient %3d, seq %3d/%3d, prompt %4d t, response %4d t, time %5.2f s, speed %5.2f t/s, cache miss %d \033[0m \nInput: %s\n\033[35mResponse: %s\033[0m\n\n",
378+
client.id, client.seq_id, n_seq, client.n_prompt, client.n_decoded,
341379
(t_main_end - client.t_start_prompt) / 1e6,
342380
(double) (client.n_prompt + client.n_decoded) / (t_main_end - client.t_start_prompt) * 1e6,
343381
n_cache_miss,
@@ -357,13 +395,21 @@ int main(int argc, char ** argv) {
357395

358396
const auto t_main_end = ggml_time_us();
359397

360-
LOG_TEE("\n\n");
398+
print_date_time();
399+
400+
LOG_TEE("\n%s: n_parallel = %d, n_sequences = %d, cont_batching = %d, system tokens = %d\n", __func__, n_clients, n_seq, cont_batching, n_tokens_system);
401+
if (params.prompt_file.empty()) {
402+
params.prompt_file = "used built-in defaults";
403+
}
404+
LOG_TEE("External prompt file: \033[32m%s\033[0m\n", params.prompt_file.c_str());
405+
LOG_TEE("Model and path used: \033[32m%s\033[0m\n\n", params.model.c_str());
406+
361407
LOG_TEE("Total prompt tokens: %6d, speed: %5.2f t/s\n", n_total_prompt, (double) (n_total_prompt ) / (t_main_end - t_main_start) * 1e6);
362408
LOG_TEE("Total gen tokens: %6d, speed: %5.2f t/s\n", n_total_gen, (double) (n_total_gen ) / (t_main_end - t_main_start) * 1e6);
363409
LOG_TEE("Total speed (AVG): %6s speed: %5.2f t/s\n", "", (double) (n_total_prompt + n_total_gen) / (t_main_end - t_main_start) * 1e6);
364410
LOG_TEE("Cache misses: %6d\n", n_cache_miss);
365411

366-
LOG_TEE("\n\n");
412+
LOG_TEE("\n");
367413

368414
llama_print_timings(ctx);
369415

‎llama.cpp‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8219,14 +8219,14 @@ void llama_print_timings(struct llama_context * ctx) {
82198219
const llama_timings timings = llama_get_timings(ctx);
82208220

82218221
LLAMA_LOG_INFO("\n");
8222-
LLAMA_LOG_INFO("%s: load time = %8.2f ms\n", __func__, timings.t_load_ms);
8223-
LLAMA_LOG_INFO("%s: sample time = %8.2f ms / %5d runs (%8.2f ms per token, %8.2f tokens per second)\n",
8222+
LLAMA_LOG_INFO("%s: load time = %10.2f ms\n", __func__, timings.t_load_ms);
8223+
LLAMA_LOG_INFO("%s: sample time = %10.2f ms / %5d runs (%8.2f ms per token, %8.2f tokens per second)\n",
82248224
__func__, timings.t_sample_ms, timings.n_sample, timings.t_sample_ms / timings.n_sample, 1e3 / timings.t_sample_ms * timings.n_sample);
8225-
LLAMA_LOG_INFO("%s: prompt eval time = %8.2f ms / %5d tokens (%8.2f ms per token, %8.2f tokens per second)\n",
8225+
LLAMA_LOG_INFO("%s: prompt eval time = %10.2f ms / %5d tokens (%8.2f ms per token, %8.2f tokens per second)\n",
82268226
__func__, timings.t_p_eval_ms, timings.n_p_eval, timings.t_p_eval_ms / timings.n_p_eval, 1e3 / timings.t_p_eval_ms * timings.n_p_eval);
8227-
LLAMA_LOG_INFO("%s: eval time = %8.2f ms / %5d runs (%8.2f ms per token, %8.2f tokens per second)\n",
8227+
LLAMA_LOG_INFO("%s: eval time = %10.2f ms / %5d runs (%8.2f ms per token, %8.2f tokens per second)\n",
82288228
__func__, timings.t_eval_ms, timings.n_eval, timings.t_eval_ms / timings.n_eval, 1e3 / timings.t_eval_ms * timings.n_eval);
8229-
LLAMA_LOG_INFO("%s: total time = %8.2f ms\n", __func__, (timings.t_end_ms - timings.t_start_ms));
8229+
LLAMA_LOG_INFO("%s: total time = %10.2f ms\n", __func__, (timings.t_end_ms - timings.t_start_ms));
82308230
}
82318231

82328232
void llama_reset_timings(struct llama_context * ctx) {

‎prompts/LLM-questions.txt‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
In the context of LLMs, what is "Attention"?
2+
In the context of LLMs, what is a completion?
3+
In the context of LLMs, what is a prompt?
4+
In the context of LLMs, what is GELU?
5+
In the context of LLMs, what is RELU?
6+
In the context of LLMs, what is softmax?
7+
In the context of LLMs, what is decoding?
8+
In the context of LLMs, what is encoding?
9+
In the context of LLMs, what is tokenizing?
10+
In the context of LLMs, what is an embedding?
11+
In the context of LLMs, what is quantization?
12+
In the context of LLMs, what is a tensor?
13+
In the context of LLMs, what is a sparse tensor?
14+
In the context of LLMs, what is a vector?
15+
In the context of LLMs, how is attention implemented?
16+
In the context of LLMs, why is attention all you need?
17+
In the context of LLMs, what is "RoPe" and what is it used for?
18+
In the context of LLMs, what is "LoRA" and what is it used for?
19+
In the context of LLMs, what are weights?
20+
In the context of LLMs, what are biases?
21+
In the context of LLMs, what are checkpoints?
22+
In the context of LLMs, what is "perplexity"?
23+
In the context of LLMs, what are models?
24+
In the context of machine-learning, what is "catastrophic forgetting"?
25+
In the context of machine-learning, what is "elastic weight consolidation (EWC)"?
26+
In the context of neural nets, what is a hidden layer?
27+
In the context of neural nets, what is a convolution?
28+
In the context of neural nets, what is dropout?
29+
In the context of neural nets, what is cross-entropy?
30+
In the context of neural nets, what is over-fitting?
31+
In the context of neural nets, what is under-fitting?
32+
What is the difference between an interpreted computer language and a compiled computer language?
33+
In the context of software development, what is a debugger?
34+
When processing using a GPU, what is off-loading?
35+
When processing using a GPU, what is a batch?
36+
When processing using a GPU, what is a block?
37+
When processing using a GPU, what is the difference between a batch and a block?
38+
When processing using a GPU, what is a scratch tensor?
39+
When processing using a GPU, what is a layer?
40+
When processing using a GPU, what is a cache?
41+
When processing using a GPU, what is unified memory?
42+
When processing using a GPU, what is VRAM?
43+
When processing using a GPU, what is a kernel?
44+
When processing using a GPU, what is "metal"?
45+
In the context of LLMs, what are "Zero-Shot", "One-Shot" and "Few-Shot" learning models?
46+
In the context of LLMs, what is the "Transformer-model" architecture?
47+
In the context of LLMs, what is "Multi-Head Attention"?
48+
In the context of LLMs, what is "Self-Attention"?
49+
In the context of transformer-model architectures, how do attention mechanisms use masks?

‎prompts/parallel-questions.txt‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
What do you know about Hobbits?
2+
What is quantum field theory?
3+
Why did the chicken cross the road?
4+
Who is the president of the United States?
5+
How do I run CMake on MacOS?
6+
Do you agree that C++ is a really finicky language compared with Python3?
7+
Is it a good idea to invest in technology?
8+
Do you like Wagner's Ring?
9+
Do you think this file input option is really neat?
10+
What should we all do about climate change?
11+
Is time-travel possible within the laws of current physics?
12+
Is it like anything to be a bat?
13+
Once the chicken has crossed the road, does it try to go back?
14+
Who is the greatest of all musical composers?
15+
What is art?
16+
Is there life elsewhere in the universe?
17+
What is intelligence?
18+
What is the difference between knowledge and intelligence?
19+
Will religion ever die?
20+
Do we understand ourselves?
21+
What is the best way to cook eggs?
22+
If you cannot see things, on what basis do you evaluate them?
23+
Explain the role of the np junction in photovoltaic cells?
24+
Is professional sport a good or bad influence on human behaviour?
25+
Is capital punishment immoral?
26+
Should we care about other people?
27+
Who are you?
28+
Which sense would you surrender if you could?
29+
Was Henry Ford a hero or a villain?
30+
Do we need leaders?
31+
What is nucleosynthesis?
32+
Who is the greatest scientist of all time?
33+
Who first observed what came to be known as the photovoltaic effect?
34+
What is nuclear fusion and why does it release energy?
35+
Can you know that you exist?
36+
What is an exoplanet?
37+
Do you like cream?
38+
What is the difference?
39+
Can I know that I exist while I'm dreaming that I'm Descartes?
40+
Who said "I didn't know I thought that until I heard myself saying it"?
41+
Does anything really matter?
42+
Can you explain the unreasonable effectiveness of mathematics?

0 commit comments

Comments
 (0)