Cannot detect the model loader

[netsho]

Hi,

I have a running text-generation-webui version 3.7.1 with llama.cpp as the default loader. Here is my CMD_FLAGS.txt

--loader llama.cpp --ctx-size 4096
--listen  --extra-flags metrics
--extensions context-progress-bar

The extra-flags metrics is added correctly, as i can see it in the "extra-flags" field of the "Model" tab.
The llama.cpp is correctly loaded at the startup

11:53:25-552471 INFO     LOADER: "llama.cpp"

The context progress bar is shown on the UI, but it doesn't display anything.
The problem is that it doesn't detect the loader

11:53:31-025352 WARNING  context-progress-bar: 'model_loader' has unexpected value: None

Do you have any idea how to resolve it ?

Thank you

[mamei16]

Hi!

Due to a bug in gradio, I have to use a hacky workaround to trigger an update of the model_loader. The update trigger is clicking the "Chat" button in the top left. So try the following:

1. Ensure that the model is fully loaded.
2. Click the Chat button (even if you're already on the Chat tab).

Also, the progress bar won't update until you either send a new message, continue the last model message or regenerate the last model message.

[netsho]

I followed your recommandations:

• The model is fully loaded - Checked in terminal and in "Model" tab.
• Clicked "Chat" button while being in the chat. Switched views between "Chat" and other tabs.
• Sent a new message
• Continued the last model message
• Regenerated the last model message
• Started a new conversation
  None of those solutions worked for me, but i did get an error on the terminal:

22:44:21-484289 WARNING  context-progress-bar: 'model_loader' has unexpected value: None
prompt processing progress, n_past = 458, n_tokens = 202, progress = 1.000000
Traceback (most recent call last):
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 541, in process_events
    response = await route_utils.call_process_api(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 276, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1928, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1514, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 833, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/extensions/context-progress-bar/script.py", line 76, in get_current_context_percentage
    num_context_tokens = int(kv_cache_tokens_match.group(1))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
prompt eval time =    3461.61 ms /   458 tokens (    7.56 ms per token,   132.31 tokens per second)
       eval time =    3631.93 ms /    27 tokens (  134.52 ms per token,     7.43 tokens per second)
      total time =    7093.55 ms /   485 tokens
22:44:35-632061 INFO     Output generated in 7.10 seconds (3.95 tokens/s, 28 tokens, context 458, seed 457735148)

This error message is printed for like 5 times (on 1-2 prompts). After that, i don't get this error message when prompting the model, which is odd. After not getting the error message, the context bar still doesn't display anything.
I have waited for like 5 minutes to make sure that time isn't the problem. Unfortunately, nothing is displayed.

[mamei16]

So the metrics endpoint somehow doesn't return the expected metrics. Could you open the file "/opt/text-generation-webui/extensions/context-progress-bar/script.py" and add a line to print the response text like so:

def get_current_context_percentage():
             raise ValueError("Please activate llama-server metrics to use the context-progress-bar extension.")
             num_context_tokens = 0
         else:
+            print(f"Metric endpoint response:\n{response.text}\n-------------")
             kv_cache_tokens_match = kv_cache_tokens_pat.search(response.text)
             num_context_tokens = int(kv_cache_tokens_match.group(1))
     else:

- and then paste the printed text from the terminal here? Maybe use this prompt so there not too much output:

Hi, please respond with just the letter "A".

[netsho]

Added the line to script.py. I clicked on "Chat" tab, created a new chat and used the prompt:
Hi, please respond with just the letter "A".

Output from the terminal:

22:49:31-835002 INFO     LOADER: "llama.cpp"
22:49:31-835684 INFO     TRUNCATION LENGTH: 4096
22:49:31-836385 INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"
22:49:31-837114 INFO     Loading the extension "twinbook"
22:49:31-856259 INFO     Loading the extension "context-progress-bar"
Twinbook: No last state saved

Running on local URL:  http://0.0.0.0:7860

22:49:39-813475 WARNING  context-progress-bar: 'model_loader' has unexpected value: None
Metric endpoint response:
# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 0
# HELP llamacpp:prompt_seconds_total Prompt process time
# TYPE llamacpp:prompt_seconds_total counter
llamacpp:prompt_seconds_total 0
# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total 0
# HELP llamacpp:tokens_predicted_seconds_total Predict process time
# TYPE llamacpp:tokens_predicted_seconds_total counter
llamacpp:tokens_predicted_seconds_total 0
# HELP llamacpp:n_decode_total Total number of llama_decode() calls
# TYPE llamacpp:n_decode_total counter
llamacpp:n_decode_total 0
# HELP llamacpp:n_busy_slots_per_decode Average number of busy slots per llama_decode() call
# TYPE llamacpp:n_busy_slots_per_decode counter
llamacpp:n_busy_slots_per_decode 0
# HELP llamacpp:prompt_tokens_seconds Average prompt throughput in tokens/s.
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds 0
# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds 0
# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing 0
# HELP llamacpp:requests_deferred Number of requests deferred.
# TYPE llamacpp:requests_deferred gauge
llamacpp:requests_deferred 0

-------------
Traceback (most recent call last):
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 541, in process_events
    response = await route_utils.call_process_api(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 276, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1928, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1514, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 833, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/opt/text-generation-webui/extensions/context-progress-bar/script.py", line 77, in get_current_context_percentage
    num_context_tokens = int(kv_cache_tokens_match.group(1))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
prompt processing progress, n_past = 117, n_tokens = 117, progress = 1.000000
prompt eval time =    1002.34 ms /   117 tokens (    8.57 ms per token,   116.73 tokens per second)
       eval time =     139.14 ms /     2 tokens (   69.57 ms per token,    14.37 tokens per second)
      total time =    1141.47 ms /   119 tokens
22:49:59-801058 INFO     Output generated in 1.14 seconds (2.62 tokens/s, 3 tokens, context 117, seed 1578123970)
Metric endpoint response:
# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 117
# HELP llamacpp:prompt_seconds_total Prompt process time
# TYPE llamacpp:prompt_seconds_total counter
llamacpp:prompt_seconds_total 1.002
# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total 2
# HELP llamacpp:tokens_predicted_seconds_total Predict process time
# TYPE llamacpp:tokens_predicted_seconds_total counter
llamacpp:tokens_predicted_seconds_total 0.139
# HELP llamacpp:n_decode_total Total number of llama_decode() calls
# TYPE llamacpp:n_decode_total counter
llamacpp:n_decode_total 2
# HELP llamacpp:n_busy_slots_per_decode Average number of busy slots per llama_decode() call
# TYPE llamacpp:n_busy_slots_per_decode counter
llamacpp:n_busy_slots_per_decode 1
# HELP llamacpp:prompt_tokens_seconds Average prompt throughput in tokens/s.
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds 116.766
# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds 14.3885
# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing 0
# HELP llamacpp:requests_deferred Number of requests deferred.
# TYPE llamacpp:requests_deferred gauge
llamacpp:requests_deferred 0

The first metrics are empty. After that, after each prompt, i get the error message, and metrics showing correct values (i guess).

[mamei16]

I figured it out: The kv_cache_tokens metric that this extension was using has recently been removed: ggml-org/llama.cpp#13660.

Looks like I may be able to make up for it by using prompt_tokens_total + n_decode_total though. I'll report back once I've taken a closer look.

[mamei16]

It's a shame, but kv_cache_tokens can't be replaced by other metrics, so the new llama.cpp loader is only supported for older versions of text-generation-webui (<= v3.4.1) or newer versions if and older version of llama-cpp-binaries (<= v0.14.0) is installed. I'll update the readme to reflect this.