Streamlit Browser Issue

[nobelchowdary]

🐛 Describe the bug

I am using the streamlit version of streamlit 1.37.1 and while I run the browser version of torchat for llama2. Getting the following error

2024-08-15 17:04:46.768 Examining the path of torch.classes raised: Tried to instantiate class '__path__._path', but it does not exist! Ensure that it is registered via torch::class_ Warning: checkpoint path ignored because an exported DSO or PTE path specified Warning: checkpoint path ignored because an exported DSO or PTE path specified

2024-08-15 17:07:17.349 Uncaught app exception Traceback (most recent call last):

File "/home/ubuntu/new_repo/venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 85, in exec_func_with_error_handling result = func()

File "/home/ubuntu/new_repo/venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 576, in code_to_exec exec(code, module.__dict__)

File "/home/ubuntu/new_repo/torchchat/torchchat.py", line 78, in <module> browser_main(args)

File "/home/ubuntu/new_repo/torchchat/browser/browser.py", line 62, in main req = CompletionRequest( TypeError: CompletionRequest.__init__() got an unexpected keyword argument 'prompt'

Any idea on it or any one of you faced it before?

Versions

Collecting environment information...
PyTorch version: 2.5.0.dev20240814+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.2.0-1017-aws-aarch64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per socket: 32
Socket(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 2 MiB (32 instances)
L1i cache: 2 MiB (32 instances)
L2 cache: 64 MiB (32 instances)
L3 cache: 36 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.5.0.dev20240814+cpu
[pip3] torchao==0.4.0+git2e7db12
[conda] Could not collect

[nobelchowdary]

The main issues are

2024-08-16 16:35:46.438 Examining the path of torch.classes raised: Tried to instantiate class '__path__._path', but it does not exist! Ensure that it is registered via torch::class_

TypeError: CompletionRequest.__init__() got an unexpected keyword argument 'prompt'

Can you help to solve this @vmpuri and @byjlw

[nobelchowdary]

The api.py file contain the CompletionRequest class. It does not contain the argument prompt which is used in the browser.py. How this issue can be resolved?

Also when tried to solve the prompt issue, there is another issue raised
AttributeError: 'OpenAiApiGenerator' object has no attribute 'completion'

[Jack-Khuu]

@vmpuri Can you take a look at this?

Seems like you have a few PR's in ready in this area. Are those related?
#1034
#1035

[vmpuri]

Just merged both of the PRs

#1034
#1035

@nobelchowdary Try pulling and trying the browser again. I've changed how the browser works - now, the UI queries the server backend.

Run the server in one terminal:

python3 torchchat.py server llama3

And launch the browser from the other:

streamlit run browser/browser.py 

[nobelchowdary]

Hi @vmpuri, when I try to run the current repo - getting this error on the browser

InternalServerError: <!doctype html> <html lang=en> <title>500 Internal Server Error</title> <h1>Internal Server Error</h1> <p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

[nobelchowdary]

Can you try checking the code and running locally. Its because the error is related to the role carrying the content

File "/home/ubuntu/tmp/tenv/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__ return self._next() File "/home/ubuntu/tmp/tenv/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded for item in iterable: File "/home/ubuntu/tmp/torchchat/server.py", line 73, in chunk_processor for chunk in chunked_completion_generator: File "/home/ubuntu/tmp/torchchat/api/api.py", line 269, in chunked_completion tokens = self.chat_formatter.encode_dialog_prompt( File "/home/ubuntu/tmp/torchchat/generate.py", line 70, in encode_dialog_prompt tokens.extend(self.encode_message(message)) File "/home/ubuntu/tmp/torchchat/generate.py", line 59, in encode_message tokens = self.encode_header(message.role) AttributeError: 'dict' object has no attribute 'role'

[SanuVishwakarma]

I am facing the same issue while deploying the text to image generation application on streamlit on cpu system.

Examining the path of torch.classes raised: Tried to instantiate class 'path.path', but it does not exist! Ensure that it is registered via torch::class

If you resolved this issue please provide the solution