Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After upgrade rasa from v2.1.3 to v2.2.1, local rasa x server doesn't work #7592

Closed
LoveMeWithoutAll opened this issue Dec 18, 2020 · 21 comments · Fixed by #7612
Closed

After upgrade rasa from v2.1.3 to v2.2.1, local rasa x server doesn't work #7592

LoveMeWithoutAll opened this issue Dec 18, 2020 · 21 comments · Fixed by #7612
Labels
area:rasa-x/backend 🎩 All issues focused on the Rasa X backend type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. type:dependencies Pull requests that update a dependency file

Comments

@LoveMeWithoutAll
Copy link

Rasa version: 2.2.1

Rasa SDK version (if used & relevant): 2.2.0

Rasa X version (if used & relevant): 0.34.0

Python version: 3.8.6

Operating system (windows, osx, ...): macOS Big Sur version 11.1

Issue:
After upgrade rasa from v2.1.3 to v2.2.1, local rasa x server doesn't work.
Only after I edited rasa cli source code, it did work.

# rasa/cli/x.py
# line 493 ~ 499
local.main(
            args,
            project_path,
            args.data,
            token=rasa_x_token,
            config_path=config_path
            # domain_path=domain_path # I removed it!
        )

Error (including full traceback):

(venv) (base) ➜  rasa git:(feature/prototype) ✗ rasa x
Traceback (most recent call last):
  File "/Users/ys/dev/rasa/venv/lib/python3.8/site-packages/rasa/cli/x.py", line 493, in run_locally
    local.main(
TypeError: main() got an unexpected keyword argument 'domain_path'

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.
(venv) (base) ➜  rasa git:(feature/prototype) ✗ rasa x --v
Traceback (most recent call last):
  File "/Users/ys/dev/rasa/venv/lib/python3.8/site-packages/rasa/cli/x.py", line 493, in run_locally
    local.main(
TypeError: main() got an unexpected keyword argument 'domain_path'

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.

Command or request that led to error:

rasa x
@LoveMeWithoutAll LoveMeWithoutAll added area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. labels Dec 18, 2020
@humcasma
Copy link

I can confirm both the issue and the workaround.

@sara-tagger
Copy link
Collaborator

Thanks for raising this issue, @Tawakalt will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

@degiz
Copy link
Contributor

degiz commented Dec 18, 2020

Hey @humcasma @LoveMeWithoutAll

Thanks for checking the latest 2.2.1! 🚀

The issue with rasa x is addressed in the upcoming release of rasa x 0.35, should happen very soon.
Then you'll have to pull the latest package and things will work!

@amn41
Copy link
Contributor

amn41 commented Dec 21, 2020

just a quick update here - we had hoped to get 0.35 out last week but it wasn't possible. It will be released in early January instead. Until then, there isn't an officially supported Rasa X version for Rasa Open Source 2.2.x, so I would recommend you stick with 2.1 or if it suits you use the workaround above. Thanks for raising the issue and finding a workaround!

@andyfangaf
Copy link

andyfangaf commented Dec 21, 2020

Spent my morning debugging this and it was driving me crazy. I can confirm that downgrading to Rasa 2.1 fixes Rasa X.

Is there a way to ensure the version incompatibility between Rasa and Rasa X doesn't happen again in the future? I believe this happened with the previous version as well, and as a newcomer, it was very frustrating to debug.

@UsamaNadeemKhan
Copy link

Spent my morning debugging this and it was driving me crazy. I can confirm that downgrading to Rasa 2.1 fixes Rasa X.

Is there a way to ensure the version incompatibility between Rasa and Rasa X doesn't happen again in the future? I believe this happened with the previous version as well, and as a newcomer, it was very frustrating to debug.

Can you share rasa-x , rasa-sdk and python versions too.
I'm having the same issue from yesterday

@BShaq
Copy link

BShaq commented Dec 22, 2020

Hi all,
How can you downgrade rasa to version 2.1?

@amn41
Copy link
Contributor

amn41 commented Dec 22, 2020

can you please try UPgrading to rasa open source 2.2.2? that should fix the issue.

https://github.com/RasaHQ/rasa/releases/tag/2.2.2

@BShaq
Copy link

BShaq commented Dec 22, 2020

Thanks, that has indeed solved the issue!

@HamzaNKhan
Copy link

Need some help here. Stuck here for past 2-3 days
version

error

@pratz-sridhar
Copy link

pratz-sridhar commented Dec 23, 2020

Need some help here. Stuck here for past 2-3 days
version

error

Did you change the x.py file in the cli folder as was shown above? You can either do that at your own risk, or wait for the next update and I expect atleast the first TypeError will be handled.

@HamzaNKhan
Copy link

Need some help here. Stuck here for past 2-3 days
version
error

Did you change the x.py file in the cli folder as was shown above? You can either do that at your own risk, or wait for the next update and I expect atleast the first TypeError will be handled.

I upgraded rasa to 2.2.2 and now getting error shown in picture

@giladpn
Copy link

giladpn commented Dec 30, 2020

I can confirm that the above problem does indeed continue to occur, and in fact it is two different problems. I can also confirm that I am working in a virtual environment and that I reinstalled the entire venv from scratch to check this issue.

To sum it up the two issues are

  1. the domain_path issue that started this thread
  2. an issue with an invalid handle that occurs even if the domain_path issue is dealt with correctly. This handle issue seems to be special to the Windows installation of rasa-x and does not occur on Ubuntu 18.04.

The handle issue occurs even if we comment out the code as suggested above. In that case (of commenting out the line of code) the traceback changes slightly.

Furthermore (not shown below to avoid confusion) even if I downgrade rasa to 2.1.3 while staying with rasa X 0.34 the handle issue continues to come up. So there seems right now to be no working combination of rasa x on Windows 10 (with either python 3.8.5 or 3.7.9). However switching to Ubuntu 18.04 with python 3.6.9 does work.

Please help.

Rasa Version : 2.2.2
Rasa SDK Version : 2.2.0
Rasa X Version : 0.34.0
Python Version : 3.8.5
Operating System : Windows-10-10.0.19041-SP0
Python Path : c:\workspace\py-pip-workspaces\rasa2-examples.venv\scripts\python.exe

So, here first is the traceback without any changes to the code

Traceback (most recent call last):
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasa\cli\x.py", line 494, in run_locally
    local.main(
TypeError: main() got an unexpected keyword argument 'domain_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasa\cli\x.py", line 506, in run_locally
    local.main(
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\local.py", line 233, in main
    _start_event_service()
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\local.py", line 138, in _start_event_service 
    common_utils.run_in_process(
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\utils\common.py", line 855, in run_in_process
    p.start()
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 959, in reduce_pipe_connection
    dh = reduction.DupHandle(conn.fileno(), access)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 170, in fileno
    self._check_closed()
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\spawn.py", line 107, in spawn_main  
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid

Here is the change I made to the code (as above)
In line 500 of file c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasa\cli\x.py:

    # noinspection PyBroadException
    try:
        try:
            local.main(
                args,
                project_path,
                args.data,
                token=rasa_x_token,
                config_path=config_path,
                # domain_path=domain_path,  # <----------------
            )

and here is the traceback after that one-line change

Traceback (most recent call last):
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasa\cli\x.py", line 494, in run_locally
    local.main(
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\local.py", line 233, in main
    _start_event_service()
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\local.py", line 138, in _start_event_service 
    common_utils.run_in_process(
  File "c:\workspace\py-pip-workspaces\rasa2-examples\.venv\lib\site-packages\rasax\community\utils\common.py", line 855, in run_in_process
    p.start()
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 959, in reduce_pipe_connection
    dh = reduction.DupHandle(conn.fileno(), access)
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 170, in fileno
    self._check_closed()
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\spawn.py", line 107, in spawn_main  
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\Gilad\miniconda3\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid

To sum it up there seems to be an issue with an invalid handle that occurs even if the domain_path issue is dealt with correctly.

@diegodorgam
Copy link

diegodorgam commented Jan 3, 2021

Tried to do the opposite, instead of removing the domain_path kwarg from the x.py L501, I added the kwarg in the file local.py L181, like this:

def main(
    args: argparse.Namespace,
    project_path: Text,
    data_path: Text,
    token: Optional[Text] = None,
    config_path: Optional[Text] = None,
    domain_path: Optional[Text] = None, # <<--- added
) -> None:

still rasa x can't start as a process. Not sure why:

Starting Rasa X in local mode... 🚀
Traceback (most recent call last):
  File ".../python3.6/site-packages/rasa/cli/x.py", line 500, in run_locally
    domain_path=domain_path,
  File ".../python3.6/site-packages/rasax/community/local.py", line 234, in main
    _start_event_service()
  File ".../python3.6/site-packages/rasax/community/local.py", line 139, in _start_event_service
    fn=_event_service, args=(LOCAL_ENDPOINTS_PATH,), daemon=True
  File ".../python3.6/site-packages/rasax/community/utils/common.py", line 855, in run_in_process
    p.start()
  File ".../python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File ".../python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File ".../python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File ".../python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File ".../python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File ".../python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File ".../python3.6/multiprocessing/connection.py", line 948, in reduce_connection
    df = reduction.DupFd(conn.fileno())
  File ".../python3.6/multiprocessing/connection.py", line 170, in fileno
    self._check_closed()
  File ".../python3.6/multiprocessing/connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

my numbers btw:

Rasa Version     : 2.2.2
Rasa SDK Version : 2.2.0
Rasa X Version   : 0.34.0
Python Version   : 3.6.12
Operating System : Darwin-19.4.0-x86_64-i386-64bit
Python Path      : ../opt/anaconda3/envs/rasa_big5_p36/bin/python

@alwx alwx added area:rasa-x/backend 🎩 All issues focused on the Rasa X backend and removed area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Jan 28, 2021
@alwx
Copy link
Contributor

alwx commented Jan 28, 2021

@degiz can this one be closed?

@alwx alwx closed this as completed Jan 28, 2021
@alwx alwx reopened this Jan 28, 2021
@alwx alwx added the type:dependencies Pull requests that update a dependency file label Jan 28, 2021
@zfallahnejad
Copy link

zfallahnejad commented Feb 8, 2021

Rasa version: 2.2.8
Rasa SDK version: 2.2.0
Rasa X version: 0.35.1
Python version: 3.6.7
Operating system: windows 10

I am facing the same issue with rasa x.

2021-02-08 19:29:47.132357: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-02-08 19:29:47.132555: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Starting Rasa X in local mode... �
Traceback (most recent call last):
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\site-packages\rasa\cli\x.py", line 500, in run_locally
    domain_path=domain_path,
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\site-packages\rasax\community\local.py", line 243, in main
    _start_event_service()
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\site-packages\rasax\community\local.py", line 139, in _start_event_service
    fn=_event_service, args=(LOCAL_ENDPOINTS_PATH,), daemon=True
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\site-packages\rasax\community\utils\common.py", line 855, in run_in_process
    p.start()
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\connection.py", line 939, in reduce_pipe_connection
    dh = reduction.DupHandle(conn.fileno(), access)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\connection.py", line 170, in fileno
    self._check_closed()
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "c:\users\zohreh\anaconda3\envs\rasax_228\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

It is worth noting that there is no issue with rasa shell, so I think I have defined the input files correctly.

@bfackland
Copy link

@zfallahnejad If you still have the OSError: handle is closed issue, there's a comment in the Rasa forums from the team:

This is a bug with the way we disable our telemetry. If you run rasa telemetry enable before running rasa x, it should work. Disabling telemetry using rasa telemetry disable appears to cause this error. We’ll work on a fix so you can run Rasa X without telemetry!

https://forum.rasa.com/t/getting-following-error-when-type-rasa-x-in-cmd-need-help-here/38253/8

@zfallahnejad
Copy link

@bfackland Thanks, running rasa telemetry enable before running rasa x solved the issue for me.

@m-vdb
Copy link
Collaborator

m-vdb commented Apr 30, 2021

Closing as the original issue has been fixed with rasa=2.2.2. And the telemetry issue as been fixed in 0.37.2

@m-vdb m-vdb closed this as completed Apr 30, 2021
@rashmimetri
Copy link

rashmimetri commented Jul 16, 2021

Rasa Version: 2.8.0
Minimum Compatible Version: 2.8.0
Rasa SDK Version: 2.8.0
Rasa X Version: 0.41.2
Python Version: 3.7.6
Operating System: Windows 10

Encountering the same error with the newer version. Any help is appreciated!

2021-07-16 17:55:24 WARNING  rasax.community.services.insights.insight_service  - No local evaluation result files found.
Traceback (most recent call last):
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasa\cli\x.py", line 502, in run_locally
    domain_path=domain_path,
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\local.py", line 257, in main
    common_utils.run_in_loop(model_service.discover_models())
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\utils\common.py", line 900, in run_in_loop
    return loop.run_until_complete(coro)
  File "C:\Users\dd\anaconda3\lib\asyncio\base_events.py", line 583, in run_until_complete
    return future.result()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 821, in discover_models
    await model_service.discover_models()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 147, in discover_models
    await _run_loop(max_retries, sleep_in_seconds)
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 133, in _run_loop
    await self._discover_models()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 185, in _discover_models
    minimum_version = await self._retry_fetching_minimum_compatible_version()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 174, in _retry_fetching_minimum_compatible_version
    minimum_version = await self.minimum_compatible_version()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\model_service.py", line 160, in minimum_compatible_version
    info = await stack_service.version()
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\rasax\community\services\stack_service.py", line 95, in version
    timeout=timeout_in_seconds,
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\aiohttp\client.py", line 582, in _request
    break
  File "c:\users\dd\desktop\rasa\venv\lib\site-packages\aiohttp\helpers.py", line 596, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

Sorry, something went wrong (see error above). Make sure to start Rasa X with valid data and valid domain and config files. Please, also check any warnings that popped up.
If you need help fixing the issue visit our forum: https://forum.rasa.com/.

@m-vdb
Copy link
Collaborator

m-vdb commented Jul 16, 2021

@rashmimetri it doesn't seem to be the same error. Could you open a new Github issue with the information you pasted in your comment? Also, have you taken a look at the forum for possible similar issues?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-x/backend 🎩 All issues focused on the Rasa X backend type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. type:dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging a pull request may close this issue.