on macOS, threebot server won't start without sudo #3116

sameh-farouk · 2021-05-25T15:33:28Z

Description

on macOS, threebot server won't start without sudo

This comment has useful debug/trace info also #3116 (comment)

Version information

OS: macOS 10.15.7
Project commit/version: js-sdk 11.0b4

Installation method

as documented on our wiki

Steps to reproduce

1- activate the env

poetry shell

2- start the server with

threebot start --local --no-cert

Traceback/Logs/Alerts

(js-sdk-DbrXJgm3-py3.9) sameh@samehs-mbp js-sdk % threebot start --local --no-cert                   
2021-05-25 17:03:40.799 | WARNING  | jumpscale.entry_points.threebot:create_wallets_if_not_exists:273 - skipping wallets creation, identity isn't configured yet
2021-05-25 17:03:40.936 | INFO     | jumpscale.entry_points.threebot:check_for_bins:34 - ✅ binaries nginx,redis-server,tmux,git required are installed.
2021-05-25 17:03:40.956 | DEBUG    | jumpscale.entry_points.threebot:test_privileged_ports_bind:58 - nginx: the configuration file /var/folders/d0/4n__wn4d6bv2rkd408ckz6s00000gn/T/tmprs2rpv6e syntax is ok
2021-05-25 17:03:40.957 | DEBUG    | jumpscale.entry_points.threebot:test_privileged_ports_bind:58 - nginx: configuration file /var/folders/d0/4n__wn4d6bv2rkd408ckz6s00000gn/T/tmprs2rpv6e test is successful
2021-05-25 17:03:41.036 | INFO     | jumpscale.core.logging.logging:register:152 - Logging from 'jumpscale.servers.gedis' is now bound to 'gedis' app
2021-05-25 17:03:41.075 | WARNING  | jumpscale.entry_points.threebot:create_wallets_if_not_exists:273 - skipping wallets creation, identity isn't configured yet
2021-05-25 17:03:41.148 | DEBUG    | jumpscale.sals.process:get_pids:476 - ignoring : 639 zombie process.
2021-05-25 17:03:41.358 | DEBUG    | jumpscale.sals.process:get_pids:476 - ignoring : 639 zombie process.
......
2021-05-25 17:03:54.438 | ERROR    | jumpscale.tools.errorhandler.errorhandler:_handle_exception:93 - jumpscale.core.exceptions.exceptions.Timeout: Wait operation exceeded timeout: 10
Traceback (most recent call last):
> File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/bin/threebot", line 5, in <module>
    cli()
    └ <Group cli>
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ ()
           │    └ <function BaseCommand.main at 0x10b61ae50>
           └ <Group cli>
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         │    │      └ <click.core.Context object at 0x10a67e460>
         │    └ <function MultiCommand.invoke at 0x10b61bc10>
         └ <Group cli>
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
           │               │       │       │      └ <click.core.Context object at 0x10a816af0>
           │               │       │       └ <function Command.invoke at 0x10b61b820>
           │               │       └ <Command start>
           │               └ <click.core.Context object at 0x10a816af0>
           └ <function MultiCommand.invoke.<locals>._process_result at 0x10a6f1820>
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           │   │      │    │           │   └ {'local': True, 'cert': False, 'identity': None, 'domain': None, 'email': None, 'development': False, 'background': False}
           │   │      │    │           └ <click.core.Context object at 0x10a816af0>
           │   │      │    └ <function start at 0x10d857280>
           │   │      └ <Command start>
           │   └ <function Context.invoke at 0x10b61a940>
           └ <click.core.Context object at 0x10a816af0>
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           │         │       └ {'local': True, 'cert': False, 'identity': None, 'domain': None, 'email': None, 'development': False, 'background': False}
           │         └ ()
           └ <function start at 0x10d857280>

  File "/Users/sameh/Documents/code/js-sdk/jumpscale/entry_points/threebot.py", line 159, in start
    j.servers.threebot.start_default(wait=True, local=local, domain=domain, email=email, cert=cert)
    │ │                                         │     │      │      │       │     │      │    └ False
    │ │                                         │     │      │      │       │     │      └ False
    │ │                                         │     │      │      │       │     └ None
    │ │                                         │     │      │      │       └ None
    │ │                                         │     │      │      └ None
    │ │                                         │     │      └ None
    │ │                                         │     └ True
    │ │                                         └ True
    │ └ <property object at 0x10b61edb0>
    └ <jumpscale.loader.J object at 0x10b5bdf70>

  File "/Users/sameh/Documents/code/js-sdk/jumpscale/servers/threebot/__init__.py", line 34, in start_default
    server.start(wait=wait, cert=cert)
    │      │     │    │     │    └ False
    │      │     │    │     └ False
    │      │     │    └ True
    │      │     └ True
    │      └ <function ThreebotServer.start at 0x10dcb59d0>
    └ ThreebotServer(
        instance_name='default',
        _package_manager=StoredFactory(PackageManager),
        domain=None,
        email=None,
        a...

  File "/Users/sameh/Documents/code/js-sdk/jumpscale/servers/threebot/threebot.py", line 850, in start
    self.nginx.start()
    │    └ <property object at 0x10dca3770>
    └ ThreebotServer(
        instance_name='default',
        _package_manager=StoredFactory(PackageManager),
        domain=None,
        email=None,
        a...

  File "/Users/sameh/Documents/code/js-sdk/jumpscale/tools/nginx/nginxserver.py", line 40, in start
    cmd.start()
    │   └ <function StartupCmd.start at 0x10dcf8160>
    └ StartupCmd(
        instance_name='nginx_main',
        start_cmd='nginx -c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf',
        ports=[],
      ...

  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/jumpscale/tools/startupcmd/startupcmd.py", line 314, in start
    self.wait_for_running(die=True, timeout=self.timeout)
    │    │                                  │    └ <property object at 0x10dcf5590>
    │    │                                  └ StartupCmd(
    │    │                                      instance_name='nginx_main',
    │    │                                      start_cmd='nginx -c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf',
    │    │                                      ports=[],
    │    │                                    ...
    │    └ <function StartupCmd.wait_for_running at 0x10dcf80d0>
    └ StartupCmd(
        instance_name='nginx_main',
        start_cmd='nginx -c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf',
        ports=[],
      ...
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/jumpscale/tools/startupcmd/startupcmd.py", line 272, in wait_for_running
    self._wait(True, die, timeout)
    │    │           │    └ 10
    │    │           └ True
    │    └ <function StartupCmd._wait at 0x10dcebf70>
    └ StartupCmd(
        instance_name='nginx_main',
        start_cmd='nginx -c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf',
        ports=[],
      ...
  File "/Users/sameh/Library/Caches/pypoetry/virtualenvs/js-sdk-DbrXJgm3-py3.9/lib/python3.9/site-packages/jumpscale/tools/startupcmd/startupcmd.py", line 248, in _wait
    raise j.exceptions.Timeout(f"Wait operation exceeded timeout: {timeout}")
          │ └ <property object at 0x10b61eae0>
          └ <jumpscale.loader.J object at 0x10b5bdf70>

The text was updated successfully, but these errors were encountered:

sameh-farouk · 2021-05-26T09:57:29Z

@xmonader @abom @OmarElawady after we start the server if we check the Nginx master process, using ps (note the command line that starts the process):

in linux

$ ps aux | egrep '[n]ginx.*-c'
sameh    3526378  0.0  0.0  11648  9280 pts/5    S+   11:13   0:00 nginx: master process startupcmd_nginx_main -c /home/sameh/sandbox/cfg/nginx/main/nginx.conf

in macOS and the threebot server running with `sudo`

$ ps aux | egrep '[n]ginx.*-c'
root             13168   0.0  0.0  4303920      8 s011  S+    5:32PM   0:00.08 nginx: master process startupcmd_nginx_main -c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf

in macOS and the threebot server running without `sudo` we hit the issue and the output of `ps` is

$ ps aux | egrep '[n]ginx.*-c'
sameh            14550   0.0  0.0  4285232   4168 s006  S+   11:41AM   0:00.02 nginx: master process startupcmd_nginx_main -c /Users/sameh/sandbox/cfg/nginx/main/ng  XPC_FLAGS=0x0

note that the command that started this process becomes truncated, this is where we hit the issue, we use this regex, in jumpscale/tools/nginx/nginxserver.py to match the running Nginx master:

js-sdk/jumpscale/tools/nginx/nginxserver.py

Line 15 in 8d6b956

    
           return r"nginx.* \-c {CONFIG_PATH}".format(CONFIG_PATH=j.sals.fs.expanduser(self.config_path))

so in mac, when our startupcmd try to make sure the Nginx started successfully using the above regex, he uses the process sal to match
rnginx.* \-c /Users/sameh/sandbox/cfg/nginx/main/nginx.conf with nginx: master process startupcmd_nginx_main -c /Users/sameh/sandbox/cfg/nginx/main/ng XPC_FLAGS=0x0 which doesn't match because the truncated cmdline.

i was able to start the server without sudo with this change below to regex. (but in general, I don't like using a magic number in the code or fixing the symptoms instead of the root caused).

        return r"nginx.* \-c {CONFIG_PATH:.38}".format(CONFIG_PATH=j.sals.fs.expanduser(self.config_path))

which will now match correctly rnginx.* \-c /Users/sameh/sandbox/cfg/nginx/main/ng with nginx: master process startupcmd_nginx_main -c /Users/sameh/sandbox/cfg/nginx/main/ng XPC_FLAGS=0x0, and it works.

any thoughts?

sameh-farouk · 2021-05-26T10:16:23Z

also, instead of using the regex and match with the process cmd line, I think it's better to use the PID file specified in our nginx.conf file to check if the Nginx started successfully or no.

OmarElawady · 2021-05-27T10:18:50Z

Checking the process pid looks like a cleaner solution for nginx. There was a similar issue on Solaris that was fixed by peaking on the process memory, would be good to know if there's a similar workaround for MacOS to make the regex method work.

abom · 2021-05-27T10:46:06Z

@OmarElawady I think psutil is currently doing that by using sysctl system call.

The problem is that we get some inconsistent output, that's why the PID check is a better solution for now IMHO.

sameh-farouk · 2021-05-27T10:59:57Z

@OmarElawady @abom this why running our server as a privileged user will get the full cmd line on macOS. I assume peaking on the process memory already implemented but required the user to have enough permissions.

glad you both agreed on using the PID solution.

sameh-farouk added this to the later milestone May 25, 2021

sameh-farouk added the type_bug label May 25, 2021

sameh-farouk self-assigned this May 25, 2021

sameh-farouk modified the milestones: later, now May 26, 2021

sameh-farouk mentioned this issue May 29, 2021

verify Nginx by pid file instead of regex string #3128

Merged

5 tasks

xmonader closed this as completed in #3128 May 30, 2021

This was referenced Jun 6, 2021

fix issue#3160 #3161

Merged

startupcmd not able to get the process id of nginx #3166

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

on macOS, threebot server won't start without sudo #3116

on macOS, threebot server won't start without sudo #3116

sameh-farouk commented May 25, 2021 •

edited

Loading

sameh-farouk commented May 26, 2021 •

edited

Loading

sameh-farouk commented May 26, 2021 •

edited

Loading

OmarElawady commented May 27, 2021

abom commented May 27, 2021

sameh-farouk commented May 27, 2021

on macOS, threebot server won't start without sudo #3116

on macOS, threebot server won't start without sudo #3116

Comments

sameh-farouk commented May 25, 2021 • edited Loading

Description

Version information

Installation method

Steps to reproduce

Traceback/Logs/Alerts

sameh-farouk commented May 26, 2021 • edited Loading

in linux

in macOS and the threebot server running with sudo

in macOS and the threebot server running without sudo we hit the issue and the output of ps is

sameh-farouk commented May 26, 2021 • edited Loading

OmarElawady commented May 27, 2021

abom commented May 27, 2021

sameh-farouk commented May 27, 2021

sameh-farouk commented May 25, 2021 •

edited

Loading

sameh-farouk commented May 26, 2021 •

edited

Loading

in macOS and the threebot server running with `sudo`

in macOS and the threebot server running without `sudo` we hit the issue and the output of `ps` is

sameh-farouk commented May 26, 2021 •

edited

Loading