Skip to content

[Core] Ray component worker_ports is trying to use a port number 17507 that is used by other components. #25793

@chaomengyuan

Description

@chaomengyuan

What happened + What you expected to happen

Traceback (most recent call last):
  File "/usr/local/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2339, in main
    return cli()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py", line 852, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ray/scripts/scripts.py", line 873, in start
    ray_params, head=False, shutdown_at_exit=block, spawn_reaper=block
  File "/usr/local/lib/python3.7/site-packages/ray/node.py", line 266, in __init__
    self._ray_params.update_pre_selected_port()
  File "/usr/local/lib/python3.7/site-packages/ray/_private/parameter.py", line 323, in update_pre_selected_port
    f"Ray component {comp} is trying to use "
ValueError: Ray component worker_ports is trying to use a port number 17507 that is used by other components.
Port information: {'gcs': 'random', 'object_manager': 'random', 'node_manager': 'random', 'gcs_server': 'random', 'client_server': 10001, 'dashboard': 8265, 'dashboard_agent_grpc': 57208, 'dashboard_agent_http': 'random', 'metrics_export': 17507, 'redis_shards': 'random', 'worker_ports': '9998 ports from 10002 to 19999'}
If you allocate ports, please make sure the same port is not used by multiple components.

From error information,'metrics_export' uses port 17507, this is because if user does not set a port for metrics_export,it will be selected by: new_port = random.randint(port, 65535),L1, L2。If this random number happens to be in range: worker_ports[10002, 19999], in later code L3, L4, where it checks if there is a port being used repeatedly, an error will be thrown up。

Versions / Dependencies

nightly version

Reproduction script

No.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Issue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray Core

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions