Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove deprecated code and associated tests #289

Merged
merged 1 commit into from
May 25, 2023
Merged

Remove deprecated code and associated tests #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,18 +25,22 @@ Description

A full list of changes and detailed notes can be found below:

- Remove deprecated code
- Update Fortran tutorials for SmartRedis
- Add support for multiple network interface binding in Orchestrator and Colocated DBs

Detailed notes

- Deprecated launcher-specific orchestrators, constants, and ML utilities
were removed. (PR289_)
- Update the Github Actions runner image from `macos-10.15`` to `macos-12``. The
former began deprecation in May 2022 and was finally removed in May 2023 (PR285_)
- The Fortran tutorials had not been fully updated to show how to handle return/error
codes. These have now all been updated (PR284_)
- Orchestrator and Colocated DB now accept a list of interfaces to bind to. The
argument name is still `interface` for backward compatibility reasons. (PR281_)

.. _PR289: https://github.com/CrayLabs/SmartSim/pull/289
.. _PR285: https://github.com/CrayLabs/SmartSim/pull/285
.. _PR284: https://github.com/CrayLabs/SmartSim/pull/284
.. _PR281: https://github.com/CrayLabs/SmartSim/pull/281
Expand Down
54 changes: 0 additions & 54 deletions smartsim/constants.py
View file
Open in desktop

This file was deleted.

9 changes: 1 addition & 8 deletions smartsim/database/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,4 @@
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# deprecated classes
from .orchestrator import (
CobaltOrchestrator,
LSFOrchestrator,
Orchestrator,
PBSOrchestrator,
SlurmOrchestrator,
)
from .orchestrator import Orchestrator
290 changes: 0 additions & 290 deletions smartsim/database/orchestrator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,293 +849,3 @@ def _fill_reserved(self):
"n",
"nnodes",
]


#
# Deprecated Orchestrator Classes
#
# Same functionality incorporated into the Orchestrator base class
#


class CobaltOrchestrator(Orchestrator):
def __init__(
self,
port=6379,
db_nodes=1,
batch=True,
hosts=None,
run_command="aprun",
interface="ipogif0",
account=None,
queue=None,
time=None,
single_cmd=True,
**kwargs,
):
"""Initialize an Orchestrator reference for Cobalt based systems

The orchestrator launches as a batch by default. If batch=False,
at launch, the orchestrator will look for an interactive
allocation to launch on.

The Cobalt orchestrator does not support multiple databases per node.

:param port: TCP/IP port, defaults to 6379
:type port: int
:param db_nodes: number of database shards, defaults to 1
:type db_nodes: int, optional
:param batch: Run as a batch workload, defaults to True
:type batch: bool, optional
:param hosts: specify hosts to launch on, defaults to None. Optional if not launching with OpenMPI
:type hosts: list[str]
:param run_command: specify launch binary. Options are ``mpirun`` and ``aprun``, defaults to ``aprun``.
:type run_command: str, optional
:param interface: network interface to use, defaults to "ipogif0"
:type interface: str, optional
:param account: account to run batch on
:type account: str, optional
:param queue: queue to launch batch in
:type queue: str, optional
:param time: walltime for batch 'HH:MM:SS' format
:type time: str, optional
"""
simplefilter("once", DeprecationWarning)
msg = "CobaltOrchestrator(...) is deprecated and will be removed in a future release.\n"
msg += "Please update your code to use Orchestrator(launcher='cobalt', ...)."
warn(msg, DeprecationWarning, stacklevel=2)
super().__init__(
port,
interface,
db_nodes=db_nodes,
batch=batch,
run_command=run_command,
single_cmd=single_cmd,
launcher="cobalt",
hosts=hosts,
account=account,
queue=queue,
time=time,
**kwargs,
)


class LSFOrchestrator(Orchestrator):
def __init__(
self,
port=6379,
db_nodes=1,
cpus_per_shard=4,
gpus_per_shard=0,
batch=True,
hosts=None,
project=None,
time=None,
interface="ib0",
single_cmd=True,
**kwargs,
):
"""Initialize an Orchestrator reference for LSF based systems

The orchestrator launches as a batch by default. If
batch=False, at launch, the orchestrator will look for an interactive
allocation to launch on.

The LSFOrchestrator port provided will be incremented if multiple
databases per host are launched (``db_per_host>1``).

Each database shard is assigned a resource set with cpus and gpus
allocated contiguously on the host:
it is the user's responsibility to check if
enough resources are available on each host.

A list of hosts to launch the database on can be specified
these addresses must correspond to
those of the first ``db_nodes//db_per_host`` compute nodes
in the allocation: for example, for 8 ``db_nodes`` and 2 ``db_per_host``
the ``host_list`` must contain the addresses of hosts 1, 2, 3, and 4.

``LSFOrchestrator`` is launched with only one ``jsrun`` command
as launch binary, and an Explicit Resource File (ERF) which is
automatically generated. The orchestrator is always launched on the
first ``db_nodes//db_per_host`` compute nodes in the allocation.

:param port: TCP/IP port
:type port: int
:param db_nodes: number of database shards, defaults to 1
:type db_nodes: int, optional
:param cpus_per_shard: cpus to allocate per shard, defaults to 4
:type cpus_per_shard: int, optional
:param gpus_per_shard: gpus to allocate per shard, defaults to 0
:type gpus_per_shard: int, optional
:param batch: Run as a batch workload, defaults to True
:type batch: bool, optional
:param hosts: specify hosts to launch on
:type hosts: list[str], optional
:param project: project to run batch on
:type project: str, optional
:param time: walltime for batch 'HH:MM' format
:type time: str, optional
:param interface: network interface to use
:type interface: str
"""
simplefilter("once", DeprecationWarning)
msg = "LSFOrchestrator(...) is deprecated and will be removed in a future release.\n"
msg += "Please update your code to use Orchestrator(launcher='lsf', ...)."
warn(msg, DeprecationWarning, stacklevel=2)
if single_cmd != True:
raise SSUnsupportedError(
"LSFOrchestrator can only be run with single_cmd=True (MPMD)."
)

super().__init__(
port,
interface,
db_nodes=db_nodes,
batch=batch,
run_command="jsrun",
launcher="lsf",
project=project,
hosts=hosts,
time=time,
cpus_per_shard=cpus_per_shard,
gpus_per_shard=gpus_per_shard,
**kwargs,
)


class SlurmOrchestrator(Orchestrator):
def __init__(
self,
port=6379,
db_nodes=1,
batch=True,
hosts=None,
run_command="srun",
account=None,
time=None,
alloc=None,
db_per_host=1,
interface="ipogif0",
single_cmd=False,
**kwargs,
):
"""Initialize an Orchestrator reference for Slurm based systems

The orchestrator launches as a batch by default. The Slurm orchestrator
can also be given an allocation to run on. If no allocation is provided,
and batch=False, at launch, the orchestrator will look for an interactive
allocation to launch on.

The SlurmOrchestrator port provided will be incremented if multiple
databases per node are launched.

SlurmOrchestrator supports launching with both ``srun`` and ``mpirun``
as launch binaries. If mpirun is used, the hosts parameter should be
populated with length equal to that of the ``db_nodes`` argument.

:param port: TCP/IP port
:type port: int
:param db_nodes: number of database shards, defaults to 1
:type db_nodes: int, optional
:param batch: Run as a batch workload, defaults to True
:type batch: bool, optional
:param hosts: specify hosts to launch on
:type hosts: list[str]
:param run_command: specify launch binary. Options are "mpirun" and "srun", defaults to "srun"
:type run_command: str, optional
:param account: account to run batch on
:type account: str, optional
:param time: walltime for batch 'HH:MM:SS' format
:type time: str, optional
:param alloc: allocation to launch on, defaults to None
:type alloc: str, optional
:param db_per_host: number of database shards per system host (MPMD), defaults to 1
:type db_per_host: int, optional
:param single_cmd: run all shards with one (MPMD) command, defaults to True
:type single_cmd: bool
"""
simplefilter("once", DeprecationWarning)
msg = "SlurmOrchestrator(...) is deprecated and will be removed in a future release.\n"
msg += "Please update your code to use Orchestrator(launcher='slurm', ...)."
warn(msg, DeprecationWarning, stacklevel=2)
super().__init__(
port,
interface,
db_nodes=db_nodes,
batch=batch,
run_command=run_command,
alloc=alloc,
db_per_host=db_per_host,
single_cmd=single_cmd,
launcher="slurm",
account=account,
hosts=hosts,
time=time,
**kwargs,
)


class PBSOrchestrator(Orchestrator):
def __init__(
self,
port=6379,
db_nodes=1,
batch=True,
hosts=None,
run_command="aprun",
interface="ipogif0",
account=None,
time=None,
queue=None,
single_cmd=True,
**kwargs,
):
"""Initialize an Orchestrator reference for PBSPro based systems

The ``PBSOrchestrator`` launches as a batch by default. If batch=False,
at launch, the ``PBSOrchestrator`` will look for an interactive
allocation to launch on.

The PBS orchestrator does not support multiple databases per node.

If ``mpirun`` is specifed as the ``run_command``, then the ``hosts``
argument is required.

:param port: TCP/IP port
:type port: int
:param db_nodes: number of compute nodes to span accross, defaults to 1
:type db_nodes: int, optional
:param batch: run as a batch workload, defaults to True
:type batch: bool, optional
:param hosts: specify hosts to launch on, defaults to None
:type hosts: list[str]
:param run_command: specify launch binary. Options are ``mpirun`` and ``aprun``, defaults to "aprun"
:type run_command: str, optional
:param interface: network interface to use, defaults to "ipogif0"
:type interface: str, optional
:param account: account to run batch on
:type account: str, optional
:param time: walltime for batch 'HH:MM:SS' format
:type time: str, optional
:param queue: queue to launch batch in
:type queue: str, optional
"""
simplefilter("once", DeprecationWarning)
msg = "PBSOrchestrator(...) is deprecated and will be removed in a future release.\n"
msg += "Please update your code to use Orchestrator(launcher='pbs', ...)."
warn(msg, DeprecationWarning, stacklevel=2)
super().__init__(
port,
interface,
db_nodes=db_nodes,
batch=batch,
run_command=run_command,
single_cmd=single_cmd,
launcher="pbs",
hosts=hosts,
account=account,
queue=queue,
time=time,
**kwargs,
)
Loading