Subprocess command time out on large repos with version >= 8

[skarimo]

Looks the following PR introduced a default timeout of 20s when running subprocess commands: https://github.com/pypa/setuptools_scm/pull/808/files

This causes issues with larger repos/slow connections where repo needs to be un-shallowed and can take >20s.

Would be great if we can revert back to the old behavior or make the timeout configurable.

[RonnyPfannschmidt]

I need more details on the setup

[skarimo]

@RonnyPfannschmidt thanks for following up. Our setup looks as follows:

def parse_fetch_on_shallow(*args, **kwargs):
    from setuptools_scm.git import parse, fetch_on_shallow
    kwargs["pre_parse"] = fetch_on_shallow
    return parse(*args, **kwargs)


setup(
    use_scm_version={
        "local_scheme": "dirty-tag",
        "write_to": os.path.join("src", "api_client", "version.py"),
        "write_to_template": version_template,
        "parse": parse_fetch_on_shallow,
    }
)

On shallow clones, it tries to fetch full history and times out with:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [74 lines of output]
      /opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!
      
              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************
      
      !!
        dist.fetch_build_eggs(dist.setup_requires)
      /home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py:159: UserWarning: "/home/runner/work/api-client-python/api-client-python" was shallow, git fetch was used to rectify
        warnings.warn(f'"{wd.path}" was shallow, git fetch was used to rectify')
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/home/runner/work/api-client-python/api-client-python/setup.py", line 27, in <module>
          setup(
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 147, in setup
          _setup_distribution = dist = klass(attrs)
                                       ^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/dist.py", line 303, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
          self.finalize_options()
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/dist.py", line 680, in finalize_options
          ep(self)
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/setuptools/dist.py", line 700, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_integration/setuptools.py", line 102, in version_keyword
          _assign_version(dist, config)
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_integration/setuptools.py", line 54, in _assign_version
          maybe_version = _get_version(config, force_write_version_files=True)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_get_version_impl.py", line 89, in _get_version
          parsed_version = parse_version(config)
                           ^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_get_version_impl.py", line 52, in parse_version
          or parse_scm_version(config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_get_version_impl.py", line 27, in parse_scm_version
          parse_result = config.parse(config.absolute_root, config=config)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/setup.py", line 24, in parse_fetch_on_shallow
          return parse(root, pre_parse=fetch_on_shallow, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py", line 201, in parse
          return _git_parse_inner(
                 ^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py", line 243, in _git_parse_inner
          pre_parse(wd)
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py", line 160, in fetch_on_shallow
          wd.fetch_shallow()
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py", line 130, in fetch_shallow
          run_git(["fetch", "--unshallow"], self.path, check=True)
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/git.py", line 53, in run_git
          return _run(["git", "--git-dir", repo / ".git", *args], cwd=repo, check=check)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/runner/work/api-client-python/api-client-python/.eggs/setuptools_scm-8.0.1-py3.11.egg/setuptools_scm/_run_cmd.py", line 138, in run
          res = subprocess.run(
                ^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/subprocess.py", line 550, in run
          stdout, stderr = process.communicate(input, timeout=timeout)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/subprocess.py", line 1209, in communicate
          stdout, stderr = self._communicate(input, endtime, timeout)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/subprocess.py", line 2109, in _communicate
          self._check_timeout(endtime, orig_timeout, stdout, stderr)
        File "/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/subprocess.py", line 1253, in _check_timeout
          raise TimeoutExpired(
      subprocess.TimeoutExpired: Command '['git', '--git-dir', '/home/runner/work/api-client-python/api-client-python/.git', 'fetch', '--unshallow']' timed out after 20 seconds
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

[RonnyPfannschmidt]

Thanks, do you have a rough idea for reasonable uppers bounds

I intend to set a different timeout for unshallow as starting point and add a configuration option later