Linux x64 machine running out of space

[AaronRobinsonMSFT]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=983418
Build error leg or test failing: baseservices-exceptions.WorkItemExecution
Pull request: #113578

+ result_exit_code=0
+ '[' 0 -ne 0 ']'
+ '[' 0 -ne 0 ']'
+ set_return 0
+ return 0
+ export _commandExitCode=0
+ _commandExitCode=0
+ /usr/bin/python3.12 /datadisks/disk1/work/A6BA0950/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 26245148 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Imt6UFh2cVJPMEN1UzRqU296REc4d21EM1RmcyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiIxMjFmYmM2MC0yMDA0LTRiZjgtODM1NC1iNzg1MDk5ZDJjMjgiLCJzaWQiOiJmMWI0MjI2My00ZjJhLTRiMDItYmYwYi0zZGYzOWFkMjEzMTciLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5Ozk4MzQxOCIsIkRlZklkIjoiMTI5Iiwiam9icmVmIjoiZjNkZDU5YzItMjEwZi00M2ZmLTgwN2YtNDIwNDViNDEzNTYwOmJmZmRkNmNlLTlhYmYtNWNmNS00MzdjLTViN2ZiMGNmNDk2OSIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC85ODM0MTgiLCJvcmNoaWQiOiJmM2RkNTljMi0yMTBmLTQzZmYtODA3Zi00MjA0NWI0MTM1NjAuYnVpbGQucnVuX3Rlc3RfcDBfY29yZWNscl9fbGludXhfeDY0X2NoZWNrZWQuX19kZWZhdWx0LjIiLCJyZXBvSWRzIjoiIiwiaXNzIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbSIsImF1ZCI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb218dnNvOjZmY2M5MmU1LTczYTctNGY4OC04ZDEzLWQ5MDQ1YjQ1ZmIyNyIsIm5iZiI6MTc0MjI1MDUzMCwiZXhwIjoxNzQyMjYzNzMwfQ.doaKZeAWINjybhPoUAwwXG5kNDi3VLTvt0A0ponMeCm5nGAj7sqfsUbNK1SYkM-XYIQbVaTSF98D2JMmiIlbAramzz2yXJzPqqqpuawCU6XtKrBMepoc7e3_XQrDka_0CdY4iPVe8RhPorUuy0kaFjeVAem7lRhEftajfpQEGvMWMOXEBRLB_AUjIixqWCZx_khHWxwqj5_ivvBMRWOPxk-3_1UV_MURBcYa6F0H0YbmI0y_nQy3w3571rTQoGdOJWCblUiGUPFTdZ20Ev0RSFwJlHhoTNUAPge3CSDZb2KpCXGXjq8q9SJbhjZNCRrzSpITWjVqYU91FnxkYLaqUg
2025-03-17T23:42:00.863Z	INFO   	run.py	managed_identity(145)	__init__	ManagedIdentityCredential will use IMDS with client_id: 3423231b-6b9e-4177-8817-76276b647404
2025-03-17T23:42:00.863Z	INFO   	run.py	azure_utils(21)	get_credential_and_access_token	VMSSManagedIdentity Auth type will be used
2025-03-17T23:42:00.865Z	INFO   	run.py	_universal(509)	on_request	Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'
Request method: 'GET'
Request headers:
    'Metadata': 'REDACTED'
    'User-Agent': 'azsdk-python-identity/1.21.0 Python/3.12.3 (Linux-6.6.64.2-9.azl3-x86_64-with-glibc2.38)'
No body was attached to the request
2025-03-17T23:42:00.874Z	INFO   	run.py	_universal(545)	on_response	Response status: 200
Response headers:
    'Content-Type': 'application/json; charset=utf-8'
    'Server': 'IMDS/150.870.65.1544'
    'x-ms-request-id': 'a6111e18-f959-4b97-8a5f-d5160d0afc6f'
    'Date': 'Mon, 17 Mar 2025 23:42:00 GMT'
    'Content-Length': '2061'
2025-03-17T23:42:00.875Z	INFO   	run.py	msal_managed_identity_client(165)	_get_token_base	ImdsCredential.get_token succeeded
2025-03-17T23:42:00.875Z	INFO   	run.py	decorators(24)	wrapper	ManagedIdentityCredential.get_token succeeded
2025-03-17T23:42:00.875Z	INFO   	run.py	azure_utils(44)	get_credential_and_access_token	Credentials are valid
2025-03-17T23:42:00.876Z	INFO   	run.py	_universal(509)	on_request	Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'
Request method: 'GET'
Request headers:
    'Metadata': 'REDACTED'
    'User-Agent': 'azsdk-python-identity/1.21.0 Python/3.12.3 (Linux-6.6.64.2-9.azl3-x86_64-with-glibc2.38)'
No body was attached to the request
2025-03-17T23:42:00.883Z	INFO   	run.py	_universal(545)	on_response	Response status: 200
Response headers:
    'Content-Type': 'application/json; charset=utf-8'
    'Server': 'IMDS/150.870.65.1544'
    'x-ms-request-id': '3bd0b43c-1d30-465f-8dbc-90249093d5ef'
    'Date': 'Mon, 17 Mar 2025 23:42:00 GMT'
    'Content-Length': '2061'
2025-03-17T23:42:00.884Z	INFO   	run.py	msal_managed_identity_client(165)	_get_token_base	ImdsCredential.get_token_info succeeded
2025-03-17T23:42:00.884Z	INFO   	run.py	decorators(24)	wrapper	ManagedIdentityCredential.get_token_info succeeded
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1164, in emit
    self.flush()
  File "/usr/lib/python3.12/logging/__init__.py", line 1144, in flush
    self.stream.flush()
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 13, in <module>
    from test_results_reader import read_results
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/datadisks/disk1/work/A6BA0950/p/reporter/test_results_reader/__init__.py", line 3, in <module>
    from helix.public import TestResult, TestResultAttachment
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/etc/helix/scripts/helix/public/__init__.py", line 5, in <module>
    import helix.event
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/etc/helix/scripts/helix/event.py", line 7, in <module>
    import helix.logs
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/etc/helix/scripts/helix/logs.py", line 70, in <module>
    append_blob_handler = AppendBlobHandler(specific_agent_log_uri, credential)
  File "/etc/helix/scripts/helix/append_blob_handler.py", line 64, in __init__
    self._unsafe_ensure_client()
  File "/etc/helix/scripts/helix/append_blob_handler.py", line 76, in _unsafe_ensure_client
    current_blobs = [blob for blob in self.container_client.walk_blobs(self.blob_name_prefix) if isinstance(blob, BlobProperties)] # type: List[BlobProperties]
  File "/etc/helix/scripts/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
  File "/etc/helix/scripts/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/etc/helix/scripts/azure/storage/blob/_list_blobs_helper.py", line 71, in _get_next_cb
    return self._command(
  File "/etc/helix/scripts/azure/storage/blob/_generated/operations/_container_operations.py", line 1324, in list_blob_hierarchy_segment
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "/etc/helix/scripts/azure/core/pipeline/_base.py", line 240, in run
    return first_node.send(pipeline_request)
  File "/etc/helix/scripts/azure/core/pipeline/_base.py", line 96, in send
    response = self.next.send(request)
  File "/etc/helix/scripts/azure/core/pipeline/_base.py", line 96, in send
    response = self.next.send(request)
  File "/etc/helix/scripts/azure/core/pipeline/_base.py", line 96, in send
    response = self.next.send(request)
  [Previous line repeated 3 more times]
  File "/etc/helix/scripts/azure/core/pipeline/policies/_authentication.py", line 157, in send
    self.on_request(request)
  File "/etc/helix/scripts/azure/core/pipeline/policies/_authentication.py", line 132, in on_request
    self._request_token(*self._scopes)
  File "/etc/helix/scripts/azure/core/pipeline/policies/_authentication.py", line 108, in _request_token
    self._token = self._get_token(*scopes, **kwargs)
  File "/etc/helix/scripts/azure/core/pipeline/policies/_authentication.py", line 98, in _get_token
    return cast(SupportsTokenInfo, self._credential).get_token_info(*scopes, options=options)
  File "/etc/helix/scripts/azure/identity/_internal/decorators.py", line 24, in wrapper
    _LOGGER.log(
Message: '%s succeeded'
Arguments: ('ManagedIdentityCredential.get_token_info',)
2025-03-17T23:42:01.218Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 48, in main
    log.info("Beginning reading of test results.")
Message: 'Beginning reading of test results.'
Arguments: ()
2025-03-17T23:42:01.228Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/A6BA0950/w/B4D70A32/e' for test results files
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 65, in main
    reporter.report_results(all_results)
  File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 27, in report_results
    results = [x for x in results if x]
  File "/datadisks/disk1/work/A6BA0950/p/reporter/test_results_reader/__init__.py", line 42, in read_results
    log.info("Searching '{}' for test results files".format(dir))
Message: "Searching '/datadisks/disk1/work/A6BA0950/w/B4D70A32/e' for test results files"
Arguments: ()
2025-03-17T23:42:01.231Z	INFO   	run.py	__init__(48)	read_results	Found results file /datadisks/disk1/work/A6BA0950/w/B4D70A32/e/baseservices/exceptions/baseservices-exceptions/baseservices-exceptions.testResults.xml with format xunit
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 65, in main
    reporter.report_results(all_results)
  File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 27, in report_results
    results = [x for x in results if x]
  File "/datadisks/disk1/work/A6BA0950/p/reporter/test_results_reader/__init__.py", line 48, in read_results
    log.info('Found results file {} with format {}'.format(file_path, f.name))
Message: 'Found results file /datadisks/disk1/work/A6BA0950/w/B4D70A32/e/baseservices/exceptions/baseservices-exceptions/baseservices-exceptions.testResults.xml with format xunit'
Arguments: ()
2025-03-17T23:42:01.232Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/A6BA0950/w/B4D70A32/uploads' for test results files
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 65, in main
    reporter.report_results(all_results)
  File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 27, in report_results
    results = [x for x in results if x]
  File "/datadisks/disk1/work/A6BA0950/p/reporter/test_results_reader/__init__.py", line 42, in read_results
    log.info("Searching '{}' for test results files".format(dir))
Message: "Searching '/datadisks/disk1/work/A6BA0950/w/B4D70A32/uploads' for test results files"
Arguments: ()
2025-03-17T23:42:01.234Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 23 test reports to '/datadisks/disk1/work/A6BA0950/w/B4D70A32/e/__test_report.json'
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 65, in main
    reporter.report_results(all_results)
  File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 30, in report_results
    self._log.info("Packing %d test reports to '%s'", len(results), path)
Message: "Packing %d test reports to '%s'"
Arguments: (23, '/datadisks/disk1/work/A6BA0950/w/B4D70A32/e/__test_report.json')
2025-03-17T23:42:01.234Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 7446 bytes
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/handlers.py", line 197, in shouldRollover
    self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
    ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 68, in <module>
    main()
  File "/datadisks/disk1/work/A6BA0950/p/reporter/run.py", line 65, in main
    reporter.report_results(all_results)
  File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 33, in report_results
    self._log.info("Packed %d bytes", os.path.getsize(path))
Message: 'Packed %d bytes'
Arguments: (7446,)
+ exit 0
['baseservices-exceptions' END OF WORK ITEM LOG: Command exited with 0]

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": "No space left on device",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=983418
Error message validated: [OSError: [Errno 28] No space left on device]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 3/18/2025 2:40:34 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=983418
Error message validated: [No space left on device]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 3/18/2025 3:40:40 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Report

Build	Definition	Test	Pull Request
985184	dotnet/runtime	Regression_4.WorkItemExecution	#113242

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

[jkotas]

@janvorli Can this be caused by your experiments that enable crash dumps for stack overflow tests in baseservices tests (#113533 and related PRs)?

[janvorli]

I have not merged that PR in, but it is possible that just the CI running for the PR has caused it.

[janvorli]

All the tests that crash a secondary process intentionally (unhandled exception, stack overflow, ...) end up generating dump via OS. They don't generate dumps via our CreateDump utility, as that's something that I have disabled in them a long time ago.
I think we should be able to disable the OS generated dumps for the intentionally crashing processes by pinvoking setrlimit with RLIMIT_CORE set to 0.

[janvorli]

I have it working locally, I will send out a PR soon.