Clean up macOS cache and more #3089

UlisesGascon · 2022-11-22T05:22:24Z

As commented in #2917 (comment). We need a basic bash script to clean up the mac, I created a fork from paulaime/CleanUpMac:

cleaner.sh

    # Maintenance script for macOS
    # Custom script (forked from https://github.com/paulaime/CleanUpMac)
    #!/usr/bin/env bash

    # Ask for the administrator password upfront
    if [ "$EUID" -ne 0  ]; then
        echo "Please run as root"
        exit
    fi


    echo 'Empty the Trash on all mounted volumes and the main HDD…'
    sudo rm -rfv /Volumes/*/.Trashes &>/dev/null
    sudo rm -rfv ~/.Trash &>/dev/null

    echo 'Clear System Log Files…'
    sudo rm -rfv /private/var/log/asl/*.asl &>/dev/null
    sudo rm -rfv /Library/Logs/DiagnosticReports/* &>/dev/null
    sudo rm -rfv /Library/Logs/Adobe/* &>/dev/null
    rm -rfv ~/Library/Containers/com.apple.mail/Data/Library/Logs/Mail/* &>/dev/null
    rm -rfv ~/Library/Logs/CoreSimulator/* &>/dev/null

    echo 'Clear Adobe Cache Files…'
    sudo rm -rfv ~/Library/Application\ Support/Adobe/Common/Media\ Cache\ Files/* &>/dev/null

    echo 'Cleanup iOS Applications…'
    rm -rfv ~/Music/iTunes/iTunes\ Media/Mobile\ Applications/* &>/dev/null

    echo 'Remove iOS Device Backups…'
    rm -rfv ~/Library/Application\ Support/MobileSync/Backup/* &>/dev/null

    echo 'Cleanup XCode Derived Data and Archives…'
    rm -rfv ~/Library/Developer/Xcode/DerivedData/* &>/dev/null
    rm -rfv ~/Library/Developer/Xcode/Archives/* &>/dev/null

    echo 'Cleanup Homebrew Cache…'
    brew cleanup --force -s &>/dev/null
    brew cask cleanup &>/dev/null
    rm -rfv /Library/Caches/Homebrew/* &>/dev/null
    brew tap --repair &>/dev/null

    echo 'Cleanup MacOS Application caches'
    rm -rfv /System/Library/Caches/*

    echo 'Cleanup any old versions of gems…'
    gem cleanup &>/dev/null

    echo 'Purge inactive memory…'
    sudo purge

Logs

Here are some logs from the execution made in test-orka-macos11-x64-1

Before the script

```
administrator@test-orka-macos11-x64-1 ~ % uptime
20:36  up 3 days,  6:08, 1 user, load averages: 1,78 1,90 1,88
administrator@test-orka-macos11-x64-1 ~ % df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi   19Gi    44%  553788 941116212    0%   /
devfs           189Ki  189Ki    0Bi   100%     653         0  100%   /dev
/dev/disk2s4     90Gi  515Mi   19Gi     3%       2 941669998    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi   19Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  504Ki   19Gi     1%      17 941669983    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   55Gi   19Gi    75%  330544 941339456    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
administrator@test-orka-macos11-x64-1 ~ % sudo find / -size +200M -exec du -hs {} \;
395M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M	/Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M	/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M	/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
306M	/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
897M	/System/Library/dyld/dyld_shared_cache_arm64e
1,0G	/System/Library/dyld/aot_shared_cache
959M	/System/Library/dyld/dyld_shared_cache_x86_64h
960M	/System/Library/dyld/dyld_shared_cache_x86_64
167M	/System/Library/KernelCollections/SystemKernelExtensions.kc
594M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
186M	/System/Library/Fonts/Apple Color Emoji.ttc
36G	/System/Library/Caches/com.apple.coresymbolicationd/data
512M	/System/Volumes/VM/swapfile0
395M	/System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M	/System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
594M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
36G	/System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data
911M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
911M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
```

Preparation and Execution

```
administrator@test-orka-macos11-x64-1 ~ % touch cleaner.sh
administrator@test-orka-macos11-x64-1 ~ % nano cleaner.sh
# Paste the file content
administrator@test-orka-macos11-x64-1 ~ % chmod 777 cleaner.sh
administrator@test-orka-macos11-x64-1 ~ % sudo ./cleaner.sh
Empty the Trash on all mounted volumes and the main HDD…
Clear System Log Files…
Clear Adobe Cache Files…
Cleanup iOS Applications…
Remove iOS Device Backups…
Cleanup XCode Derived Data and Archives…
Cleanup Homebrew Cache…
Cleanup MacOS Application caches
/System/Library/Caches/MobileActivation/data_ark.plist
/System/Library/Caches/MobileActivation
/System/Library/Caches/OnDemandResources/AssetPacks
/System/Library/Caches/OnDemandResources
/System/Library/Caches/com.apple.CVMS
/System/Library/Caches/com.apple.Components2.SystemCache.Components
/System/Library/Caches/com.apple.IntlDataCache.le
/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
/System/Library/Caches/com.apple.SystemMigrationUserSizeCache
/System/Library/Caches/com.apple.apfsd.stats.plist
/System/Library/Caches/com.apple.coresymbolicationd/data
/System/Library/Caches/com.apple.coresymbolicationd
/System/Library/Caches/com.apple.factorydata
/System/Library/Caches/com.apple.kernelcaches
rm: /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions: Operation not permitted
rm: /System/Library/Caches/com.apple.kext.caches/Directories/System/Library: Operation not permitted
rm: /System/Library/Caches/com.apple.kext.caches/Directories/System: Operation not permitted
rm: /System/Library/Caches/com.apple.kext.caches/Directories: Operation not permitted
rm: /System/Library/Caches/com.apple.kext.caches/Startup: Operation not permitted
rm: /System/Library/Caches/com.apple.kext.caches: Operation not permitted
/System/Library/Caches/com.apple.preferencepanes.systemcache
Cleanup any old versions of gems…
Purge inactive memory…
```

After the execution

```
administrator@test-orka-macos11-x64-1 ~ % uptime
20:42  up 3 days,  6:14, 1 user, load averages: 2,91 2,12 1,94
administrator@test-orka-macos11-x64-1 ~ % df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi   19Gi    44%  553788 941116212    0%   /
devfs           188Ki  188Ki    0Bi   100%     650         0  100%   /dev
/dev/disk2s4     90Gi  515Mi   19Gi     3%       2 941669998    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi   19Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  504Ki   19Gi     1%      17 941669983    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   55Gi   19Gi    75%  330421 941339579    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
administrator@test-orka-macos11-x64-1 ~ % sudo find / -size +200M -exec du -hs {} \;
395M    /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M    /Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M    /Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M    /Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
306M    /System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
897M    /System/Library/dyld/dyld_shared_cache_arm64e
1,0G    /System/Library/dyld/aot_shared_cache
959M    /System/Library/dyld/dyld_shared_cache_x86_64h
960M    /System/Library/dyld/dyld_shared_cache_x86_64
167M    /System/Library/KernelCollections/SystemKernelExtensions.kc
594M    /System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M    /System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
186M    /System/Library/Fonts/Apple Color Emoji.ttc
512M    /System/Volumes/VM/swapfile0
395M    /System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M    /System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M    /System/Volumes/Data/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M    /System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
594M    /System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M    /System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
911M    /System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
911M    /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
```

After Reboot!

```
administrator@test-orka-macos11-x64-1 ~ % uptime
20:47  up 56 secs, 1 user, load averages: 2,47 0,75 0,28
administrator@test-orka-macos11-x64-1 ~ % df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi   55Gi    21%  553788 941116212    0%   /
devfs           188Ki  188Ki    0Bi   100%     650         0  100%   /dev
/dev/disk2s4     90Gi  1.0Mi   55Gi     1%       1 941669999    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi   55Gi     1%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  516Ki   55Gi     1%      17 941669983    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   20Gi   55Gi    27%  330435 941339565    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
administrator@test-orka-macos11-x64-1 ~ % sudo find / -size +200M -exec du -hs {} \;
395M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M	/Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M	/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M	/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
306M	/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
897M	/System/Library/dyld/dyld_shared_cache_arm64e
1,0G	/System/Library/dyld/aot_shared_cache
959M	/System/Library/dyld/dyld_shared_cache_x86_64h
960M	/System/Library/dyld/dyld_shared_cache_x86_64
167M	/System/Library/KernelCollections/SystemKernelExtensions.kc
594M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
186M	/System/Library/Fonts/Apple Color Emoji.ttc
395M	/System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
86M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
82M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
88M	/System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
594M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
911M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
911M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
911M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
```

Conclusions

The script seems working, as we deleted big files like /System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data with 36G and the machine seems stable 🤖 .

Not sure why, but running df -h without rebooting returned weird results (very similar to prior execution). In any case we will reboot the machine after manually deleting the cache files, so we can ignore this.

Diff: Before vs after

Diff: Before vs after reboot

Next steps

If we are happy with this approach we can start to work on a Jenkins job that will execute this on a cron base (once per week?) 🦾

The text was updated successfully, but these errors were encountered:

richardlau · 2022-11-22T15:08:44Z

Does the script require sudo? Jenkins runs as the iojs user who cannot sudo to root.

AshCripps · 2022-11-22T15:48:43Z

I believe it will to delete some of the files.

UlisesGascon · 2022-12-06T13:03:11Z

Let me check if I can rewrite the script to run without sudo. In the worst-case scenario we change some folder permissions with an Ansible script and run the script with Jenkins with the same permission level.

AshCripps · 2022-12-06T16:27:12Z

I dont think we will be able to change folder permission as they are system generated and dont think we should really mess with them.

UlisesGascon · 2023-01-31T20:11:52Z

It only took me a few minutes to install and run the script to manually fix #3166. Perhaps I could create a simple Ansible playbook to manage it? Even if it still requires manual interaction, at least we can document the process and use Ansible to manage it. 🤔

joyeecheung · 2023-09-10T19:15:20Z

just noticed that brew cleanup might be useful too, I ran it as administrator and it cleaned up ~500MB for nodejs/jenkins-alerts#526

joyeecheung · 2023-09-10T19:36:04Z

Some other finds:

/Users/iojs//jenkins_err.log can take up hundreds of megabyes. It should be safe to clean it up?
/Users/iojs/Library/Caches/node-gyp/ can build up to gigabytes. I think this is just where node-gyp download headers too, it should be save to just remove everything underneath.
Maybe doing git cleanup in the workspace works too. But I remember we could also just delete the workspace directory all together and it will create new ones anyway

UlisesGascon · 2023-09-11T06:37:31Z

just noticed that brew cleanup might be useful too, I ran it as administrator and it cleaned up ~500MB for nodejs/jenkins-alerts#526

In the cleaner.sh we had something related to it that also include cask related stuff:

echo 'Cleanup Homebrew Cache…'
brew cleanup --force -s &>/dev/null
brew cask cleanup &>/dev/null
rm -rfv /Library/Caches/Homebrew/* &>/dev/null
brew tap --repair &>/dev/null

/Users/iojs//jenkins_err.log can take up hundreds of megabyes. It should be safe to clean it up?

I think is ok to clean it up regularly

/Users/iojs/Library/Caches/node-gyp/ can build up to gigabytes. I think this is just where node-gyp download headers too, it should be save to just remove everything underneath.

Not sure, probably it is ok to clean up.

Maybe doing git cleanup in the workspace works too. But I remember we could also just delete the workspace directory all together and it will create new ones anyway

I think that we can remove the directory directly.

Should we move this cleaner.sh as file in this repo (as tools/macos_cleaner.sh) ? That way I think that will be more useful in long term and we also can close this issue.

joyeecheung · 2023-09-11T09:13:04Z

I think a PR adding it to the repo would be useful. We could also have some more eyeballs in the review to make sure we are not deleting things that we should not delete..

UlisesGascon · 2023-09-11T10:00:41Z

I think a PR adding it to the repo would be useful. We could also have some more eyeballs in the review to make sure we are not deleting things that we should not delete.

I created a draft PR #3481, feel free to add your improvements @joyeecheung 👍

UlisesGascon · 2024-05-07T13:13:52Z

I will close this issue as the PR #3481 was merged already.

UlisesGascon mentioned this issue Nov 22, 2022

No space left on macOS host? #2917

Closed

UlisesGascon added the platform:osx label Nov 22, 2022

UlisesGascon mentioned this issue Jan 31, 2023

Agent test-orka-macos11-x64-2 Disk space is too low #3166

Closed

UlisesGascon mentioned this issue Jul 7, 2023

Orka VMs restoring #3413

Closed

3 tasks

UlisesGascon self-assigned this Jul 11, 2023

UlisesGascon mentioned this issue Aug 12, 2023

low disk space in Macos machines #3450

Closed

5 tasks

UlisesGascon mentioned this issue Sep 11, 2023

Add clean up script for macos #3481

Merged

mhdawson mentioned this issue Jan 18, 2024

All OSX machines are offline #3612

Closed

UlisesGascon closed this as completed May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up macOS cache and more #3089

Clean up macOS cache and more #3089

UlisesGascon commented Nov 22, 2022 •

edited by targos

Loading

richardlau commented Nov 22, 2022

AshCripps commented Nov 22, 2022

UlisesGascon commented Dec 6, 2022

AshCripps commented Dec 6, 2022

UlisesGascon commented Jan 31, 2023 •

edited

Loading

joyeecheung commented Sep 10, 2023

joyeecheung commented Sep 10, 2023

UlisesGascon commented Sep 11, 2023

joyeecheung commented Sep 11, 2023

UlisesGascon commented Sep 11, 2023

UlisesGascon commented May 7, 2024

Clean up macOS cache and more #3089

Clean up macOS cache and more #3089

Comments

UlisesGascon commented Nov 22, 2022 • edited by targos Loading

Logs

Conclusions

Next steps

richardlau commented Nov 22, 2022

AshCripps commented Nov 22, 2022

UlisesGascon commented Dec 6, 2022

AshCripps commented Dec 6, 2022

UlisesGascon commented Jan 31, 2023 • edited Loading

joyeecheung commented Sep 10, 2023

joyeecheung commented Sep 10, 2023

UlisesGascon commented Sep 11, 2023

joyeecheung commented Sep 11, 2023

UlisesGascon commented Sep 11, 2023

UlisesGascon commented May 7, 2024

UlisesGascon commented Nov 22, 2022 •

edited by targos

Loading

UlisesGascon commented Jan 31, 2023 •

edited

Loading