new generic easyblock for installing Rust crates with cargo: `Cargo` and `CargoPythonPackage` by Micket · Pull Request #2902 · easybuilders/easybuild-easyblocks

Micket · 2023-03-06T09:58:30Z

(created using eb --new-pr)

fixes #2899

easybuild/easyblocks/generic/cargo.py

Micket · 2023-03-06T15:44:11Z

This actually works as is, but we should clean some things up. Example easyconfig, ripgrep-13.0.0.eb

easyblock = 'Cargo'

name = 'ripgrep'
version = '13.0.0'

homepage = 'https://github.com/BurntSushi/ripgrep'
description = """ripgrep is a line-oriented search tool that recursively searches the current directory for a regex
pattern."""

toolchain = SYSTEM

source_urls = ['https://crates.io/api/v1/crates/']
sources = [
    {'download_filename': '%(name)s/%(version)s/download', 'filename': SOURCE_TAR_GZ},
    {'download_filename': 'aho-corasick/0.7.18/download', 'filename': 'aho-corasick-0.7.18.tar.gz'},
    {'download_filename': 'atty/0.2.14/download', 'filename': 'atty-0.2.14.tar.gz'},
    {'download_filename': 'base64/0.13.0/download', 'filename': 'base64-0.13.0.tar.gz'},
    {'download_filename': 'bitflags/1.2.1/download', 'filename': 'bitflags-1.2.1.tar.gz'},
    {'download_filename': 'bstr/0.2.16/download', 'filename': 'bstr-0.2.16.tar.gz'},
    {'download_filename': 'bytecount/0.6.2/download', 'filename': 'bytecount-0.6.2.tar.gz'},
    {'download_filename': 'cc/1.0.68/download', 'filename': 'cc-1.0.68.tar.gz'},
    {'download_filename': 'cfg-if/0.1.10/download', 'filename': 'cfg-if-0.1.10.tar.gz'},
    {'download_filename': 'cfg-if/1.0.0/download', 'filename': 'cfg-if-1.0.0.tar.gz'},
    {'download_filename': 'clap/2.33.3/download', 'filename': 'clap-2.33.3.tar.gz'},
    {'download_filename': 'crossbeam-utils/0.8.5/download', 'filename': 'crossbeam-utils-0.8.5.tar.gz'},
    {'download_filename': 'encoding_rs/0.8.28/download', 'filename': 'encoding_rs-0.8.28.tar.gz'},
    {'download_filename': 'encoding_rs_io/0.1.7/download', 'filename': 'encoding_rs_io-0.1.7.tar.gz'},
    {'download_filename': 'fnv/1.0.7/download', 'filename': 'fnv-1.0.7.tar.gz'},
    {'download_filename': 'fs_extra/1.2.0/download', 'filename': 'fs_extra-1.2.0.tar.gz'},
    {'download_filename': 'globset/0.4.7/download', 'filename': 'globset-0.4.7.tar.gz'},
    {'download_filename': 'grep/0.2.8/download', 'filename': 'grep-0.2.8.tar.gz'},
    {'download_filename': 'grep-cli/0.1.6/download', 'filename': 'grep-cli-0.1.6.tar.gz'},
    {'download_filename': 'grep-matcher/0.1.5/download', 'filename': 'grep-matcher-0.1.5.tar.gz'},
    {'download_filename': 'grep-pcre2/0.1.5/download', 'filename': 'grep-pcre2-0.1.5.tar.gz'},
    {'download_filename': 'grep-printer/0.1.6/download', 'filename': 'grep-printer-0.1.6.tar.gz'},
    {'download_filename': 'grep-regex/0.1.9/download', 'filename': 'grep-regex-0.1.9.tar.gz'},
    {'download_filename': 'grep-searcher/0.1.8/download', 'filename': 'grep-searcher-0.1.8.tar.gz'},
    {'download_filename': 'hermit-abi/0.1.18/download', 'filename': 'hermit-abi-0.1.18.tar.gz'},
    {'download_filename': 'ignore/0.4.18/download', 'filename': 'ignore-0.4.18.tar.gz'},
    {'download_filename': 'itoa/0.4.7/download', 'filename': 'itoa-0.4.7.tar.gz'},
    {'download_filename': 'jemallocator/0.3.2/download', 'filename': 'jemallocator-0.3.2.tar.gz'},
    {'download_filename': 'jemalloc-sys/0.3.2/download', 'filename': 'jemalloc-sys-0.3.2.tar.gz'},
    {'download_filename': 'jobserver/0.1.22/download', 'filename': 'jobserver-0.1.22.tar.gz'},
    {'download_filename': 'lazy_static/1.4.0/download', 'filename': 'lazy_static-1.4.0.tar.gz'},
    {'download_filename': 'libc/0.2.97/download', 'filename': 'libc-0.2.97.tar.gz'},
    {'download_filename': 'libm/0.1.4/download', 'filename': 'libm-0.1.4.tar.gz'},
    {'download_filename': 'log/0.4.14/download', 'filename': 'log-0.4.14.tar.gz'},
    {'download_filename': 'memchr/2.4.0/download', 'filename': 'memchr-2.4.0.tar.gz'},
    {'download_filename': 'memmap2/0.3.0/download', 'filename': 'memmap2-0.3.0.tar.gz'},
    {'download_filename': 'num_cpus/1.13.0/download', 'filename': 'num_cpus-1.13.0.tar.gz'},
    {'download_filename': 'once_cell/1.7.2/download', 'filename': 'once_cell-1.7.2.tar.gz'},
    {'download_filename': 'packed_simd_2/0.3.5/download', 'filename': 'packed_simd_2-0.3.5.tar.gz'},
    {'download_filename': 'pcre2/0.2.3/download', 'filename': 'pcre2-0.2.3.tar.gz'},
    {'download_filename': 'pcre2-sys/0.2.5/download', 'filename': 'pcre2-sys-0.2.5.tar.gz'},
    {'download_filename': 'pkg-config/0.3.19/download', 'filename': 'pkg-config-0.3.19.tar.gz'},
    {'download_filename': 'proc-macro2/1.0.27/download', 'filename': 'proc-macro2-1.0.27.tar.gz'},
    {'download_filename': 'quote/1.0.9/download', 'filename': 'quote-1.0.9.tar.gz'},
    {'download_filename': 'regex/1.5.4/download', 'filename': 'regex-1.5.4.tar.gz'},
    {'download_filename': 'regex-automata/0.1.10/download', 'filename': 'regex-automata-0.1.10.tar.gz'},
    {'download_filename': 'regex-syntax/0.6.25/download', 'filename': 'regex-syntax-0.6.25.tar.gz'},
    {'download_filename': 'ryu/1.0.5/download', 'filename': 'ryu-1.0.5.tar.gz'},
    {'download_filename': 'same-file/1.0.6/download', 'filename': 'same-file-1.0.6.tar.gz'},
    {'download_filename': 'serde/1.0.126/download', 'filename': 'serde-1.0.126.tar.gz'},
    {'download_filename': 'serde_derive/1.0.126/download', 'filename': 'serde_derive-1.0.126.tar.gz'},
    {'download_filename': 'serde_json/1.0.64/download', 'filename': 'serde_json-1.0.64.tar.gz'},
    {'download_filename': 'strsim/0.8.0/download', 'filename': 'strsim-0.8.0.tar.gz'},
    {'download_filename': 'syn/1.0.73/download', 'filename': 'syn-1.0.73.tar.gz'},
    {'download_filename': 'termcolor/1.1.2/download', 'filename': 'termcolor-1.1.2.tar.gz'},
    {'download_filename': 'textwrap/0.11.0/download', 'filename': 'textwrap-0.11.0.tar.gz'},
    {'download_filename': 'thread_local/1.1.3/download', 'filename': 'thread_local-1.1.3.tar.gz'},
    {'download_filename': 'unicode-width/0.1.8/download', 'filename': 'unicode-width-0.1.8.tar.gz'},
    {'download_filename': 'unicode-xid/0.2.2/download', 'filename': 'unicode-xid-0.2.2.tar.gz'},
    {'download_filename': 'walkdir/2.3.2/download', 'filename': 'walkdir-2.3.2.tar.gz'},
    {'download_filename': 'winapi/0.3.9/download', 'filename': 'winapi-0.3.9.tar.gz'},
    {'download_filename': 'winapi-i686-pc-windows-gnu/0.4.0/download', 'filename': 'winapi-i686-pc-windows-gnu-0.4.0.tar.gz'},
    {'download_filename': 'winapi-util/0.1.5/download', 'filename': 'winapi-util-0.1.5.tar.gz'},
    {'download_filename': 'winapi-x86_64-pc-windows-gnu/0.4.0/download', 'filename': 'winapi-x86_64-pc-windows-gnu-0.4.0.tar.gz'},
]
checksums = [
    {'ripgrep-13.0.0.tar.gz': 'f37c9d2c2bc7e00bd2653e13771397b94e452583da9b9494eabef627618d64bf'},
    {'aho-corasick-0.7.18.tar.gz': '1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f'},
    {'atty-0.2.14.tar.gz': 'd9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8'},
    {'base64-0.13.0.tar.gz': '904dfeac50f3cdaba28fc6f57fdcddb75f49ed61346676a78c4ffe55877802fd'},
    {'bitflags-1.2.1.tar.gz': 'cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693'},
    {'bstr-0.2.16.tar.gz': '90682c8d613ad3373e66de8c6411e0ae2ab2571e879d2efbf73558cc66f21279'},
    {'bytecount-0.6.2.tar.gz': '72feb31ffc86498dacdbd0fcebb56138e7177a8cc5cea4516031d15ae85a742e'},
    {'cc-1.0.68.tar.gz': '4a72c244c1ff497a746a7e1fb3d14bd08420ecda70c8f25c7112f2781652d787'},
    {'cfg-if-0.1.10.tar.gz': '4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822'},
    {'cfg-if-1.0.0.tar.gz': 'baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd'},
    {'clap-2.33.3.tar.gz': '37e58ac78573c40708d45522f0d80fa2f01cc4f9b4e2bf749807255454312002'},
    {'crossbeam-utils-0.8.5.tar.gz': 'd82cfc11ce7f2c3faef78d8a684447b40d503d9681acebed6cb728d45940c4db'},
    {'encoding_rs-0.8.28.tar.gz': '80df024fbc5ac80f87dfef0d9f5209a252f2a497f7f42944cff24d8253cac065'},
    {'encoding_rs_io-0.1.7.tar.gz': '1cc3c5651fb62ab8aa3103998dade57efdd028544bd300516baa31840c252a83'},
    {'fnv-1.0.7.tar.gz': '3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1'},
    {'fs_extra-1.2.0.tar.gz': '2022715d62ab30faffd124d40b76f4134a550a87792276512b18d63272333394'},
    {'globset-0.4.7.tar.gz': 'f0fc1b9fa0e64ffb1aa5b95daa0f0f167734fd528b7c02eabc581d9d843649b1'},
    {'grep-0.2.8.tar.gz': '51cb840c560b45a2ffd8abf00190382789d3f596663d5ffeb2e05931c20e8657'},
    {'grep-cli-0.1.6.tar.gz': '2dd110c34bb4460d0de5062413b773e385cbf8a85a63fc535590110a09e79e8a'},
    {'grep-matcher-0.1.5.tar.gz': '6d27563c33062cd33003b166ade2bb4fd82db1fd6a86db764dfdad132d46c1cc'},
    {'grep-pcre2-0.1.5.tar.gz': 'c6ecedbb372bd549ae8db02418f82e87e38622844409b6cbcac47dbc95eede41'},
    {'grep-printer-0.1.6.tar.gz': '05c271a24daedf5675b61a275a1d0af06e03312ab7856d15433ae6cde044dc72'},
    {'grep-regex-0.1.9.tar.gz': '121553c9768c363839b92fc2d7cdbbad44a3b70e8d6e7b1b72b05c977527bd06'},
    {'grep-searcher-0.1.8.tar.gz': '7fbdbde90ba52adc240d2deef7b6ad1f99f53142d074b771fe9b7bede6c4c23d'},
    {'hermit-abi-0.1.18.tar.gz': '322f4de77956e22ed0e5032c359a0f1273f1f7f0d79bfa3b8ffbc730d7fbcc5c'},
    {'ignore-0.4.18.tar.gz': '713f1b139373f96a2e0ce3ac931cd01ee973c3c5dd7c40c0c2efe96ad2b6751d'},
    {'itoa-0.4.7.tar.gz': 'dd25036021b0de88a0aff6b850051563c6516d0bf53f8638938edbb9de732736'},
    {'jemallocator-0.3.2.tar.gz': '43ae63fcfc45e99ab3d1b29a46782ad679e98436c3169d15a167a1108a724b69'},
    {'jemalloc-sys-0.3.2.tar.gz': '0d3b9f3f5c9b31aa0f5ed3260385ac205db665baa41d49bb8338008ae94ede45'},
    {'jobserver-0.1.22.tar.gz': '972f5ae5d1cb9c6ae417789196c803205313edde988685da5e3aae0827b9e7fd'},
    {'lazy_static-1.4.0.tar.gz': 'e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646'},
    {'libc-0.2.97.tar.gz': '12b8adadd720df158f4d70dfe7ccc6adb0472d7c55ca83445f6a5ab3e36f8fb6'},
    {'libm-0.1.4.tar.gz': '7fc7aa29613bd6a620df431842069224d8bc9011086b1db4c0e0cd47fa03ec9a'},
    {'log-0.4.14.tar.gz': '51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710'},
    {'memchr-2.4.0.tar.gz': 'b16bd47d9e329435e309c58469fe0791c2d0d1ba96ec0954152a5ae2b04387dc'},
    {'memmap2-0.3.0.tar.gz': '20ff203f7bdc401350b1dbaa0355135777d25f41c0bbc601851bbd6cf61e8ff5'},
    {'num_cpus-1.13.0.tar.gz': '05499f3756671c15885fee9034446956fff3f243d6077b91e5767df161f766b3'},
    {'once_cell-1.7.2.tar.gz': 'af8b08b04175473088b46763e51ee54da5f9a164bc162f615b91bc179dbf15a3'},
    {'packed_simd_2-0.3.5.tar.gz': '0e64858a2d3733fdd61adfdd6da89aa202f7ff0e741d2fc7ed1e452ba9dc99d7'},
    {'pcre2-0.2.3.tar.gz': '85b30f2f69903b439dd9dc9e824119b82a55bf113b29af8d70948a03c1b11ab1'},
    {'pcre2-sys-0.2.5.tar.gz': 'dec30e5e9ec37eb8fbf1dea5989bc957fd3df56fbee5061aa7b7a99dbb37b722'},
    {'pkg-config-0.3.19.tar.gz': '3831453b3449ceb48b6d9c7ad7c96d5ea673e9b470a1dc578c2ce6521230884c'},
    {'proc-macro2-1.0.27.tar.gz': 'f0d8caf72986c1a598726adc988bb5984792ef84f5ee5aa50209145ee8077038'},
    {'quote-1.0.9.tar.gz': 'c3d0b9745dc2debf507c8422de05d7226cc1f0644216dfdfead988f9b1ab32a7'},
    {'regex-1.5.4.tar.gz': 'd07a8629359eb56f1e2fb1652bb04212c072a87ba68546a04065d525673ac461'},
    {'regex-automata-0.1.10.tar.gz': '6c230d73fb8d8c1b9c0b3135c5142a8acee3a0558fb8db5cf1cb65f8d7862132'},
    {'regex-syntax-0.6.25.tar.gz': 'f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b'},
    {'ryu-1.0.5.tar.gz': '71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e'},
    {'same-file-1.0.6.tar.gz': '93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502'},
    {'serde-1.0.126.tar.gz': 'ec7505abeacaec74ae4778d9d9328fe5a5d04253220a85c4ee022239fc996d03'},
    {'serde_derive-1.0.126.tar.gz': '963a7dbc9895aeac7ac90e74f34a5d5261828f79df35cbed41e10189d3804d43'},
    {'serde_json-1.0.64.tar.gz': '799e97dc9fdae36a5c8b8f2cae9ce2ee9fdce2058c57a93e6099d919fd982f79'},
    {'strsim-0.8.0.tar.gz': '8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a'},
    {'syn-1.0.73.tar.gz': 'f71489ff30030d2ae598524f61326b902466f72a0fb1a8564c001cc63425bcc7'},
    {'termcolor-1.1.2.tar.gz': '2dfed899f0eb03f32ee8c6a0aabdb8a7949659e3466561fc0adf54e26d88c5f4'},
    {'textwrap-0.11.0.tar.gz': 'd326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060'},
    {'thread_local-1.1.3.tar.gz': '8018d24e04c95ac8790716a5987d0fec4f8b27249ffa0f7d33f1369bdfb88cbd'},
    {'unicode-width-0.1.8.tar.gz': '9337591893a19b88d8d87f2cec1e73fad5cdfd10e5a6f349f498ad6ea2ffb1e3'},
    {'unicode-xid-0.2.2.tar.gz': '8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3'},
    {'walkdir-2.3.2.tar.gz': '808cf2735cd4b6866113f648b791c6adc5714537bc222d9347bb203386ffda56'},
    {'winapi-0.3.9.tar.gz': '5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419'},
    {'winapi-i686-pc-windows-gnu-0.4.0.tar.gz': 'ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6'},
    {'winapi-util-0.1.5.tar.gz': '70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178'},
    {'winapi-x86_64-pc-windows-gnu-0.4.0.tar.gz': '712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f'},
]

# Coveniently convert build output to sources list:
# sed -e "s| *Downloaded \(.*\) v\(.*\)|    {'download_filename': '\\1/\\2/download', 'filename': '\\1-\\2.tar.gz'},|" vendor_output.txt | sort

builddependencies = [
    ('Rust', '1.65.0', '', ('GCCcore', '12.2.0')),
]

sanity_check_paths = {
    'files': ['bin/rg'],
    'dirs': [],
}

sanity_check_commands = ["rg --help"]

moduleclass = 'tools'

boegel · 2023-03-15T15:06:46Z

@Micket One idea to avoid the messy easyconfig could be to add a custom easyconfig parameter crates in which we only specify name/version/checksum, and let the Cargo easyblock then generate the sources/checksums easyconfig parameters...

crates = [
    ('ripgrep', '13.0.0', 'f37c9d2c2bc7e00bd2653e13771397b94e452583da9b9494eabef627618d64bf'),
    ...
]

Micket · 2023-03-19T23:01:23Z

So, i added support for crates = [(name, version), ...] but there is a major downside: inbject checksum doesn't work.

easyblock = 'Cargo'

name = 'ripgrep'
version = '13.0.0'

homepage = 'https://github.com/BurntSushi/ripgrep'
description = """ripgrep is a line-oriented search tool that recursively searches the current directory for a regex
pattern."""

toolchain = SYSTEM

crates = [
    (name, version),
    ('aho-corasick', '0.7.18'),
    ('atty', '0.2.14'),
    ('base64', '0.13.0'),
    ('bitflags', '1.2.1'),
    ('bstr', '0.2.16'),
    ('bytecount', '0.6.2'),
    ('cc', '1.0.68'),
    ('cfg-if', '0.1.10'),
    ('cfg-if', '1.0.0'),
    ('clap', '2.33.3'),
    ('crossbeam-utils', '0.8.5'),
    ('encoding_rs', '0.8.28'),
    ('encoding_rs_io', '0.1.7'),
    ('fnv', '1.0.7'),
    ('fs_extra', '1.2.0'),
    ('globset', '0.4.7'),
    ('grep', '0.2.8'),
    ('grep-cli', '0.1.6'),
    ('grep-matcher', '0.1.5'),
    ('grep-pcre2', '0.1.5'),
    ('grep-printer', '0.1.6'),
    ('grep-regex', '0.1.9'),
    ('grep-searcher', '0.1.8'),
    ('hermit-abi', '0.1.18'),
    ('ignore', '0.4.18'),
    ('itoa', '0.4.7'),
    ('jemallocator', '0.3.2'),
    ('jemalloc-sys', '0.3.2'),
    ('jobserver', '0.1.22'),
    ('lazy_static', '1.4.0'),
    ('libc', '0.2.97'),
    ('libm', '0.1.4'),
    ('log', '0.4.14'),
    ('memchr', '2.4.0'),
    ('memmap2', '0.3.0'),
    ('num_cpus', '1.13.0'),
    ('once_cell', '1.7.2'),
    ('packed_simd_2', '0.3.5'),
    ('pcre2', '0.2.3'),
    ('pcre2-sys', '0.2.5'),
    ('pkg-config', '0.3.19'),
    ('proc-macro2', '1.0.27'),
    ('quote', '1.0.9'),
    ('regex', '1.5.4'),
    ('regex-automata', '0.1.10'),
    ('regex-syntax', '0.6.25'),
    ('ryu', '1.0.5'),
    ('same-file', '1.0.6'),
    ('serde', '1.0.126'),
    ('serde_derive', '1.0.126'),
    ('serde_json', '1.0.64'),
    ('strsim', '0.8.0'),
    ('syn', '1.0.73'),
    ('termcolor', '1.1.2'),
    ('textwrap', '0.11.0'),
    ('thread_local', '1.1.3'),
    ('unicode-width', '0.1.8'),
    ('unicode-xid', '0.2.2'),
    ('walkdir', '2.3.2'),
    ('winapi', '0.3.9'),
    ('winapi-i686-pc-windows-gnu', '0.4.0'),
    ('winapi-util', '0.1.5'),
    ('winapi-x86_64-pc-windows-gnu', '0.4.0'),
]

# Coveniently convert build output to sources list:
# sed -e "s| *Downloaded \(.*\) v\(.*\)|    ('\\1', '\\2'),|" vendor_output.txt | sort

builddependencies = [
    ('Rust', '1.65.0', '', ('GCCcore', '12.2.0')),
]

sanity_check_paths = {
    'files': ['bin/rg'],
    'dirs': [],
}

sanity_check_commands = ["rg --help"]

moduleclass = 'tools'

Trying to inject checksums fails because there is no sources section anymore, and checksums are added after that or patches only.

So, the output claims it adds checksums

== Temporary log file in case of crash /dev/shm/eb-axy40zd7/easybuild-_98lg9rg.log
== injecting sha256 checksums in /apps/c3se-easyconfigs/ripgrep-13.0.0.eb
== fetching sources & patches for ripgrep-13.0.0.eb...
== backup of easyconfig file saved to /apps/c3se-easyconfigs/ripgrep-13.0.0.eb.bak_20230319230008_2243952...
== injecting sha256 checksums for sources & patches in ripgrep-13.0.0.eb...
== * ripgrep-13.0.0.tar.gz: f37c9d2c2bc7e00bd2653e13771397b94e452583da9b9494eabef627618d64bf
== * aho-corasick-0.7.18.tar.gz: 1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f
...

but it doesn't.

boegel · 2023-03-21T18:50:32Z

@Micket How about using exts_list instead the custom crates, would that make sense?
That could also fix the --inject-checksums issue, I think...

Micket · 2023-03-21T19:02:03Z

Yeah, so i'm afraid what would happen if we actually needed extensions to go along with a cargo package in the future. It also seems a bit ugly to abuse, since it does make a lot of assumptions that the sources must be downloaded from crates.io since i set the alt_location and such.
I also want to keep this open to the possibility of downloading from other repos other than crates.io in the future without locking myself into one path to much.
There would probably be a bunch of places where i need to prevent the normal loop-over-each-extension steps from executing, since these really aren't extensions.
So, all in all, i'm not super tempted by that option.

Quite a few of the cargo packages i expect to have some of the sources (probably the main application) grabbed from github. The checksums would work just fine in those cases.

We can just keep the empty "sources = []" field in those cases where it's crates.io stuff.

I think this is basically good to go as is.

Micket · 2023-03-21T22:03:51Z

Currently, easyconfig might look something like (includes improved way to generate crates list)

easyblock = 'Cargo'

name = 'ripgrep'
version = '13.0.0'

homepage = 'https://github.com/BurntSushi/ripgrep'
description = """ripgrep is a line-oriented search tool that recursively searches the current directory for a regex
pattern."""

toolchain = SYSTEM

# Coveniently generate crates list:
# cargo tree --prefix none | sed -e "s|\(.*\) v\([0-9\.]*\).*|    ('\\1', '\\2'),|"
crates = [
    (name, version),
    ('aho-corasick', '0.7.18'),
    ('atty', '0.2.14'),
    ('base64', '0.13.0'),
    ('bitflags', '1.2.1'),
    ('bstr', '0.2.16'),
    ('bytecount', '0.6.2'),
    ('cc', '1.0.68'),
    ('cfg-if', '0.1.10'),
    ('cfg-if', '1.0.0'),
    ('clap', '2.33.3'),
    ('crossbeam-utils', '0.8.5'),
    ('encoding_rs', '0.8.28'),
    ('encoding_rs_io', '0.1.7'),
    ('fnv', '1.0.7'),
    ('fs_extra', '1.2.0'),
    ('globset', '0.4.7'),
    ('grep', '0.2.8'),
    ('grep-cli', '0.1.6'),
    ('grep-matcher', '0.1.5'),
    ('grep-pcre2', '0.1.5'),
    ('grep-printer', '0.1.6'),
    ('grep-regex', '0.1.9'),
    ('grep-searcher', '0.1.8'),
    ('hermit-abi', '0.1.18'),
    ('ignore', '0.4.18'),
    ('itoa', '0.4.7'),
    ('jemallocator', '0.3.2'),
    ('jemalloc-sys', '0.3.2'),
    ('jobserver', '0.1.22'),
    ('lazy_static', '1.4.0'),
    ('libc', '0.2.97'),
    ('libm', '0.1.4'),
    ('log', '0.4.14'),
    ('memchr', '2.4.0'),
    ('memmap2', '0.3.0'),
    ('num_cpus', '1.13.0'),
    ('once_cell', '1.7.2'),
    ('packed_simd_2', '0.3.5'),
    ('pcre2', '0.2.3'),
    ('pcre2-sys', '0.2.5'),
    ('pkg-config', '0.3.19'),
    ('proc-macro2', '1.0.27'),
    ('quote', '1.0.9'),
    ('regex', '1.5.4'),
    ('regex-automata', '0.1.10'),
    ('regex-syntax', '0.6.25'),
    ('ryu', '1.0.5'),
    ('same-file', '1.0.6'),
    ('serde', '1.0.126'),
    ('serde_derive', '1.0.126'),
    ('serde_json', '1.0.64'),
    ('strsim', '0.8.0'),
    ('syn', '1.0.73'),
    ('termcolor', '1.1.2'),
    ('textwrap', '0.11.0'),
    ('thread_local', '1.1.3'),
    ('unicode-width', '0.1.8'),
    ('unicode-xid', '0.2.2'),
    ('walkdir', '2.3.2'),
    ('winapi', '0.3.9'),
    ('winapi-i686-pc-windows-gnu', '0.4.0'),
    ('winapi-util', '0.1.5'),
    ('winapi-x86_64-pc-windows-gnu', '0.4.0'),
]
sources = []
checksums = [
    {'ripgrep-13.0.0.tar.gz': 'f37c9d2c2bc7e00bd2653e13771397b94e452583da9b9494eabef627618d64bf'},
    {'aho-corasick-0.7.18.tar.gz': '1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f'},
    {'atty-0.2.14.tar.gz': 'd9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8'},
    {'base64-0.13.0.tar.gz': '904dfeac50f3cdaba28fc6f57fdcddb75f49ed61346676a78c4ffe55877802fd'},
    {'bitflags-1.2.1.tar.gz': 'cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693'},
    {'bstr-0.2.16.tar.gz': '90682c8d613ad3373e66de8c6411e0ae2ab2571e879d2efbf73558cc66f21279'},
    {'bytecount-0.6.2.tar.gz': '72feb31ffc86498dacdbd0fcebb56138e7177a8cc5cea4516031d15ae85a742e'},
    {'cc-1.0.68.tar.gz': '4a72c244c1ff497a746a7e1fb3d14bd08420ecda70c8f25c7112f2781652d787'},
    {'cfg-if-0.1.10.tar.gz': '4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822'},
    {'cfg-if-1.0.0.tar.gz': 'baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd'},
    {'clap-2.33.3.tar.gz': '37e58ac78573c40708d45522f0d80fa2f01cc4f9b4e2bf749807255454312002'},
    {'crossbeam-utils-0.8.5.tar.gz': 'd82cfc11ce7f2c3faef78d8a684447b40d503d9681acebed6cb728d45940c4db'},
    {'encoding_rs-0.8.28.tar.gz': '80df024fbc5ac80f87dfef0d9f5209a252f2a497f7f42944cff24d8253cac065'},
    {'encoding_rs_io-0.1.7.tar.gz': '1cc3c5651fb62ab8aa3103998dade57efdd028544bd300516baa31840c252a83'},
    {'fnv-1.0.7.tar.gz': '3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1'},
    {'fs_extra-1.2.0.tar.gz': '2022715d62ab30faffd124d40b76f4134a550a87792276512b18d63272333394'},
    {'globset-0.4.7.tar.gz': 'f0fc1b9fa0e64ffb1aa5b95daa0f0f167734fd528b7c02eabc581d9d843649b1'},
    {'grep-0.2.8.tar.gz': '51cb840c560b45a2ffd8abf00190382789d3f596663d5ffeb2e05931c20e8657'},
    {'grep-cli-0.1.6.tar.gz': '2dd110c34bb4460d0de5062413b773e385cbf8a85a63fc535590110a09e79e8a'},
    {'grep-matcher-0.1.5.tar.gz': '6d27563c33062cd33003b166ade2bb4fd82db1fd6a86db764dfdad132d46c1cc'},
    {'grep-pcre2-0.1.5.tar.gz': 'c6ecedbb372bd549ae8db02418f82e87e38622844409b6cbcac47dbc95eede41'},
    {'grep-printer-0.1.6.tar.gz': '05c271a24daedf5675b61a275a1d0af06e03312ab7856d15433ae6cde044dc72'},
    {'grep-regex-0.1.9.tar.gz': '121553c9768c363839b92fc2d7cdbbad44a3b70e8d6e7b1b72b05c977527bd06'},
    {'grep-searcher-0.1.8.tar.gz': '7fbdbde90ba52adc240d2deef7b6ad1f99f53142d074b771fe9b7bede6c4c23d'},
    {'hermit-abi-0.1.18.tar.gz': '322f4de77956e22ed0e5032c359a0f1273f1f7f0d79bfa3b8ffbc730d7fbcc5c'},
    {'ignore-0.4.18.tar.gz': '713f1b139373f96a2e0ce3ac931cd01ee973c3c5dd7c40c0c2efe96ad2b6751d'},
    {'itoa-0.4.7.tar.gz': 'dd25036021b0de88a0aff6b850051563c6516d0bf53f8638938edbb9de732736'},
    {'jemallocator-0.3.2.tar.gz': '43ae63fcfc45e99ab3d1b29a46782ad679e98436c3169d15a167a1108a724b69'},
    {'jemalloc-sys-0.3.2.tar.gz': '0d3b9f3f5c9b31aa0f5ed3260385ac205db665baa41d49bb8338008ae94ede45'},
    {'jobserver-0.1.22.tar.gz': '972f5ae5d1cb9c6ae417789196c803205313edde988685da5e3aae0827b9e7fd'},
    {'lazy_static-1.4.0.tar.gz': 'e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646'},
    {'libc-0.2.97.tar.gz': '12b8adadd720df158f4d70dfe7ccc6adb0472d7c55ca83445f6a5ab3e36f8fb6'},
    {'libm-0.1.4.tar.gz': '7fc7aa29613bd6a620df431842069224d8bc9011086b1db4c0e0cd47fa03ec9a'},
    {'log-0.4.14.tar.gz': '51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710'},
    {'memchr-2.4.0.tar.gz': 'b16bd47d9e329435e309c58469fe0791c2d0d1ba96ec0954152a5ae2b04387dc'},
    {'memmap2-0.3.0.tar.gz': '20ff203f7bdc401350b1dbaa0355135777d25f41c0bbc601851bbd6cf61e8ff5'},
    {'num_cpus-1.13.0.tar.gz': '05499f3756671c15885fee9034446956fff3f243d6077b91e5767df161f766b3'},
    {'once_cell-1.7.2.tar.gz': 'af8b08b04175473088b46763e51ee54da5f9a164bc162f615b91bc179dbf15a3'},
    {'packed_simd_2-0.3.5.tar.gz': '0e64858a2d3733fdd61adfdd6da89aa202f7ff0e741d2fc7ed1e452ba9dc99d7'},
    {'pcre2-0.2.3.tar.gz': '85b30f2f69903b439dd9dc9e824119b82a55bf113b29af8d70948a03c1b11ab1'},
    {'pcre2-sys-0.2.5.tar.gz': 'dec30e5e9ec37eb8fbf1dea5989bc957fd3df56fbee5061aa7b7a99dbb37b722'},
    {'pkg-config-0.3.19.tar.gz': '3831453b3449ceb48b6d9c7ad7c96d5ea673e9b470a1dc578c2ce6521230884c'},
    {'proc-macro2-1.0.27.tar.gz': 'f0d8caf72986c1a598726adc988bb5984792ef84f5ee5aa50209145ee8077038'},
    {'quote-1.0.9.tar.gz': 'c3d0b9745dc2debf507c8422de05d7226cc1f0644216dfdfead988f9b1ab32a7'},
    {'regex-1.5.4.tar.gz': 'd07a8629359eb56f1e2fb1652bb04212c072a87ba68546a04065d525673ac461'},
    {'regex-automata-0.1.10.tar.gz': '6c230d73fb8d8c1b9c0b3135c5142a8acee3a0558fb8db5cf1cb65f8d7862132'},
    {'regex-syntax-0.6.25.tar.gz': 'f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b'},
    {'ryu-1.0.5.tar.gz': '71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e'},
    {'same-file-1.0.6.tar.gz': '93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502'},
    {'serde-1.0.126.tar.gz': 'ec7505abeacaec74ae4778d9d9328fe5a5d04253220a85c4ee022239fc996d03'},
    {'serde_derive-1.0.126.tar.gz': '963a7dbc9895aeac7ac90e74f34a5d5261828f79df35cbed41e10189d3804d43'},
    {'serde_json-1.0.64.tar.gz': '799e97dc9fdae36a5c8b8f2cae9ce2ee9fdce2058c57a93e6099d919fd982f79'},
    {'strsim-0.8.0.tar.gz': '8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a'},
    {'syn-1.0.73.tar.gz': 'f71489ff30030d2ae598524f61326b902466f72a0fb1a8564c001cc63425bcc7'},
    {'termcolor-1.1.2.tar.gz': '2dfed899f0eb03f32ee8c6a0aabdb8a7949659e3466561fc0adf54e26d88c5f4'},
    {'textwrap-0.11.0.tar.gz': 'd326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060'},
    {'thread_local-1.1.3.tar.gz': '8018d24e04c95ac8790716a5987d0fec4f8b27249ffa0f7d33f1369bdfb88cbd'},
    {'unicode-width-0.1.8.tar.gz': '9337591893a19b88d8d87f2cec1e73fad5cdfd10e5a6f349f498ad6ea2ffb1e3'},
    {'unicode-xid-0.2.2.tar.gz': '8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3'},
    {'walkdir-2.3.2.tar.gz': '808cf2735cd4b6866113f648b791c6adc5714537bc222d9347bb203386ffda56'},
    {'winapi-0.3.9.tar.gz': '5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419'},
    {'winapi-i686-pc-windows-gnu-0.4.0.tar.gz': 'ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6'},
    {'winapi-util-0.1.5.tar.gz': '70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178'},
    {'winapi-x86_64-pc-windows-gnu-0.4.0.tar.gz': '712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f'},
]

builddependencies = [
    ('Rust', '1.65.0', '', ('GCCcore', '12.2.0')),
]

sanity_check_paths = {
    'files': ['bin/rg'],
    'dirs': [],
}

sanity_check_commands = ["rg --help"]

moduleclass = 'tools'

boegel · 2023-03-22T08:57:24Z

Yeah, so i'm afraid what would happen if we actually needed extensions to go along with a cargo package in the future. It also seems a bit ugly to abuse, since it does make a lot of assumptions that the sources must be downloaded from crates.io since i set the alt_location and such. I also want to keep this open to the possibility of downloading from other repos other than crates.io in the future without locking myself into one path to much. There would probably be a bunch of places where i need to prevent the normal loop-over-each-extension steps from executing, since these really aren't extensions. So, all in all, i'm not super tempted by that option.

Quite a few of the cargo packages i expect to have some of the sources (probably the main application) grabbed from github. The checksums would work just fine in those cases.

We can just keep the empty "sources = []" field in those cases where it's crates.io stuff.

I think this is basically good to go as is.

Makes total sense!

@PetrKralCZ Can you take this for a spin for bamtofastq (which breaks over time because we don't have full control over the installation process, see easybuilders/easybuild-easyconfigs#15636) and SnapATAC2?

@Micket Do you think it makes sense to come up with a nice way to generate what should be added to crates for a given Rust software package?
I wonder if we can support something like python3 -m easybuild.easyblocks.cargo ripgrep 13.0.0 that just spits out crates = [...] (doesn't need to be done in this PR)

Micket · 2023-03-22T14:45:15Z

Yes, I added a simpler method for fetching this information

# cargo tree --prefix none | sed -e "s|\(.*\) v\([0-9\.]*\).*|    ('\\1', '\\2'),|"

without needing to download anything extra. Unfortunately, it needs the TOML file from the actual application, manually downloaded, first, so that complicates it a bit.
I couldn't find a way to just probe this information from crates.io directly, though i feel it really should be possible; all the deps are listed there after all.
You can still use the cargo tree command above to simplify creating them until we have something nicer.

Micket · 2023-03-22T17:20:34Z

ok, annoying. can't put non-crates into the same path as the vendored packages, else it complains when it can't find the .cargo json thing.

also

error: no matching package named `redox_syscall` found
location searched: registry `crates-io`
required by package `tempfile v3.2.0`
    ... which satisfies dependency `tempfile = "*"` (locked to 3.2.0) of package `bamtofastq v1.4.0 (/dev/shm/bamtofastq/1.4.0/GCCcore-10.3.0/bamtofastq-1.4.0)`

cargo tree apparently just lies...

└── tempfile v3.2.0
    ├── cfg-if v1.0.0
    ├── libc v0.2.103
    ├── rand v0.8.4
    │   ├── libc v0.2.103
    │   ├── rand_chacha v0.3.1
    │   │   ├── ppv-lite86 v0.2.10
    │   │   └── rand_core v0.6.3
    │   │       └── getrandom v0.2.3
    │   │           ├── cfg-if v1.0.0
    │   │           └── libc v0.2.103
    │   └── rand_core v0.6.3 (*)
    └── remove_dir_all v0.5.3

doesn't show "redox_syscall". It is listed in the Cargo.lock file

[[package]]
name = "tempfile"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dac1c663cfc93810f88aed9b8941d48cabf856a1b111c29a40439018d870eb22"
dependencies = [
 "cfg-if",
 "libc",
 "rand",
 "redox_syscall",
 "remove_dir_all",
 "winapi",
]

So, i don't know why that is broken...

easybuild/easyblocks/generic/cargo.py

akesandgren · 2023-05-23T14:43:14Z

Class Cargo should probably verify that Rust is available, like pythonbundle/package does for Python.

boegel · 2023-05-23T16:20:55Z

Class Cargo should probably verify that Rust is available, like pythonbundle/package does for Python.

Maybe, but it indirectly already does since it assumes the cargo command is available (which could also be provided by the OS when using system toolchain).

So, I won't let this block this PR, taking this for a final spin before (finally) merging it...

akesandgren · 2023-05-24T05:39:58Z

If should at least check for existans of rustc or cargo and if not available bail out cleanly instead of possibly crashing due to not finding the cmd?

boegel · 2023-05-24T05:41:04Z

Extensively tested via:

There's an open problem when --optarch is used (cfr. discussion in easybuilders/easybuild-easyconfigs#17666), but that can be tackled in a follow-up PR.

Thanks a lot for all the effort on the @Micket!

adding easyblocks: cargo.py

22be21a

Micket marked this pull request as draft March 6, 2023 09:59

Micket added the new label Mar 6, 2023

Micket added 3 commits March 6, 2023 14:54

Fix hound errors

7cc78df

Fix hound errors

df16d82

Fix hound errors

863dfb8

boegel mentioned this pull request Mar 6, 2023

control installing of Rust packages vscentrum/vsc-software-stack#83

Closed

Micket added 2 commits March 6, 2023 15:10

Use profile consistently

f93988f

Fix copy/paste typo

3740be1

Micket commented Mar 6, 2023

View reviewed changes

easybuild/easyblocks/generic/cargo.py Outdated Show resolved Hide resolved

easybuild/easyblocks/generic/cargo.py Outdated Show resolved Hide resolved

easybuild/easyblocks/generic/cargo.py Outdated Show resolved Hide resolved

easybuild/easyblocks/generic/cargo.py Show resolved Hide resolved

boegel changed the title ~~new easyblock for cargo~~ new generic easyblock for installing Rust crates with cargo Mar 15, 2023

boegel added this to the release after 4.7.1 milestone Mar 15, 2023

Add support for crate list, minor other fixes

b1a7072

Use alt location for crates

20dca5b

Micket marked this pull request as ready for review March 21, 2023 22:02

Micket mentioned this pull request Mar 22, 2023

switch bamtofastq to Cargo easyblock easybuilders/easybuild-easyconfigs#17595

Merged

2 tasks

Micket commented Mar 22, 2023

View reviewed changes

easybuild/easyblocks/generic/cargo.py Outdated Show resolved Hide resolved

boegel requested changes Mar 29, 2023

View reviewed changes

Micket added 3 commits March 29, 2023 18:52

Join argument list, fix typos

f10d6a1

Add missing import for change_dir

7904bf0

Fix trailing space

ecf69fd

Micket added 2 commits March 29, 2023 20:26

Fix cargo checksum into each extracted dir

9d4225b

Remove unused var

10b604d

Micket mentioned this pull request Apr 8, 2023

update easyconfig for Longshot 0.4.5 to use Cargo easyblock easybuilders/easybuild-easyconfigs#17666

Merged

2 tasks

Micket added 3 commits April 13, 2023 04:13

Include script for generating crate list

d04f012

Trailing whitespace

e83b8c6

blank line

00c0f10

Micket mentioned this pull request Apr 30, 2023

{tools}[GCCcore/12.2.0] Evcxr-REPL v0.14.2 easybuilders/easybuild-easyconfigs#17831

Merged

2 tasks

Micket added 5 commits May 2, 2023 01:13

Add cargopythonpackage

f15f300

Fix code style

13807fb

Move config.toml modifications to extract step

8511325

Remove debug prints

9067045

Add partial support for vendoring of non-crates.io crates

84af134

Micket mentioned this pull request May 4, 2023

{math}[foss/2022a] rustworkx v0.12.1 w/ Python 3.10.4 easybuilders/easybuild-easyconfigs#17857

Merged

2 tasks

Micket added 4 commits May 5, 2023 16:56

Fix mistake in vendoring of sources

348332b

Break up long line

0803ce9

Fix style

480ffdc

Switch cargo to extensioneasyblock to allow use as an extension

b34fd46

Micket mentioned this pull request May 22, 2023

{tools}[system/system] ripunzip v0.4.0 easybuilders/easybuild-easyconfigs#17959

Merged

2 tasks

boegel approved these changes May 24, 2023

View reviewed changes

boegel merged commit eb236c0 into easybuilders:develop May 24, 2023

boegel mentioned this pull request May 24, 2023

figure out how to control version of Rust packages that are installed as dependency easybuilders/easybuild-easyconfigs#15763

Closed

boegel changed the title ~~new generic easyblock for installing Rust crates with cargo~~ new generic easyblock for installing Rust crates with cargo: Cargo and CargoPythonPackage May 27, 2023

boegel changed the title ~~new generic easyblock for installing Rust crates with cargo: Cargo and CargoPythonPackage~~ new generic easyblock for installing Rust crates with cargo: Cargo and CargoPythonPackage May 27, 2023

boegel mentioned this pull request Jun 21, 2023

{bio}[GCCcore/8.3.0] Varlociraptor v1.5.0 easybuilders/easybuild-easyconfigs#9491

Closed

Micket deleted the 20230306095828_new_pr_cargo branch April 3, 2024 10:43

Comments

Conversation

Micket commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Micket commented Mar 6, 2023

Uh oh!

boegel commented Mar 15, 2023

Uh oh!

Micket commented Mar 19, 2023

Uh oh!

boegel commented Mar 21, 2023

Uh oh!

Micket commented Mar 21, 2023

Uh oh!

Micket commented Mar 21, 2023

Uh oh!

boegel commented Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Micket commented Mar 22, 2023

Uh oh!

Micket commented Mar 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

akesandgren commented May 23, 2023

Uh oh!

boegel commented May 23, 2023

Uh oh!

akesandgren commented May 24, 2023

Uh oh!

boegel commented May 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Micket commented Mar 6, 2023 •

edited

Loading

boegel commented Mar 22, 2023 •

edited

Loading