Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(taskfiles): Add cmake and remote utils; Update checksum data path to an array. #16

Merged
merged 17 commits into from
Jan 30, 2025
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 173 additions & 12 deletions taskfiles/utils.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ tasks:
# CHECKSUM UTILS
# ===

# @param {string} DATA_DIR The directory to compute the checksum for.
# @param {[]string} [DATA_PATHS] List of paths to compute the checksum for.
# @param {string} OUTPUT_FILE
# @param {[]string} [EXCLUDE_PATHS] A list of paths, relative to `DATA_DIR`, to exclude from the
# checksum.
# @param {[]string} [EXCLUDE_PATHS] List of paths, relative to any `DATA_PATHS`, to exclude from
# the checksum.
compute-checksum:
desc: "Tries to compute a checksum for the given directory and output it to a file."
internal: true
Expand All @@ -18,7 +18,6 @@ tasks:
cmds:
- >-
tar cf -
--directory "{{.DATA_DIR}}"
--group 0
--mtime "UTC 1970-01-01"
--numeric-owner
Expand All @@ -27,15 +26,18 @@ tasks:
{{- range .EXCLUDE_PATHS}}
--exclude="{{.}}"
{{- end}}
. 2> /dev/null
{{- range .DATA_PATHS}}
"{{.}}"
{{- end}}
2> /dev/null
| md5sum > {{.OUTPUT_FILE}}
# Ignore errors so that dependent tasks don't fail
ignore_error: true

# @param {string} DATA_DIR The directory to validate the checksum for.
# @param {[]string} [DATA_PATHS] List of paths to validate the checksum for.
# @param {string} OUTPUT_FILE
# @param {[]string} [EXCLUDE_PATHS] A list of paths, relative to `DATA_DIR`, to exclude from the
# checksum.
# @param {[]string} [EXCLUDE_PATHS] List of paths, relative to any `DATA_PATHS`, to exclude from
# the checksum.
validate-checksum:
desc: "Validates the checksum of the given directory matches the checksum in the given file, or
deletes the checksum file otherwise."
Expand All @@ -48,16 +50,19 @@ tasks:
cmds:
- task: "compute-checksum"
vars:
DATA_DIR: "{{.DATA_DIR}}"
DATA_PATHS:
ref: ".DATA_PATHS"
EXCLUDE_PATHS:
ref: "default (list) .EXCLUDE_PATHS"
ref: ".EXCLUDE_PATHS"
OUTPUT_FILE: "{{.TMP_CHECKSUM_FILE}}"
- defer: "rm -f '{{.TMP_CHECKSUM_FILE}}'"
# Check that the directory exists and the checksum matches; otherwise delete the checksum file
- >-
(
test -d "{{.DATA_DIR}}"
&& diff -q '{{.TMP_CHECKSUM_FILE}}' '{{.CHECKSUM_FILE}}' 2> /dev/null
{{- range .DATA_PATHS}}
test -e "{{.}}" &&
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Ignoring the quotes) I probably should've anticipated this, but this won't work for patterns since if a wildcard path glob returns multiple files, test -e will fail because it only expects one argument.

{{- end}}
diff -q '{{.TMP_CHECKSUM_FILE}}' '{{.CHECKSUM_FILE}}' 2> /dev/null
) || rm -f '{{.CHECKSUM_FILE}}'

# ===
Expand Down Expand Up @@ -141,3 +146,159 @@ tasks:
\( -iname "*.cpp" -o -iname "*.h" -o -iname "*.hpp" \) \
-print0 | \
xargs -0 clang-tidy {{.FLAGS}}

# ===
# CMAKE UTILS
# ===

# Runs cmake configure and build steps for the given source and build directories.
#
# @param {string} BUILD_DIR Cmake build directory to create.
# @param {string} SOURCE_DIR Project source directory containing the CMakeLists.txt file.
# @param {string}[optional] CHECKSUM_FILE Path to store the checksum of built files.
# @param {string}[optional] CMAKE_ARGS Any additional arguments to pass to cmake configure.
cmake-build:
label: "cmake-build: {{.SOURCE_DIR}} {{.BUILD_DIR}}"
internal: true
vars:
CHECKSUM_FILE: '{{default (printf "%s.md5" .BUILD_DIR) .CHECKSUM_FILE}}'
CMAKE_ARGS: "{{default nil .CMAKE_ARGS}}"
requires:
vars: ["BUILD_DIR", "SOURCE_DIR"]
sources:
- "{{.SOURCE_DIR}}/**/*"
- exclude: "{{.SOURCE_DIR}}/.cache/**/*"
- exclude: "{{.SOURCE_DIR}}/compile_commands.json"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this generated in the build directory?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'll remove this. It is old code from when I was figuring out task.

You're right, but this line was aimed at a symlink related to my vim linting setup.

generates: ["{{.CHECKSUM_FILE}}"]
deps:
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_PATHS: ["{{.BUILD_DIR}}"]
EXCLUDE_PATHS: ["install_manifest.txt"]
cmds:
- >-
cmake
-S "{{.SOURCE_DIR}}"
-B "{{.BUILD_DIR}}"
{{.CMAKE_ARGS}}
- >-
cmake
--build "{{.BUILD_DIR}}"
--parallel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add an optional arg to control the number of jobs (useful for memory constrained environments)?

# This command must be last
- task: "compute-checksum"
vars:
DATA_PATHS: ["{{.BUILD_DIR}}"]
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
EXCLUDE_PATHS: ["install_manifest.txt"]

# Runs cmake install step for the given build directory.
#
# @param {string} BUILD_DIR Cmake build directory.
# @param {string} INSTALL_PREFIX Path prefix for installing the project.
# @param {string}[optional] CHECKSUM_FILE Path to store the checksum of installed files.
# @param {[]string}[optional] DATA_PATHS Paths to compute the the checksum for. Overrides using
# the install prefix path.
cmake-install:
label: "cmake-install: {{.BUILD_DIR}} {{.INSTALL_PREFIX}}"
internal: true
vars:
CHECKSUM_FILE: '{{default (printf "%s.md5" .INSTALL_PREFIX) .CHECKSUM_FILE}}'
# Convert the install prefix to a single element array for the default case where DATA_PATHS
# is not manually set.
_INSTALL_PREFIX_ARRAY: ["{{.INSTALL_PREFIX}}"]
DATA_PATHS:
ref: "default ._INSTALL_PREFIX_ARRAY .DATA_PATHS"
requires:
vars: ["BUILD_DIR", "INSTALL_PREFIX"]
sources:
- "{{.BUILD_DIR}}/**/*"
- exclude: "{{.BUILD_DIR}}/install_manifest.txt"
generates: ["{{.CHECKSUM_FILE}}"]
deps:
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_PATHS:
ref: ".DATA_PATHS"
cmds:
- >-
cmake
--install "{{.BUILD_DIR}}"
--prefix "{{.INSTALL_PREFIX}}"
# This command must be last
- task: "compute-checksum"
vars:
DATA_PATHS:
ref: ".DATA_PATHS"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"

# ===
# REMOTE UTILS
# ===

# Runs curl to download the provided URL.
#
# @param {string} URL
# @param {string} URL_SHA256 Content hash to verify downloaded file against.
# @param {string}[optional] OUTPUT_FILE File path to store the download file.
curl:
label: "curl: {{.OUTPUT_FILE}}"
internal: true
vars:
OUTPUT_FILE: "{{default (base .URL) .OUTPUT_FILE}}"
requires:
vars: ["URL", "URL_SHA256"]
generates:
- "{{.OUTPUT_FILE}}"
status:
- >-
diff
<(echo "{{.URL_SHA256}}")
<(openssl dgst -sha256 "{{.OUTPUT_FILE}}"
| awk '{print $2}')
cmds:
- |-
mkdir -p "{{dir .OUTPUT_FILE}}"
curl -L "{{.URL}}" -o "{{.OUTPUT_FILE}}"

# Runs curl to download the provided URL and tar to extract its contents.
#
# @param {string} OUTPUT_DIR Path to extract downloaded tar file contents to.
# @param {string} URL
# @param {string} URL_SHA256 Content hash to verify downloaded tar file against.
# @param {string}[optional] CHECKSUM_FILE File path to store the checksum of downloaded tar file.
# @param {int}[optional] STRIP Number of leading components to strip from file names for tar.
# @param {string}[optional] TAR_FILE File path to store the downloaded tar file.
fetch-src:
label: "fetch-src: {{.OUTPUT_DIR}}"
internal: true
vars:
CHECKSUM_FILE: '{{default (printf "%s.md5" .OUTPUT_DIR) .CHECKSUM_FILE}}'
STRIP: "{{default 1 .STRIP}}"
TAR_FILE: '{{default (printf "%s.tar.gz" .OUTPUT_DIR) .TAR_FILE}}'
requires:
vars: ["OUTPUT_DIR", "URL", "URL_SHA256"]
sources: ["{{.TASKFILE}}"]
generates: ["{{.CHECKSUM_FILE}}", "{{.TAR_FILE}}"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to double-check if these will cause issues.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I guess you want .TASKFILE to be a source so that if the way of extracting the file changes, it will be extracted again, right?
  • I think .TARFILE should be a source (rather than a generated file) since it's the generated file of curl.

deps:
- task: "curl"
vars:
URL: "{{.URL}}"
URL_SHA256: "{{.URL_SHA256}}"
OUTPUT_FILE: "{{.TAR_FILE}}"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_PATHS: ["{{.OUTPUT_DIR}}"]
cmds:
- |-
rm -rf "{{.OUTPUT_DIR}}"
mkdir -p "{{.OUTPUT_DIR}}"
tar -x --strip-components="{{.STRIP}}" -C "{{.OUTPUT_DIR}}" -f "{{.TAR_FILE}}"
# This command must be last
- task: "compute-checksum"
vars:
DATA_PATHS: ["{{.OUTPUT_DIR}}"]
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"