Skip to content

Add log upload support for build target failed using teatime.py script #648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
erman-gurses wants to merge 1 commit into from
Closed

Add log upload support for build target failed using teatime.py script #648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 20 additions & 21 deletions .github/workflows/build_linux_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,17 @@ jobs:
strategy:
fail-fast: true
env:
BASE_BUILD_DIR: build
LOG_DIR: build/logs
CACHE_DIR: ${{ github.workspace }}/.container-cache
CCACHE_DIR: "${{ github.workspace }}/.container-cache/ccache"
CCACHE_MAXSIZE: "700M"
AMDGPU_FAMILIES: ${{ inputs.amdgpu_families }}
TEATIME_FORCE_INTERACTIVE: 0
TEATIME_S3_UPLOAD: "1"
TEATIME_S3_BUCKET: "therock-artifacts"
TEATIME_S3_SUBDIR: "${{ github.run_id }}-linux/logs/${{ inputs.amdgpu_families }}"
TEATIME_FAIL_ON_UPLOAD_ERROR: "1"
BUCKET: ${{ github.event.repository.name == 'TheRock' && 'therock-artifacts' || 'therock-artifacts-external' }}
steps:
- name: "Checking out repository"
Expand Down Expand Up @@ -91,9 +97,18 @@ jobs:
run: |
./build_tools/fetch_sources.py --jobs 12

- name: Install python deps
- name: Install python and dev deps
run: |
pip install -r requirements.txt
pip install -r requirements-dev.txt

- name: Configure AWS Credentials
if: always()
uses: aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 # v4.1.0
Comment on lines +105 to +107
Copy link
Copy Markdown
Member

@ScottTodd ScottTodd May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@geomin12 we're getting a lot of these if: always() and if: ${{ !cancelled() }} conditions in these workflows. Let's see if we can restructure the flow a bit through migrating to more scripts such that the flow is easier to understand.

Here in particular, if fetching sources fails, we'd still configure AWS credentials and try to upload logs, but we wouldn't try to build. In that case I would prefer for the workflow to fail fast and not continue to try cleanup steps that will fail.

with:
aws-region: us-east-2
role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts
role-duration-seconds: 14400 # 4 hours

- name: Build Projects
id: build
Expand Down Expand Up @@ -135,24 +150,10 @@ jobs:
echo "-------------"
ccache -s

- name: Configure AWS Credentials
if: always()
uses: aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 # v4.1.0
with:
aws-region: us-east-2
role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts

# TODO: Move to script
- name: Create Index Files
- name: Create Artifact Index
if: always()
run: |
curl --silent --fail --show-error --location \
https://raw.githubusercontent.com/joshbrunty/Indexer/6d8cbfd15d3853b482e6a49f2d875ded9188b721/indexer.py \
--output build/indexer.py
python build/indexer.py -f '*.tar.xz*' build/artifacts/
python3 build_tools/create_log_index.py \
--build-dir=build \
--amdgpu-family=${{ env.AMDGPU_FAMILIES }}
python third-party/indexer/indexer.py -f '*.tar.xz*' ${{ env.BASE_BUILD_DIR }}/artifacts/

# TODO: Move to script
- name: Upload Artifacts
Expand All @@ -163,12 +164,10 @@ jobs:
--include "*.tar.xz*"
aws s3 cp build/artifacts/index.html s3://${{env.BUCKET}}/${{github.run_id}}-linux/index-${{env.AMDGPU_FAMILIES}}.html

- name: Upload Logs
- name: Index and Upload Logs
if: always()
run: |
python3 build_tools/upload_logs_to_s3.py \
--build-dir=build \
--s3-base-path="s3://${{env.BUCKET}}/${{github.run_id}}-linux/logs/${{env.AMDGPU_FAMILIES}}"
python3 build_tools/teatime.py --label "index_and_upload" --log-timestamps build/logs/temp.log

- name: Add Links to Job Summary
if: always()
Expand Down
53 changes: 25 additions & 28 deletions .github/workflows/build_windows_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,23 @@ jobs:
strategy:
fail-fast: true
env:
BASE_BUILD_DIR_POWERSHELL: B:\tmpbuild
# Base build dir for Windows
BASE_BUILD_DIR: B:\tmpbuild
LOG_DIR: B:\tmpbuild\logs
CACHE_DIR: "${{github.workspace}}/.cache"
CCACHE_DIR: "${{github.workspace}}/.cache/ccache"
CCACHE_MAXSIZE: "4000M"
TEATIME_FORCE_INTERACTIVE: 0
AMDGPU_FAMILIES: ${{ inputs.amdgpu_families }}
TEATIME_S3_UPLOAD: "1"
TEATIME_S3_BUCKET: "therock-artifacts"
TEATIME_S3_SUBDIR: "${{ github.run_id }}-windows/logs/${{ inputs.amdgpu_families }}"
TEATIME_FAIL_ON_UPLOAD_ERROR: "1"
steps:
- name: "Create build dir"
shell: powershell
run: |
$buildDir = "$env:BASE_BUILD_DIR_POWERSHELL\"
$buildDir = "$env:BASE_BUILD_DIR\"
echo "BUILD_DIR_POWERSHELL=$buildDir" >> $env:GITHUB_ENV
mkdir "$buildDir"
Write-Host "Generated Build Directory: $buildDir"
Expand All @@ -67,9 +73,10 @@ jobs:
with:
python-version: "3.12"

- name: Install python deps
- name: Install python and dev deps
run: |
pip install -r requirements.txt
pip install -r requirements-dev.txt

- name: Install requirements
run: |
Expand Down Expand Up @@ -133,6 +140,14 @@ jobs:
path: amdgpu-windows-interop
lfs: true

- name: Configure AWS Credentials
if: always()
uses: aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 # v4.1.0
with:
aws-region: us-east-2
role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts
role-duration-seconds: 14400 # 4 hours

- name: Configure Projects
run: |
# clear cache before build and after download
Expand Down Expand Up @@ -189,52 +204,34 @@ jobs:
$fsout | % {$_.Used/=1GB;$_.Free/=1GB;$_} | Write-Host
get-disk | Select-object @{Name="Size(GB)";Expression={$_.Size/1GB}} | Write-Host

- name: Configure AWS Credentials
if: always()
uses: aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 # v4.1.0
with:
aws-region: us-east-2
role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts

# TODO: Move to script
- name: Create Index Files
- name: Create Artifact Index
if: always()
run: |
curl --silent --fail --show-error --location \
https://raw.githubusercontent.com/joshbrunty/Indexer/6d8cbfd15d3853b482e6a49f2d875ded9188b721/indexer.py \
--output ${{ env.BUILD_DIR_BASH }}/indexer.py
python ${{ env.BUILD_DIR_BASH }}/indexer.py -f '*.tar.xz*' ${{ env.BUILD_DIR_BASH }}/artifacts/
python build_tools/create_log_index.py \
--build-dir=${{ env.BUILD_DIR_BASH }} \
--amdgpu-family=${{ env.AMDGPU_FAMILIES }}
python third-party/indexer/indexer.py -f '*.tar.xz*' ${{ env.BUILD_DIR_BASH }}/artifacts/

# TODO: Move to script
- name: Upload Artifacts
shell: powershell
run: |
$Env:PATH += ";C:\Program Files\Amazon\AWSCLIV2"
aws s3 cp ${{ env.BASE_BUILD_DIR_POWERSHELL }}\artifacts s3://therock-artifacts/${{github.run_id}}-windows/ `
aws s3 cp ${{ env.BASE_BUILD_DIR }}\artifacts s3://therock-artifacts/${{github.run_id}}-windows/ `
--recursive --no-follow-symlinks `
--exclude "*" `
--include "*.tar.xz*"
aws s3 cp ${{ env.BASE_BUILD_DIR_POWERSHELL }}\artifacts\index.html s3://therock-artifacts/${{github.run_id}}-windows/index-${{env.AMDGPU_FAMILIES}}.html
aws s3 cp ${{ env.BASE_BUILD_DIR }}\artifacts\index.html s3://therock-artifacts/${{github.run_id}}-windows/index-${{env.AMDGPU_FAMILIES}}.html

- name: Upload Logs
- name: Index and Upload Logs
if: always()
shell: powershell
run: |
$Env:PATH += ";C:\Program Files\Amazon\AWSCLIV2"
python3 build_tools/upload_logs_to_s3.py `
--build-dir=${{ env.BASE_BUILD_DIR_POWERSHELL }} `
--s3-base-path="s3://therock-artifacts/${{github.run_id}}-windows/logs/${{env.AMDGPU_FAMILIES}}"
python3 build_tools/teatime.py --label "index_and_upload" --log-timestamps build/logs/temp.log

- name: Add Links to Job Summary
if: always()
run: |
LOG_URL="https://therock-artifacts.s3.us-east-2.amazonaws.com/${{github.run_id}}-windows/logs/${{env.AMDGPU_FAMILIES}}/index.html"
echo "[Build Logs](${LOG_URL})" >> $GITHUB_STEP_SUMMARY

ARTIFACT_INDEX="${{ env.BASE_BUILD_DIR_POWERSHELL }}/artifacts/index.html"
ARTIFACT_INDEX="${{ env.BASE_BUILD_DIR }}/artifacts/index.html"
if [ -f "${ARTIFACT_INDEX}" ]; then
ARTIFACT_URL="https://therock-artifacts.s3.us-east-2.amazonaws.com/${{github.run_id}}-windows/index-${{env.AMDGPU_FAMILIES}}.html"

Expand Down
57 changes: 40 additions & 17 deletions build_tools/create_log_index.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,41 @@ def normalize_path(p: Path) -> str:
return str(p).replace("\\", "/") if is_windows() else str(p)


def index_log_files(build_dir: Path, amdgpu_family: str):
log_dir = build_dir / "logs"
def get_indexer_path() -> Path:
"""
Resolve to the local third-party/indexer/indexer.py copy.
"""
indexer_path = (
Path(__file__).resolve().parent.parent / "third-party/indexer/indexer.py"
)
if indexer_path.is_file():
log(f"[INFO] Using bundled indexer.py: {indexer_path}")
return indexer_path
else:
log(f"[ERROR] Bundled indexer.py not found at: {indexer_path}")
sys.exit(2)
Comment on lines +24 to +36
Copy link
Copy Markdown
Member

@ScottTodd ScottTodd May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The else clause should never be hit. I would drop the error handling here.



def index_log_files(log_dir: Path, amdgpu_family: str):
index_file = log_dir / "index.html"

# TODO: Fork indexer.py locally to avoid relying on an external GitHub source at runtime.
indexer_path = build_dir / "indexer.py"
if not log_dir.is_dir():
log(f"[WARN] Log directory '{log_dir}' not found. Skipping indexing.")
return

indexer_path = get_indexer_path()
log(
f"[INFO] Found '{log_dir}' directory. Indexing '*.log' files using indexer: {indexer_path}"
)

if log_dir.is_dir():
log(f"[INFO] Found '{log_dir}' directory. Indexing '*.log' files...")
try:
subprocess.run(
["python", str(indexer_path), "-f", "*.log", normalize_path(log_dir)],
[sys.executable, str(indexer_path), "-f", "*.log", normalize_path(log_dir)],
check=True,
)
else:
log(f"[WARN] Log directory '{log_dir}' not found. Skipping indexing.")
return
except subprocess.CalledProcessError as e:
log(f"[ERROR] Failed to run indexer.py: {e}")
sys.exit(2)

if index_file.exists():
log(
Expand All @@ -46,19 +65,23 @@ def index_log_files(build_dir: Path, amdgpu_family: str):
updated = content.replace(
'a href=".."', f'a href="../../index-{amdgpu_family}.html"'
)
index_file.write_text(updated)

# Ensure full write and flush to disk
with open(index_file, "w", encoding="utf-8") as f:
f.write(updated)
f.flush()
os.fsync(f.fileno())

log("[INFO] Log index links updated.")
else:
log(f"[WARN] '{index_file}' not found. Skipping link rewrite.")


if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Create HTML index for log files.")
parser.add_argument(
"--build-dir",
"--log-dir",
type=Path,
default=Path(os.getenv("BUILD_DIR", "build")),
help="Build directory containing logs (default: 'build' or $BUILD_DIR)",
default=Path(os.getenv("LOG_DIR", "build/logs")),
help="Directory containing log files (default: 'build/logs' or $LOG_DIR)",
)
parser.add_argument(
"--amdgpu-family",
Expand All @@ -72,4 +95,4 @@ def index_log_files(build_dir: Path, amdgpu_family: str):
log("[ERROR] --amdgpu-family not provided and AMDGPU_FAMILIES env var not set")
sys.exit(1)

index_log_files(args.build_dir, args.amdgpu_family)
index_log_files(args.log_dir, args.amdgpu_family)
Loading
Loading