Skip to content

Conversation

@yongwww
Copy link
Member

@yongwww yongwww commented Jan 19, 2024

update the images to align with the latest upgrades of emsdk and nodejs.

@tqchen
Copy link
Member

tqchen commented Jan 19, 2024

cc @Hzfengsy

@yongwww yongwww force-pushed the update_images branch 4 times, most recently from e7edc07 to 243e9dd Compare January 22, 2024 17:52
@yongwww
Copy link
Member Author

yongwww commented Jan 23, 2024

The CI failed during sccache --start-server with s3 bucket, it was introduced by the sccache upgrade before #16366. Seems we can set env. variables SCCACHE_BUCKET, SCCACHE_REGION, and AWS_PROFILE(or AWS_ACCESS_KEY_ID plus AWS_SECRET_ACCESS_KEY instead of AWS_PROFILE+.aws/credentials) as mentioned in sccache s3

@yongwww yongwww changed the title [CI] Update image tag to 20240119-070113-614a7a9e3 [WIP][CI] Update image tag to 20240119-070113-614a7a9e3 Jan 23, 2024
@yongwww yongwww changed the title [WIP][CI] Update image tag to 20240119-070113-614a7a9e3 [CI] Update image tag to 20240126-070121-8ade9c30e Jan 27, 2024
@yongwww
Copy link
Member Author

yongwww commented Jan 27, 2024

updated to the newer image (tag:20240126-070121-8ade9c30e), it was built on Jan 26. CI is green now.

Actually the current CI doesn’t really use the s3 bucket tvm-sccache-prod for sccache. Instead, the local sccahe cache is used. I have a testing pr to hardcode an invalid bucket and CI still passed. The sccache 0.3.3 doesn’t verify the aws credential during the —start-server, whereas the newer sccache 0.7.x does. In this PR, I changed it to use local cache if no aws credentials is available.

I conducted a benchmark comparing sccache with and without an S3 bucket on an AWS g5.4xlarge instance. The results showed that the build time without the S3 bucket (using a local cache, with an average build time of 219.48 seconds) is faster compared to using an S3 bucket (232.01 seconds average build time). Given these findings, I recommend not enabling the S3 bucket for sccache at this time.

Regarding AWS credentials, they are currently not passed to the worker container. One method is to pass AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY through the container as shown here. However, this approach raises security concerns. We should explore alternatives like using IAM roles, task IAMs, or similar solutions for future needs, especially if we decide to enable S3 for sccache. Useful references include discussions on the best practices for passing AWS credentials to Docker containers (Stack Overflow) and information about IAM roles for Amazon EC2 (AWS EC2 User Guide) and task IAM roles (AWS ECS Developer Guide).

cc: @tqchen @vinx13 @ajtulloch @naut-brian

@tqchen tqchen merged commit a15fd1d into apache:main Jan 27, 2024
@yongwww yongwww deleted the update_images branch January 27, 2024 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants