Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expiring assumed role token after 6 hours when using FargateCluster with max duration set to 12 hours #422

Open
JavrelWork opened this issue Jan 11, 2024 · 0 comments

Comments

@JavrelWork
Copy link

Describe the issue:
I was advised on SO to post here, not sure its a bug, I may just not know something...

Im running dask locally and spinning up fargate cluster in AWS.
Its assuming a role correctly and starting up the cluster.
After six hours its permissions fail, I have set the maximum session duration on the role to 12 hours but it seems to be getting a defailt 6 hour token anyway and not asking for a 12 hour token.

Is there a way to instruct it to get a 12 hour token or do I need to refresh the session token somehow?

Minimal Complete Verifiable Example:
Cant post a full Minimal Complete Verifiable Example, would need something that goes over 6 hrs.
But our cluster is setup like:

cluster = FargateCluster(
        cloudwatch_logs_group=f"{project}_log_group",
        cluster_name_template=f"{project}-{uuid}",
        environment={"DEFAULT_GSF_VERSION": gsf_version},
        image=f"{aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/{project}:latest",
        n_workers=9,
        scheduler_cpu=4096,
        scheduler_mem=30720,
        scheduler_timeout="240 minutes",
        security_groups=[aws_security_group_id],
        task_role_arn=f"arn:aws:iam::{aws_account_id}:role/{project}ECSTaskRole",
        worker_cpu=4096,
        worker_mem=30720,
        worker_extra_args=algorithm_extra_cluster_args,
        fargate_use_private_ip=use_private_ip,
        vpc=vpc,
        subnets=subnets,
        skip_cleanup=True,
        platform_version="LATEST",
        shutdown_on_close=True,
        mount_points=mount_points,
        volumes=volumes,
    )
    cluster.adapt(minimum=9, maximum=80)
    logger.info(f"dashboard_link is: {cluster.dashboard_link}")

Anything else we need to know?:

I have set the maximum session duration on my role to 43200(12 hours), seems it still only gets a 6 hour token.
I tried adding duration_seconds=43200 to AWS config files hoping to override the default 6 hour token, still lost perms after 6 hours

Environment:

  • Dask version:022.12.0
  • Python version:3.8.3
  • Operating System:linux
  • Install method (conda, pip, source):poetry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant