Working example of Github OIDC and using Session Tags in IAM policies? #419
Comments
|
Correct me if I'm wrong, but most of the examples I found do a double assume role. E.g., the first Assumed Role can only be assumed by the Github OIDC prepared case, and that assumed role only has one permission and that is to allow a second role to be assumed AND at that point aws-actions/configure-aws-credentails can be used again. And it does NOT skip tagging when assuming the second role. And of course this does work! However, there is nothing preventing anyone inside a github org / repo / branch from cloning the aws-actions/configure-aws-credentials code into their own code base (or similar code), and then using it as a local 'action' and then subverting that code to use a different github org/repo/branch and in essence redirect the security constraints on the second assume role. The rogue code can determine what values to give the session tags at the time the second role is assumed. I know, it is a lot of 'what ifs'. However, in a security conscience environment you don't want any of those what ifs to come to pass, so you ensure that they can not. That is why Github OIDC w/ the Cloud provider is so great. It establishes w/o question the trusted entity. The issue is that it is weakly defined, thus leading to work arounds that could subvert the security that was intended. Again, the AWS Session Tags needs to be accomplished w/ the Github OIDC is executed. For example this invokes the Github OIDC to generate a JWT: The returned JWT might look like this: (in fact this is exactly what I got back from the above curl, w/ most values changed for security purposes in this thread) and ideally, since I specified or something similar, that meets the contract for AWS IAM roles to establish BOTH the trust for the role, as well as to utilize the session tags in IAM policies associated w/ the role. This would prevent having two roles AND would allow one to create an AWS IAM role that is both assumable by Github actions AND contains exact information in the session tags to allow complex policies to be created and governed by the values of the aws:PrincipalTag. E.g., I can have policies that are specific to a github org, or repo or branch or workflow. This really provides very powerful constructs to allow the ease of workflows in Github and utilizing AWS Resources for CI/CD. 🤷 Again, I'm probably uneducated and need more education to understand what the expected approach is. |
|
I'm not sure if this helps you, but the example in the README works fine if you add a |
|
@manics, yes I've accomplished that part. But if you notice there are no 'policies'. They are missing. That is the part that I want to add. And more importantly I want to add them to the role that can be assumed by aws-actions/configure-aws-credentials. Not a two role combination. However, because the Github OIDC is not implemented correctly (as to my understanding ) then you can not make policies in that role w/o making them very, very specific to either the github org or the repo/etc. E.g., I have 1600+ repos spread over a number of github organizations. Meaning I would have a large number of roles. The ideal is what AWS describes in that if the following (or something like it) is provided by Github OIDC: then a very few roles can be defined AND have very specific policies that include the Github provided information. Information that can not be spoofed. For example. Suppose I have an S3 bucket that we use for 'dynamic configuration'. And further suppose that Github OIDC provided the AWS Session Tags as described. Then I could add the following policy statement to the Role: If you'll notice, I'm able to use the This is the power of Github OIDC and AWS, when implemented correctly. |
|
Another thought. If when the Github OIDC is generated the '<GITHUB_ACTION>' could be identified and added to the AWS Session Tags, then this would allow another layer of trust. E.g., I'm assuming a lot about actions. In that if actions are sourced from github repos, and all of the security that github org/repos imply, then I have one additional layer of trust that can be established. In that I can make roles that only trust aws-actions/configure-aws-credentials. Or if I really want to go to the work of writing my own action. I can place that action in a github org/repo that only 3 people have access to, thus limiting the possible attack vectors. Again, this too would be quite powerful in the Github OIDC and AWS combination. |
|
I will chime in with my use case, because I am facing similar issues. I have a working OIDC connection to a single role in one of my AWS accounts, but I have over 100 accounts to set up for OIDC. This connection works and access can be controlled by allowing each repo in the IAM role trust policy. I was planning to create an OIDC provider and role in each account, but quickly found that this will not easily scale. I would like to use a single account in our network as the entry point for OIDC connections, and then that session will assume another role in the target account. This is currently documented and working, however, in these secondary assumed roles, in the target aws accounts, I want to be able to have a condition which checks which repo the request is coming from in the IAM trust policy. It appears this should be possible with Session Tagging, but like @roskelleycj has been demonstrating, no session tags are being set on the OIDC session. The documentation for this action appears to indicate that that Session Tags are supported for OIDC sessions, but it is unclear if that is actually the case: Is this a deficency in GitHub's JWT implementation? or something to fix in this action? Its starting to look like this is required on the GitHub side and is not supported... |
|
Looks like @glassechidna ran into a similar issue and made a novel workaround. Although, his approach to the problem is different than mine. In that he wants AWS to change their STS implementation. 🤷 I figured since AWS documented what their approach AND Github OIDC shows that you must provide the |
|
Hi @roskelleycj, I am facing the same challenge. If these custom claims were included in the JWT generated by Github OIDC it would be the missing piece to writing powerful and secure IAM policies. I am now faced with the choice between permissive single policies shared by many GH repos or creating specific policies for each repo. Have you had any traction with Github on this? |
|
Unfortunately the fact that the tags must be in the JWT makes this a GitHub issue more than an AWS issue. AFAIK, the only thing that GitHub lets consumers modify about the JWT is the Audience. Otherwise it's entirely immutable. So either GitHub would need to allow appending extra claims or AWS would need to add session tag mapping to OIDC providers. |
|
You nailed it @RichiCoder1, this needs to be defined in the JWT because AssumeRoleWithWebIdentity() lacks a tags parameter like AssumeRole() does. Meaning AWS doesn't provide session tag mapping for OIDC directly. However, I'm unsure how or if there is a way to modify the JWT from GitHub in the necessary fashion. If anyone knows a way to do this, please leave a comment 🙂 |
peterwoodworth
added
service-limitation
|
Does this announcement remove the service limitation? |
|
Unfortunately no. That just allows customizing the subject claim, while this ticket requires adding additional custom claims. |
|
I too am running into the same issue. I want to be able to write an OIDC policy that will allow each repo to write to its own folder in a bucket. We have over 8k repos so not really an option to have a role for each repo. The claims are there in the JWT, but AWS doesnt let you access them directly This could be fixed from either end really. Being able to use the claims or the tags would be very powerful for a lot of things. Is anyone aware if GH are working on this? |
|
As mentioned this 'novel hack' appears to continue to work well, with several key modifications to the Lambda Function to limit the blast radius. For example the job_workflow_ref claim can be used to control what org/repo/.github/workflow/.yaml has access to the IAM Role. Additionally, this works well as you can put all of this in a central AWS Account using APIGateway and you can do STS Assume Role to the various accounts that need to be accessed. It would be much simpler if either GitHub or AWS or BOTH worked together to find a solution to allows the AssumeRoleWithWebIdentity to contain tags based upon the JWT. Even if the only tag was the One note about the novel hack, sometimes APIGateway and GitHub Actions do not agree about the time that the JWT was issued. (E.g., it looks like GitHub's clocks are in the future occasionally) and AWS APIGateway rejects the request out of hand. 😞 a jitter has to be applied OR a few retries have to be applied, thus making it slower too. |
|
@roskelleycj I am the author of the novel hack. I wrote it in a rush shortly before GHA officially launched OIDC support, assuming (hoping) that better integration would arrive sooner rather than later. It seems that wish hasn't come true yet. I didn't realise anyone was actually using it. A few months later (the start of this year) I wrote a follow up solution that I've since been using. I built it on the assumption that a) a more general solution might be more useful and b) it might be preferable to still use OIDC providers in AWS IAM. I originally wanted to include every claim in Github's JWT, but role session tags are limited to 500 bytes 😢
|
|
@aidansteele I would also like to see that feature! I find it so exhausting opening tickets with tech support for feature requests. Same response every time, then feels like it goes into a black hole. No visibility, no updates. |
That's not necessarily true: when you file a request with the Technical Support button in the console, you're sending that request over to the AWS Premium Support team. Premium Support will then send that ticket over to the responsible AWS service team as a feature request. Likewise, when you file a feature request that's applicable to a service in GitHub, our team will create a ticket and send it to the same AWS service team. Many teams at AWS do prioritize feature development based on which customers need it, but the requests end up in the same bucket for evaluation most of the time. It is very true that we use demand to prioritize work though. Number of customers that submit a ticket through AWS PS is one data point that teams use to evaluate priority, but another one is the number of people that have a +1 reaction on a GitHub issue (you can take a look at some of the feature requests in the AWS SDKs on GitHub and see all the +1s we have), so we encourage people to use GitHub reactions as well :) Looking at the blog post by @aidansteele, it looks like you're asking for two different behavior changes service side: IAM role trust policies should have a permissions boundary, and a way to map arbitrary claims from the OIDC token to role session tags. Cool! We'll forward those over to the correct teams. There's still a little bit of research that the team that develops this action needs to do around this subject, so I'm leaving this open while we do that, but I wanted to let @aidansteele know that we read the blog post and generated some action items from it for the responsible teams. |
|
@kellertk Thank you so much for the detailed correction. It's really great to hear that these requests aren't "disappearing into the void". I'll be sure to 👍 more feature requests when I see them on Github, I didn't realise they carried so much weight. Specific to this topic: it's also great to hear that the service teams are aware of the requests. I can only imagine how much work is involved in making fundamental changes like that to such a sensitive system. |
|
@kellertk Until the tagging issues above are addressed, it would be great to get the README updated because it is currently very misleading. Specifically, https://github.com/aws-actions/configure-aws-credentials#session-tagging should clarify that tags are not usable when the action is configured for AssumeRoleWithWebIdentity. Thank you. |
|
I have same issue. We have ~100 github repositories that push docker images to ecr. I did not want to create roles for every repo. I worked around with 2 roles and assume chain. First role: Get credentials with assumeRoleWithWeb identity, second role to tag session with repo name. With this chain I am able to use |
|
@icaliskanoglu Thanks for sharing that. Would you be willing to share the policies/roles you are using for the two-step role chaining? |
|
@icaliskanoglu What is the role trust policy that you have for the first and second role? As well as the permission policy for the first role? From my understanding, you need to allow sts:TagSession to pass tags when you assume the second role. However, it is not clear how you would restrict this. If sts:TagSession is unrestricted, and the permission policy in the second role trusts the PrincipalTag, it might allow a malicious actor to call sts directly and inject whatever tags they want into the second session. |
|
So there are no way to achieve this without having to do a double assume role ? Is there a way to have a generic first role to use everywhere and then the on the second assumerole to filter depending on which repository it ran by, to which final role it can go ? |
@kedare Yes, add a condition on the trust policy of the second role. For example, if you have a generic first role called {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/generic-gh-role"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:PrincipalTag/Repository": "my-org/my-repo"
}
}
}
]
}But this doesn't avoid the security issue mentioned by @roskelleycj in #419 (comment) where a malicious actor could use a fork of this action to override these tags. The only proper fix is:
|
|
@kedare There's an additional solution that's not been mentioned so far: After #739, you can declare an We use this to define a Reusable Workflow that declares an Next, we use a customised Finally, we leverage "Approving workflow runs from private forks" to ensure the -- We've found this approach to be "as close as we can get" to an IAM native solution, while still retaining clear escape paths once either aws-actions/configure-aws-credentials#419 (this issue!) or community/discussions#12031 are resolved. We've internally +1'd this issue with our AWS and GitHub representatives, but would encourage interested community members to do the same. Hope that helps 😃 |
|
@unlobito would be great with an example for all the steps. Cause I have a hard time piecing the docs together without a concrete example of how to achieve within a GitHub Reusable workflow. Perhaps share an example of identity provider policy json document and an example workflow. I was baffled that reusable workflow OIDC is not usable together with AWS STS 😓 I have Google workload identity working with reusable workflows no problem... |
|
@unlobito that's a cool idea! What policy is required for the role being assumed? Does it act as a sort of permissions boundary for the |
|
@vixus0 That's exactly right- the IAM policy being assumed mostly has wildcard permissions across resources on the account, but the IAM role's policy document enforces our Using wildcards is non-ideal, but we expect we can get rid of the @jeterson I'm still on annual leave for a few more days, but hopefully the info above is useful in the meantime. I'll try to get approval for releasing samples of our deployed IAM policy + Reusable Workflow once I'm back at work though 😊 |
|
As promised, we've just released a sample of our setup for enforcing per-repo privileges through https://github.com/Skyscanner/gha-aws-oidc-sample Unfortunately, this doesn't fully solve the issue we're currently on (since session tags remain unavailable until either GitHub or AWS update their OIDC implementations), but does deliver on identifying individual repositories in CloudTrail through I'd still recommend +1ing this issue with your AWS and GitHub representatives, but hope this is useful as a workaround in the meantime :) This wouldn't be complete without a massive "Thank You" to my Skyscanner colleagues for help with design and review- shout outs to my …and of course, thank you (the community!) for all of your notes on this issue + community/discussions#12031 💚 |
|
This is all doable using Cognito Identity as has been mentioned in a separate issue, but it 's not something that has been well documented/tested. I spent some time to do such and decided to create my first open source contribution related to this. Maybe some of you can find something that helps you out over at https://github.com/catnekaise and help test/review this. Details of how we would use Cognito Identity in GitHub Actions to enable ABAC can be found here. |
|
Great and thorough work, @djonser 👏 I wasn't aware of the possibility of using Cognito for this. I've taken a quick look at your repositories and website, and I'll be trying this out! Thank you for sharing code, examples and documentation with the community - very much appreciated. |
That seems straight forward enough, could possibly create a CDK/CF component to simplify standing that up |
Since I do enjoy AWS CDK quite a bit myself I created constructs (and ABAC-utilities) for this use-case in catnekaise/actions-constructs. If you end up trying it out then please provide any feedback you can, since this topic we are discussing here is quite niche even before we involved CDK :). |
That looks awesome! Love the claims->role mapping and the utils on top of role permission assignment. |
|
Amazing work, @djonser. Both on finding this in the first place and a great write up! You have made my whole week 😁 I wrote a blog post with a sample CloudFormation excerpt and posted it here: https://awsteele.com/blog/2023/10/25/aws-role-session-tags-for-github-actions.html Have you discovered any shortcomings to the Cognito approach? Do you prefer the classic or enhanced flow? I wrote in my post that I think I prefer the classic flow, but it's all still very new to me. |
I'm really glad its helping you out!. Also thanks for the signal-boost, more than I could have expected :). The only technical shortcoming I ran into with Cognito Identity was the ambiguity in Session Limits and "session tags into a packed binary format that has a separate limit" without being able to find more info about this limit. I wrote more about it here and its all relates to I believe the Basic (Classic) AuthFlow is a good default but the Enhanced (Simplified) AuthFlow is an interesting building-block that I will have to do some more write-up on. I wrote a little bit about the AuthFlow differences here, but I labeled the differences as traits and kept use-cases out. |
|
Hi, I am just learning about Github - AWS OIDC therefore I may be a bit confused or missed something important. Here is my use case: Originally I planned one role per service. flowchart
gha[GithubAction]-->|OIDC assume web identity| jump[jump account];
jump-->|assume role| workload[workload account]
but based on the reading and the discussion it may get out of hand at scale. If I understand everything mentioned here and IAM/STS documentation correctly. With the Cognito solution. I am not 100% sure how the tags get propagated if anyone could point me in the right direction? I am going to setup a test environment to play around. |
|
In your use case where you want a single jump-account for GitHub Actions OIDC authentication before proceeding into the workload accounts (role-chaining), Cognito Identity (ABAC) can help so that you only require a single role (lets call it entry-role) in the jump-account that then allows trust policies in the workload accounts to reference this single entry-role and its claims (principalTags). You can see a reference example of such a trust policy and more here. Having assumed this first role via Cognito Identity, you can then role-chain and bring with you all the GitHub Actions claims (principalTags) to the workload account when you assume a role there. This will allow you to use the IAM permissions from your example. A very important part in this will be to include the following in the trust policies of the roles in the workload accounts. This makes sure that the workflow in GitHub Actions cannot change the value of repository during role-chaining. This should be done for each claim/principalTag you intend to pass/propagate. {
"Condition": {
"StringEquals": {
"aws:requestTag/repository": "${aws:principalTag/repository}"
}
}
} Propagating tagsTags do not get propagated automatically, you have to set the session tags explicitly. After you have received credentials via Cognito Identity you would use these credentials and it would look like this using the AWS-cli directly: aws sts assume-role \
--role-arn arn:aws:iam::222222222222:role/workload-role \
--role-session-name chained \
--tags Key=repository,Value='${{ github.repository }}' Key=actor,Value='${{ github.actor }}' # Additional claims passed hereUsing Putting it all together.For this example, jump-account has ID Jump-Account
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Resource": "arn:aws:iam::222222222222:role/workload-role"
}
]
}Workload AccountThis is the minimum you need to do for the trust policy. You should consider validating other principalTags to narrow which repository, environment, workflow, etc is allowed to assume this role. In the example below I used {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:role/service-role/cognito-gha"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Condition": {
"StringEquals": {
"aws:requestTag/repository": "${aws:principalTag/repository}",
"aws:requestTag/actor": "${aws:principalTag/actor}",
"aws:requestTag/commit": "${aws:principalTag/sha}"
},
"StringLike": {
"aws:principalTag/repository": "MY_ORG_NAME/*"
}
}
}
]
}GitHub Actions Workflowname: Role Chain
on:
workflow_dispatch:
jobs:
job1:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
# Either use this action directly or use it as a template (copy the action.yml)
- name: Authenticate using Cognito Identity
uses: catnekaise/cognito-idpool-basic-auth@alpha
with:
cognito-identity-pool-id: "eu-west-1:11111111-example"
set-in-environment: true
role-arn: "arn:aws:iam::111111111111:role/service-role/cognito-gha"
aws-region: "eu-west-1"
aws-account-id: "111111111111"
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: "arn:aws:iam::222222222222:role/workload-role"
aws-region: "eu-west-1"
role-chaining: true
- name: Use Credentials
run: |
aws sts get-caller-identityI hope this helps you out and points you in the right direction. |
|
@djonser Thank you for the outstanding detail! |
|
It's not quite the same, but I've found the following to work: Only allows restrictions on name of the repo, but for our requirements it works fine |
Hi Jason, I just read your content, this is exactly what I am trying to do the same but OIDC with 1 account works fine but after assuming the role to another account, the authentication does not work. |
and others
I'm trying to figure out how one would achieve using Github's OIDC w/ AWS AssumeRoleWithWebIdentity.
I can get Github OIDC and aws-actions/configure-aws-credentials to work really well straight out of the box and as advertised. 🎉 (And thank you for making it easy.)
However, when I try to do more complex conditions in IAM policies it seems to fall apart.
By more complex I mean to use Session Tags. E.g.,
aws:PrincipalTag:Repositorywhich seems to be documented to work. However, this rather cryptic message is also present:And in fact when you inspect the code those 'documented session tags' are deleted when using a WebIdentity or webIdentityTokenFile.
And the document seems to indicate that if you need 'Session Tags' you need to look to the Github OIDC for providing those. So how does one do that? How do you provide the expected AWS Session Tag contract in the JWT? When all that you can use to configure Github OIDC is this:
There does appear to be one other poor soul that has bumped into the same problem.
Is anyone willing and able to educate the uneducated? Thank you for your kindness, in advance. 😊
The text was updated successfully, but these errors were encountered: