ERROR: AADSTS700024: Client assertion is not within its valid time range #372
Comments
|
Hi @krukowskid , could you provide the workflow file, run it again with debug mode, and provide the debug log? |
|
same issue here this is a real pain. The token are only valid for 5 minutes, and if you don't use it until very far in your workflow, then it just throw the error shown by OP I tried azure/[email protected] same issue. I'm not using any other way to login into azure. |
Hi @benjamin-rousseau-shift could you provide your workflow file and debug log? Do you also use OIDC login? OIDC login with SP should have an expiration of 1 hour and OIDC with User-assigned managed identity should have 24 hours. |
|
I will try to give you that , I am using OIDC with a service principal using federated credentials. |
|
here is my workflow definition (its reusable workflow). I have also enabled debug but it doesnt make sense to paste it here because it's so noisy. Workflow is failing in reusable workflow definitionname: 'reusable/run-tests'
on:
workflow_call:
inputs:
environment:
required: true
type: string
system-under-test:
required: false
type: string
default: xwow
test-configuration:
required: true
type: string
tests-filter:
description: 'Filter for selecting tests to run'
required: true
type: string
tests-web-url:
required: false
type: string
tests-apigateway-url:
required: false
type: string
report-name:
description: 'Name for execution report and attachments'
required: false
default: Default
type: string
allure-reports:
required: false
default: false
type: boolean
allure-project-id:
required: false
type: string
secrets:
KrukowskidBotAppId:
required: false
KrukowskidBotPrivateKey:
required: false
ad-username:
required: false
ad-password:
required: false
azure-client-id:
required: false
azure-tenant-id:
required: false
azure-subscription-id:
required: false
identity-url:
required: false
identity-client-id:
required: false
backoffice-identity-url:
required: false
backoffice-client-id:
required: false
backoffice-client-secret:
required: false
backoffice-identity-scope:
required: false
allure-server-password:
required: false
permissions:
id-token: write
contents: write
actions: read
checks: write
jobs:
run-tests:
name: run-tests
environment: ${{ inputs.environment }}
runs-on:
labels: ubuntu-latest-8core32ram
timeout-minutes: 20
env:
E2E-ENVIRONMENT: ${{ inputs.test-configuration }}
E2E-SUT: ${{ inputs.system-under-test }}
ALLURE_SERVER_URL: ${{ vars.ALLURE_SERVER_URL }}
ALLURE_SERVER_USER: ${{ vars.ALLURE_SERVER_USER }}
ALLURE_SERVER_PASSWORD: ${{ secrets.allure-server-password }}
defaults:
run:
shell: pwsh
steps:
- name: Generate token
if: ${{ github.repository != 'Krukowskid/Krukowskid.Tests' }}
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.KrukowskidBotAppId }}
private_key: ${{ secrets.KrukowskidBotPrivateKey }}
- name: Checkout Tests
if: ${{ github.repository != 'Krukowskid/Krukowskid.Tests' }}
uses: actions/checkout@v3
with:
repository: Krukowskid/Krukowskid.Tests
token: "${{ steps.generate_token.outputs.token }}"
ref: main
- name: Checkout Tests
if: ${{ github.repository == 'Krukowskid/Krukowskid.Tests' }}
uses: actions/checkout@v3
- name: Azure login
uses: Azure/[email protected]
with:
client-id: ${{ secrets.azure-client-id }}
tenant-id: ${{ secrets.azure-tenant-id }}
subscription-id: ${{ secrets.azure-subscription-id }}
- name: Setup .NET
uses: actions/setup-dotnet@v3
with:
dotnet-version: 7.0.x
- name: Check Other Chrome Version
run: /usr/bin/google-chrome --version
- name: Restore dependencies
run: dotnet restore src
- name: List Config Files
run: ls src/Krukowskid.Tests.Common/Krukowskid.Tests.Common.Configuration
- name: Add TestResults dir
run: |
mkdir src/TestAutomation
mkdir src/TestAutomation/TestResults
mkdir src/TestAutomation/TestResults/AllureReports
- name: 🦿 Override WebUrl
if: ${{ inputs.tests-web-url != '' }}
shell: bash --noprofile --norc {0}
run: |
echo "Setting E2E_TESTS__WEB__URL env var to ${{ inputs.tests-web-url }}"
echo "E2E_TESTS__WEB__URL=${{ inputs.tests-web-url }}" >> $GITHUB_ENV
- name: 🦿 Override ApiGatewayUrl
if: ${{ inputs.tests-apigateway-url != '' }}
shell: bash --noprofile --norc {0}
run: |
echo "Setting E2E_TESTS__APIGATEWAY__URL env var to ${{ inputs.tests-apigateway-url }}"
echo "E2E_TESTS__APIGATEWAY__URL=${{ inputs.tests-apigateway-url }}" >> $GITHUB_ENV
- name: 🏗 Build
run: dotnet build src --no-restore
- name: List Files
run: |
ls src -lR > src/TestAutomation/TestResults/post-build-files.txt
ls ${{ github.workspace }}
- name: 🦾 Install browser for Playwright tests
shell: pwsh
run: src/Krukowskid.Tests.UI/Krukowskid.Tests.UI.x/bin/Debug/net7.0/playwright.ps1 install --with-deps chromium
- name: 🧪 Run tests for specified filter and rerun failed
shell: bash --noprofile --norc {0}
env:
LC_ALL: en_US.utf8
run: |
counter=1
exitcode=0
reset="\e[0m"
warn="\e[0;33m"
green="\e[0;92m"
blue="\e[0;94m"
while [ $counter -lt 4 ]
do
if [ $filter ]
then
echo -e "${warn}Run number: $counter. Re-running failed tests filter: $filter ${reset}"
# run test and forward output also to a file in addition to stdout (tee command)
cp src/TestAutomation/TestResults/runtestsoutput.log src/TestAutomation/TestResults/runtestsoutput_first.log
dotnet test src --no-build --filter=$filter --verbosity minimal --logger trx --results-directory src/TestAutomation/TestResults --settings:src/Krukowskid.Tests.Common/Krukowskid.Tests.Common.Configuration/cicd.runsettings | tee src/TestAutomation/TestResults/runtestsoutput.log
else
echo -e "${blue}First run. Running tests with filter "${{ inputs.tests-filter }}" ${reset}"
dotnet test src --no-build --filter "${{ inputs.tests-filter }}" --verbosity minimal --logger trx --results-directory src/TestAutomation/TestResults --settings:src/Krukowskid.Tests.Common/Krukowskid.Tests.Common.Configuration/cicd.runsettings | tee src/TestAutomation/TestResults/runtestsoutput.log
fi
# capture dotnet test exit status, different from tee
exitcode=${PIPESTATUS[0]}
if [ $exitcode == 0 ]
then
echo -e "${green}Running tests succeeded after $counter attempts.${reset}"
exit 0
fi
filter=$(cat src/TestAutomation/TestResults/runtestsoutput.log | grep -o -P '(?<=\sFailed\s)\w*'| grep -v -x 'Krukowskid' | awk 'BEGIN { ORS="|" } { print("Name=" $0) }' | grep -o -P '.*(?=\|$)')
((counter++))
done
exit $exitcode
- name: List Files
if: always()
run: ls src -lR > src/TestAutomation/TestResults/post-tests-files.txt
- name: 📈 Generate Github Report
uses: dorny/test-reporter@v1
if: always()
with:
name: ${{ inputs.report-name }} Test Execution Report
path: 'src/TestAutomation/TestResults/*.trx'
reporter: 'dotnet-trx'
list-suites: 'all'
fail-on-error: 'false'
- name: Find Allure Reports
if: ${{ always() && inputs.allure-reports == true }}
shell: bash
run: |
find src -type d -name "allure-results"
- name: Copy Allure Reports
if: ${{ always() && inputs.allure-reports == true }}
shell: bash
run: |
find src -type d -name "allure-results" -exec cp -r -v {}/. src/TestAutomation/TestResults/AllureReports \;
- name: 📈 Upload Allure Reports
uses: unickq/send-to-allure-docker-service-action@v1
if: ${{ always() && github.ref_name == 'main' && inputs.allure-reports == true }}
continue-on-error: true
with:
allure_results: src/TestAutomation/TestResults/AllureReports
project_id: ${{ inputs.allure-project-id }}
auth: true
generate: true
- name: Upload additional reports
uses: actions/upload-artifact@v3
if: always()
with:
name: ${{ inputs.report-name }}TestReports
path: |
src/TestAutomation
src/**/TestResults
src/**/bin/**/allureConfig.json
src/**/bin/**/appSettings.*.json |
|
Hi @krukowskid , From the description of this issue, I see the error is thrown from Azure CLI. But in the steps of "reusable workflow definition", I can't tell which step throws the exception. Could you answer these questions for the further analysis?
|
Its thrown in dotnet tests (
Its github hosted (large) runner., same problem on
same as on
on the day i was creating an issue 1.4.6 was the latest. I will try with 1.5.0 |
|
@krukowskid, |
YanaXu
added
question
|
I am using DefaultAzureCredential. Locally (with visualstudioidentity) it works, it also works with azure login action with secret |
|
@krukowskid , What I can see from |
|
In dotnet code I am using DefaultAzureCredential from Azure.Identity package. During authentication it loops trough all possible methods of authentication. When running test on runner it's using |
|
Sticking my me too on this problem, exactly the same error message and reporting of a 5 minute token. Out of curiosity, is there a point where the v1 tag should be dropped back to a previously working commit in order to avoid lots of issues? I know that best practice is that workflows should us commit hashes instead of tags when referencing actions but I'm sure there are lots of workflows that don't. |
Hi @shaneholder could you please provide more details about your issues? As we know, |
|
I don't know why but I can't replicate it anymore. However if you are still curious on how my workflow looks like : what I'm suspecting is that for the ubuntu runner we are using, azure cli might have been updated ? (I'm not sure which version of ubuntu we are running, but it might be that azure cli latest was not yet the right version for our distrib ?) |
|
Scratch that I actually still face it, but my real pipeline is a bit different as it also install azure-cli-core using pip3 for some requirements with the azure ansible collection. I wonder if it's the azure-cli-core (2.34.0) that messes up with the token expiration even though I login with the action before even installing this azure-cli-core, I am lost. EDIT: it's not, I tested by forcing the installation of 2.55.0 with pip3 and still the same thing. I'm trying some more workflows to see if I can replicate in an isolated environment |
|
@benjamin-rousseau-shift i think the issue is with the underlying OIDC token issued by Github (5 minutes expiry). it seems like its not a fault of Azure Cli. I've started having issues similar to yours after migrating to federated identity. I solved them: I'm using python, but you can implement this fix in any other language: |
|
@4c74356b41 By doing this I think you're basically doing exactly the same thing as the github action. |
|
@4c74356b41 @benjamin-rousseau-shift, you are right. The GitHub OIDC provider issues a JWT ID token with a 5-minute expiration time. Its lifespan is not officially documented. By decoding the OIDC token, we can find it is actually expired in 5 minutes. You can also verify this in the sample token. During login, Azure CLI will use the GitHub OIDC token to fetch an access token from MSAL. This access token will be stored in
For Regarding @krukowskid's issue, the error In my local testing, it works seamlessly under normal conditions, returning the access token from the cache without needing to fetch a new access token from MSAL. I am wondering if you use |
|
not sure if I'm interpreting what you say right. basically what you are saying that the default token in token cache should still be valid for 75 minutes on average and if we somehow retrieve that it should work (even though OIDC token expired)? |
|
@4c74356b41, you're correct. Azure CLI stores the access token fetched from MSAL, which is valid for 75 minutes on average. If you are trying to retrieve this token from cache, it should work without the need of OIDC token. But if you are retrieving a new access token from remote MSAL, it needs OIDC token. |
|
mkay, can you, please, help me understand how to reliably request token from the cache and not a new token? |
|
The temporary solution does not work when using source "azure-arm" "image" {
location = "${var.location}"
// Auth
use_azure_cli_auth = true
subscription_id = "${var.subscription_id}"
// Rest omitted.
}the process takes 6 hours to create fresh VM images and at the end of script when packer wants to create the final image in the azure gallery, we receive the same error: Sorry to mention you @jiasli: Does your fix takes in to account such scenarios as well? basically long-running pipelines (up to 6 hours) by refreshing the access token in background by providing refresh_tokens and get access_tokens in turn? |
|
❗ ❗ ❗If you are encountering
Check your scenario and use the provided workaround. We're actively working to resolve this issue. Thank you for your understanding. |
We are most likely hitting azure-cli issue from Azure/login#372 Signed-off-by: Marcel Zieba <[email protected]>
hey, we are using 2.60.0 and still seeing: az version output: any pointers? |
|
Hi @4c74356b41, please review scenario 2. The Azure CLI currently does not support ID token refresh. |
|
sorry, why scenario 2? this is not my scenario, I'm running 2.60.0 |
|
@4c74356b41, so it worked for you with azure-cli <= 2.58.0? Scenario 1 applies to users who encountered the issue only in azure-cli == 2.59.0. |
|
oh, okay, no, i dont think it did. i migrated to federated identity and this started happening. i was under the assumption that 2.60.0 is supposed to fix this underlying issue? |
|
@4c74356b41, not yet. Version 2.60.0 only addressed scenario 1. We're still waiting for Azure CLI to support ID token refresh for the other scenarios. Please check if the workarounds work for you. |
|
okay, my bad then. any chance I can track this work? |
|
@4c74356b41, I'll keep this issue open and update you once the improvement is ready. |
|
For my case, where we're only using ARM tokens, the new Azure CLI v2.60.0 baked into the new GitHub Actions runner image ( Thanks! 🙏 |
|
Converted one of my Azure DevOps service connections today to test the Workflow Federated Identity scenarios and the first pipeline I ran that used Azure CLI for longer than 10 minutes failed with this error ubuntu-latest
Add this to the day long outage for Workflow Federated Identity this month https://status.dev.azure.com/_event/499193080 and I remain unconvinced by the "new" way. The draconian measure of reducing the old secret way of working down to a 3 month expiry date is all well and good PROVIDED the new way works. I certainly cannot proceed with confidence right now - certainly not in production. |
|
Any update on this issue. My workflow is also failing with error |
any issues\milestone we can track on our own? thanks |
Let's track the feature here: Azure/azure-cli#28708. Currently, Azure CLI does not support ID token refresh. |
YanaXu
removed
the
needs-author-feedback
Hi! I am facing a similar issue (#180) that appears to have been resolved, but I'm still encountering this problem when executing dotnet tests in GitHub Runner.
I'm using action version
1.4.6andazure.identitypackage version1.10.4+DefaultAzureCredential(). The issue doesn't occur on integration tests where nearly all of them utilize tokens. However, if I run API/UI tests where I employ identity in one or two tests, it fails with above error. Do you have any suggestions or workarounds?The text was updated successfully, but these errors were encountered: