Skip to content

Conversation

@bim-msft
Copy link
Contributor

@bim-msft bim-msft commented Feb 6, 2020


This checklist is used to make sure that common guidelines for a pull request are followed.

@bim-msft bim-msft requested a review from jiasli February 6, 2020 09:57
@bim-msft
Copy link
Contributor Author

bim-msft commented Feb 6, 2020

@jiasli Reduced the name prefix length to meet the rule and reran all tests according to your advice, please review it again.

@bim-msft
Copy link
Contributor Author

bim-msft commented Feb 6, 2020

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jiasli
Copy link
Member

jiasli commented Feb 7, 2020

After live run KeyVaultCertificateDefaultPolicyScenarioTest, the recording yaml is removed. CI reports failure:

  File "/opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages/azure/cli
    /command_modules/resource/custom.py", line 736, in create_resource_group
  ...
knack.util.CLIError: Please run 'az login' to setup account.

By looking into ResourceGroupPreparer then live_only_execute, see this line returns True:

if self.test_class_instance.in_recording:

Then find usage of in_recording reveals this line in azure_devtools.

self.in_recording = self.is_live or not os.path.exists(self.recording_file)

How it fails:

  1. Because group create/delete commands are not recorded and there is no network traffic in the test body, after live run the yaml is not created
  2. When running as recording, ReplayableTest gets is_live == False, but still thinks this is a live test because the yaml doesn't exists. In other words, the current design requires ResourceGroupPreparer to be paired with a recording yaml for the test to be treated as recording. Even if no --live or AZURE_TEST_RUN_LIVE is specified, lacking of yaml will still turn this test into a live test
  3. The test is ran as live. This doesn't cause problem locally because az has been logged in, but in CI it has not.

To solve it, simply remove:

@ResourceGroupPreparer(name_prefix='cli_test_kv_cert_default_policy')

@yonzhan yonzhan added this to the S165 milestone Feb 7, 2020
@bim-msft bim-msft merged commit 601dfc0 into Azure:dev Feb 7, 2020
@bim-msft bim-msft deleted the bim_kv_fix_test branch February 7, 2020 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants