Connectedk8s nightly testing #5566
Conversation
modified: src/connectedk8s/azext_connectedk8s/_params.py modified: src/connectedk8s/azext_connectedk8s/_utils.py modified: src/connectedk8s/azext_connectedk8s/custom.py new file: src/connectedk8s/azext_connectedk8s/patch-file.yaml
modified: src/connectedk8s/azext_connectedk8s/_utils.py modified: src/connectedk8s/azext_connectedk8s/custom.py
modified: src/connectedk8s/azext_connectedk8s/custom.py
modified: src/connectedk8s/azext_connectedk8s/commands.py modified: src/connectedk8s/azext_connectedk8s/custom.py new file: src/connectedk8s/azext_connectedk8s/tests/latest/recordings/test_forcedelete.yaml modified: src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
modified: src/connectedk8s/azext_connectedk8s/custom.py renamed: src/connectedk8s/azext_connectedk8s/patch-file.yaml -> src/connectedk8s/azext_connectedk8s/remove_crd_finalizer.yaml modified: src/connectedk8s/azext_connectedk8s/tests/latest/recordings/test_forcedelete.yaml modified: src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
modified: src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
modified: src/connectedk8s/azext_connectedk8s/custom.py
modified: src/connectedk8s/setup.py
modified: src/connectedk8s/azext_connectedk8s/tests/latest/recordings/test_forcedelete.yaml
modified: src/connectedk8s/azext_connectedk8s/tests/latest/recordings/test_forcedelete.yaml
….com/rohan-dassani/azure-cli-extensions into timeout_in_crd_deletion_force_delete
modified: src/connectedk8s/azext_connectedk8s/tests/latest/recordings/test_forcedelete.yaml
modified: src/connectedk8s/setup.py
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
…ectedk8s_scenario.py
…ectedk8s_scenario.py
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Show resolved
Hide resolved
| @ResourceGroupPreparer(name_prefix='conk8stest', location='eastus2euap', random_name_length=16) | ||
| def test_upgrade(self,resource_group): | ||
|
|
||
| managed_cluster_name = self.create_random_name(prefix='test-upgrade', length=24) |
There was a problem hiding this comment.
Is it mandatory to create the aks cluster and onboard it every time? Is there any way with which we can onboard once and run other commands/tests on top of it . If not all, atleast if we can reuse it for some tests so that we can save time for the overall test run
There was a problem hiding this comment.
AKS cluster creation everytime will take a lot of time. What is the avg time our current pipeline is taking?
There was a problem hiding this comment.
all the tests actually run parallelly, the avg time is 50 mins but the point here is when the pipeline is triggered it just not only goes to our extension it sequentially goes to all of the extension spend around 2 mins each (again this is parallel too) and then executes our tests too. Pipeline ex - https://dev.azure.com/azclitools/internal/_build/results?buildId=33353&view=results
Having only one aks means , testing everything inside one function , this might actually increase the execution time of the tests and not sure if this is a cleaner approach
There was a problem hiding this comment.
why does it go to other extensions if we set the pipeline variable to run only our extension ?
There was a problem hiding this comment.
Also, do we know how many tests run in parallel at once? Even with parallel if it is taking 50 mins, we should get that clarity
There was a problem hiding this comment.
when we set the pipeline paramter , there is a parameter called user_parallelism = 8 by default , and I have seen 8 or maybe more extensions that are run parallely
There was a problem hiding this comment.
That is number of extensions run parallely, I am asking about number of tests per extension parallely.
There was a problem hiding this comment.
If tests per extension don't run parallely, then there would be no point in create aks and onboarding it everytime. Also, does the pipeline provide an option to run selected tests per extension or all tests will be run always
There was a problem hiding this comment.
- The live test was not designed to be run every time. It will only run automatically at specific times, such as every Thursday morning.
- Each extension will assign an agent to run the test, which is why you will see so many extensions running at the same time, but they all run for only two minutes, because they are skipped because the running conditions are not met.
- If you want to run the tests on every commit, please remove the @live_only decorator and we will run the recording tests after every commit. (Please remember to upload the recording file generated by the live test)
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
|
Connectedk8s |
…ectedk8s_scenario.py
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Show resolved
Hide resolved
src/connectedk8s/azext_connectedk8s/tests/latest/test_connectedk8s_scenario.py
Outdated
Show resolved
Hide resolved
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
…ectedk8s_scenario.py
|
@wangzelin007 can you please review this PR |
…ectedk8s_scenario.py
| class Connectedk8sScenarioTest(LiveScenarioTest): | ||
|
|
||
| def test_connectedk8s(self): | ||
| @live_only() |
There was a problem hiding this comment.
May I ask why these tests marked as @live_only()?
There was a problem hiding this comment.
I also have the same problem.
There was a problem hiding this comment.
- If you want to run recording tests on every commit, please remove the @live_only decorator and we will run the recording tests after every commit. (Please remember to upload the recording file generated by the live test)
- Otherwise none of the following tests will actually be executed：
There was a problem hiding this comment.
@wangzelin007 if you remember we earlier attempted to try the recording mode but it didnt work out because of some issue in our connectedk8s connect code ( we are fetching tenant-id from cache because of which the tests were failing in recording mode) so for now we have decided to go with live mode only , we know this will not run as integration tests but till the time we find a solution for our command, we will make this a practice from our end that the member that raises the PR will trigger the CLI testing pipeline
@sirireddy12 if you have anything to add
5 participants
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally?For new extensions:
About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update
src/index.jsonautomatically.The precondition is to put your code inside this repository and upgrade the version in the pull request but do not modify
src/index.json.