VCR Test Support for LongRunningOperations

doc/recording_vcr_tests.md

@@ -17,29 +17,74 @@ TEST_DEF = [
   },
   {
     'test_name': 'name2',
-    'command': 'command2'
+    'script': script_class()
   }
 ]
 ```
 
-Simply add your new entries and run all tests. The test driver will automatically detect the new tests and run the command, show you the output, and query if it is correct. If so, it will save the HTTP recording into a YAML file, as well as the raw output into a dictionary called `expected_results.res`. When the test is replayed, as long as the test has an entry in `expected_results.res` and a corresponding .yaml file, the test will proceed automatically. If either the entry or the .yaml file are missing, the test will be re-recorded.
+Simply add your new entries and run all tests. The test driver will automatically detect the new tests and run the commands, show you the output, and query if it is correct. If so, it will save the HTTP recording into a YAML file, as well as the raw output into a dictionary called `expected_results.res`. When the test is replayed, as long as the test has an entry in `expected_results.res` and a corresponding .yaml file, the test will proceed automatically. If either the entry or the .yaml file are missing, the test will be re-recorded.
 
 If the tests are run on TravisCI, any tests which cannot be replayed will automatically fail. 
 
-##Recording Tests
+##Types of Tests
 
-Many tests, for example those which simply retrieve information, can simply be played back, verified and recorded.
+The system currently accepts individual commands and script test objects. Individual commands will always display the output and query the user if the results are correct. These are the "old" style tests.
 
-Other tests, such as create and delete scenarios, may require additional commands to set up for recording, or may require additional commands to verify the proper operation of a command. For example, several create commands output nothing on success. Thus, you'll find yourself staring at nothing with a prompt asking if that is the expected response.
+To allow for more complex testing scenarios involving creating and deleting resources, long-running operations, or automated verification, use the script object method. To do so, simply create a class in the `command_specs.py` file with the following structure:
 
-For these scenarios, I recommend having a second shell open, from which you can run any setup commands and then run any commands you need to verify the proper operation of the command in order to answer the prompt.
+```Python
+class MyScenarioTest(CommandTestScript):
+  def __init__(self):
+    super(MyScenarioTest, self).__init__(self.set_up, self.test_body, self.tear_down)
+
+  def set_up(self):
+    # Setup logic here
+    pass
+
+  def test_body(self):
+    # Main test logic
+    pass
+
+  def tear_down(self):
+    # clean up logic here
+    pass
+```
+
+The `set_up` and `tear_down` methods are optional and can be omitted. A number of helper methods are available for structuring your script tests.
+
+####run(command_string)
+
+This method executes a given command and returns the output. The results are not sent to the display or to expected results. Use this for:
+
+- running commands that produce no output (the next statement will usually be a test)
+- running commands that are needed for conditional logic or in setup/cleanup logic
+
+####rec(command_string)
+
+This method runs a given command and records its output to the display for manual verification. Using this command will force the user to verify the output via a Y/N prompt. If the user accepts the output, it will be saved to `expected_results.res`.
+
+####test(command_string, checks)
+
+This method runs a given command and automatically validates the output. The results are saved to `expected_results.res` if valid, but nothing is display on the screen. Valid checks include: `bool`, `str` and `dict`. A check with a `dict` can be used to check for multiple matching parameters (and logic). Child `dict` objects can be used as values to verify properties within nested objects.
+
+####set_env(variable_name, value)
+
+This method is a wrapper around `os.environ` and simply sets an environment variable to the specified value.
+
+####pop_env(variable_name)
+
+Another wrapper around `os.environ` this pops the value of the indicated environment variable.
+
+####print_(string)
+
+This method allows you to write to the display output, but does not add to the `expected_results.res` file. One application of this would be to print information ahead of a `rec` statement so the person validating the output knows what to look for.
+
+##Long Running Operations (LRO)
 
-I don't recommend trying to structure your tests so that one test sets up for another, because in general you cannot guarantee the order in which the tests will run. Also, I don't recommend attempting to record large batches of tests at once. I generally add one to three tests at a time and leave the remaining new tests commented out. Running `testall.bat` will let me record these. Then I uncomment a few more and so on, until they are all recorded.
+The system now allows the testing of long running operations. Regardless of the time required to record the test, playback will truncate the long running operation to finish very quickly. However, because re-recording these actions can take a very long time, it is recommended that each LRO scenario be individually tested (possibly in tandem with a delete operation) rather than as part of a larger scenario.
 
 ##Limitations
 
 The current system saves time, but has some limitations.
 
-+ Certain commands require manual steps to setup or verify
 + You can't test for things like 'this input results in an exception'. It simply tests that the response equals an expected response.
-+ This system does not work with long running operations. While it technically does, the resulting recording takes as long as the original call, which negates some of the key benefits of automated testing.

lintall

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+export PYTHONPATH=$PATHONPATH:./src
+pylint -r n src/azure
+python scripts/command_modules/pylint.py

src/azure/cli/commands/__init__.py

@@ -58,7 +58,7 @@ def __call__(self, poller):
                 if self.progress_file:
                     print('.', end='', file=self.progress_file)
                     self.progress_file.flush()
-                time.sleep(self.poll_interval_ms / 1000.0)
+                    time.sleep(self.poll_interval_ms / 1000.0)
             result = poller.result()
             succeeded = True
             return result

src/azure/cli/utils/command_test_script.py

@@ -83,10 +83,7 @@ def _check_json(source, checks):
         if isinstance(checks, bool):
             result_val = str(result).lower().replace('"', '')
             bool_val = result_val in ('yes', 'true', 't', '1')
-            try:
-                assert bool_val == checks
-            except AssertionError as ex:
-                raise ex
+            assert bool_val == checks
         elif isinstance(checks, str):
             assert result.replace('"', '') == checks
         elif isinstance(checks, dict):
@@ -95,8 +92,7 @@ def _check_json(source, checks):
         elif checks is None:
             assert result is None or result == ''
         else:
-            raise IncorrectUsageError('test only accepts a dictionary of json properties or ' + \
-                                      'a boolean value.')
+            raise IncorrectUsageError('unsupported type \'{}\' in test'.format(type(checks)))
     def set_env(self, key, val): #pylint: disable=no-self-use
         os.environ[key] = val
 

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/__init__.py

@@ -279,7 +279,7 @@ def exists_container(args):
     'storage blob service-properties', None, _blob_data_service_factory,
     [
         AutoCommandDefinition(BlockBlobService.get_blob_service_properties,
-                              '[ServiceProperties]', 'show'),
+                              'ServiceProperties', 'show'),
         AutoCommandDefinition(BlockBlobService.set_blob_service_properties,
                               'ServiceProperties', 'set')
     ], command_table, PARAMETER_ALIASES, STORAGE_DATA_CLIENT_ARGS)

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/tests/command_specs.py

@@ -4,7 +4,6 @@
 import json
 import os
 import sys
-from time import sleep
 
 from six import StringIO
 
@@ -27,6 +26,31 @@ def _get_connection_string(runner):
     connection_string = out.replace('Connection String : ', '')
     runner.set_env('AZURE_STORAGE_CONNECTION_STRING', connection_string)
 
+class StorageAccountCreateAndDeleteTest(CommandTestScript):
+    def set_up(self):
+        self.account = 'testcreatedelete'
+        self.run('storage account delete -g {} -n {}'.format(RESOURCE_GROUP_NAME, self.account))
+        result = json.loads(self.run('storage account check-name --name {} -o json'.format(self.account)))
+        if not result['nameAvailable']:
+            raise RuntimeError('Failed to delete pre-existing storage account {}. Unable to continue test.'.format(self.account))
+
+    def test_body(self):
+        account = self.account
+        rg = RESOURCE_GROUP_NAME
+        s = self
+        s.run('storage account create --type Standard_LRS -l westus -n {} -g {}'.format(account, rg))
+        s.test('storage account check-name --name {}'.format(account),
+               {'nameAvailable': False, 'reason': 'AlreadyExists'})
+        s.run('storage account delete -g {} -n {}'.format(RESOURCE_GROUP_NAME, account))
+        s.test('storage account check-name --name {}'.format(account), {'nameAvailable': True})
+
+    def tear_down(self):
+        self.run('storage account delete -g {} -n {}'.format(RESOURCE_GROUP_NAME, self.account))
+
+    def __init__(self):
+        super(StorageAccountCreateAndDeleteTest, self).__init__(
+            self.set_up, self.test_body, self.tear_down)
+
 class StorageAccountScenarioTest(CommandTestScript):
 
     def test_body(self):
@@ -54,7 +78,8 @@ def test_body(self):
         s.run('storage account set -g {} -n {} --type Standard_LRS'.format(rg, account))
 
     def __init__(self):
-        super(StorageAccountScenarioTest, self).__init__(None, self.test_body, None)
+        super(StorageAccountScenarioTest, self).__init__(
+            None, self.test_body, None)
 
 class StorageBlobScenarioTest(CommandTestScript):
 
@@ -63,9 +88,9 @@ def set_up(self):
         self.rg = RESOURCE_GROUP_NAME
         self.proposed_lease_id = 'abcdabcd-abcd-abcd-abcd-abcdabcdabcd'
         self.new_lease_id = 'dcbadcba-dcba-dcba-dcba-dcbadcbadcba'
-        self.date = '2016-04-08T12:00Z'
+        self.date = '2016-04-01t12:00z'
         _get_connection_string(self)
-        sas_token = self.run('storage account generate-sas --services b --resource-types sco --permission rwdl --expiry 2017-01-01t00:00z')
+        sas_token = self.run('storage account generate-sas --services b --resource-types sco --permission rwdl --expiry 2100-01-01t00:00z')
         self.set_env('AZURE_SAS_TOKEN', sas_token)
         self.set_env('AZURE_STORAGE_ACCOUNT', STORAGE_ACCOUNT_NAME)
         self.pop_env('AZURE_STORAGE_CONNECTION_STRING')
@@ -187,7 +212,7 @@ def set_up(self):
         self.share1 = 'testshare01'
         self.share2 = 'testshare02'
         _get_connection_string(self)
-        sas_token = self.run('storage account generate-sas --services f --resource-types sco --permission rwdl --expiry 2017-01-01t00:00z')
+        sas_token = self.run('storage account generate-sas --services f --resource-types sco --permission rwdl --expiry 2100-01-01t00:00z')
         self.set_env('AZURE_SAS_TOKEN', sas_token)
         self.set_env('AZURE_STORAGE_ACCOUNT', STORAGE_ACCOUNT_NAME)
         self.pop_env('AZURE_STORAGE_CONNECTION_STRING')
@@ -301,28 +326,20 @@ def __init__(self):
         super(StorageFileScenarioTest, self).__init__(self.set_up, self.test_body, self.tear_down)
 
 TEST_DEF = [
-    # STORAGE ACCOUNT TESTS
     {
         'test_name': 'storage_account',
         'script': StorageAccountScenarioTest()
     },
-    # TODO: Enable when item #117262541 is complete
-    #{
-    #    'test_name': 'storage_account_create',
-    #    'command': 'storage account create --type Standard_LRS -l westus -g travistestresourcegroup --account-name teststorageaccount04'
-    #},
     {
-        'test_name': 'storage_account_delete',
-        'command': 'storage account delete -g travistestresourcegroup --account-name teststorageaccount04'
+        'test_name': 'storage_account_create_and_delete',
+        'script': StorageAccountCreateAndDeleteTest()
     },
-    # STORAGE CONTAINER TESTS
     {
         'test_name': 'storage_blob',
         'script': StorageBlobScenarioTest()
     },
-    # STORAGE SHARE TESTS
     {
         'test_name': 'storage_file',
         'script': StorageFileScenarioTest()
-    },
+    }
 ]