mysql_info - Add connector_name and connector_version to returned value #497
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
@laurent-indermuehle great job, thank you!
Could you please add integration tests?
If it's impossible or hard to determine what the new values return in each particular case we could check if they return anything from the possible options using or. For versions it can be just a check that it's not none.
changelogs/fragments/497_mysql_info_returns_connector_name_and_version.yml
Outdated
Show resolved
Hide resolved
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
Co-authored-by: Andrew Klychkov <[email protected]>
In case of MySQLdb is renamed in mysqlclient. In that case, the integration tests will catch this the day we update the connector version.
|
Green! At last! :) |
Andersson007
approved these changes
Jan 30, 2023
There was a problem hiding this comment.
@laurent-indermuehle great addition! Thanks!
How about adding yourself to author: ? (feel free to merge once you add yourself or now)
Where should I do that? In the documentation block of |
|
|
Ok thanks. @Andersson007 Now for the difficult question. Should I backport this new feature? |
@laurent-indermuehle as it's an exceptional case i think we can backport it to stable-2 |
laurent-indermuehle
deleted the
lie_mysql_info_returns_connector_name_and_version
branch
Backport to stable-1: 💚 backport PR created✅ Backport PR branch: Backported as #498 🤖 @patchback |
Backport to stable-2: 💚 backport PR created✅ Backport PR branch: Backported as #499 🤖 @patchback |
patchback bot
pushed a commit
that referenced
this pull request
Jan 30, 2023
…ue (#497) * Add methods to retrieve connector name and version * Document that mysqlclient is also named MySQLdb * Document version_added * Add connector name and version in the returned block * Cut condition to display any name that is return In case of MySQLdb is renamed in mysqlclient. In that case, the integration tests will catch this the day we update the connector version. Co-authored-by: Andrew Klychkov <[email protected]> (cherry picked from commit a5f3296)
patchback bot
pushed a commit
that referenced
this pull request
Jan 30, 2023
…ue (#497) * Add methods to retrieve connector name and version * Document that mysqlclient is also named MySQLdb * Document version_added * Add connector name and version in the returned block * Cut condition to display any name that is return In case of MySQLdb is renamed in mysqlclient. In that case, the integration tests will catch this the day we update the connector version. Co-authored-by: Andrew Klychkov <[email protected]> (cherry picked from commit a5f3296)
laurent-indermuehle
added a commit
to laurent-indermuehle/community.mysql
that referenced
this pull request
Jan 31, 2023
Andersson007
pushed a commit
that referenced
this pull request
Feb 1, 2023
… and connector_version to returned value (#498) * mysql_info - Add connector_name and connector_version to returned value (#497) * Add methods to retrieve connector name and version * Document that mysqlclient is also named MySQLdb * Document version_added * Add connector name and version in the returned block * Cut condition to display any name that is return In case of MySQLdb is renamed in mysqlclient. In that case, the integration tests will catch this the day we update the connector version. Co-authored-by: Andrew Klychkov <[email protected]> (cherry picked from commit a5f3296) * Cut fragment not relevant to the collection usage * Updated version_added for stable-1 --------- Co-authored-by: Laurent Indermühle <[email protected]>
Andersson007
pushed a commit
that referenced
this pull request
Feb 1, 2023
… and connector_version to returned value (#499) * mysql_info - Add connector_name and connector_version to returned value (#497) * Add methods to retrieve connector name and version * Document that mysqlclient is also named MySQLdb * Document version_added * Add connector name and version in the returned block * Cut condition to display any name that is return In case of MySQLdb is renamed in mysqlclient. In that case, the integration tests will catch this the day we update the connector version. Co-authored-by: Andrew Klychkov <[email protected]> (cherry picked from commit a5f3296) * Cut fragment not relevant to the collection usage * Update version_added for stable-2 --------- Co-authored-by: Laurent Indermühle <[email protected]>
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Mar 2, 2023
7.3.0 Major Changes ------------- kubernetes.core ~~~~~~~~~~~~~~~ - refactor K8sAnsibleMixin into module_utils/k8s/ (ansible-collections/kubernetes.core#481). Minor Changes ------------- Ansible-core ~~~~~~~~~~~~ - Make using blocks as handlers a parser error (ansible/ansible#79968) - ansible-test - Specify the configuration file location required by test plugins when the config file is not found. This resolves issue: ansible/ansible#79411 - ansible-test - Update error handling code to use Python 3.x constructs, avoiding direct use of ``errno``. - ansible-test acme test container - update version to update used Pebble version, underlying Python and Go base containers, and Python requirements (ansible/ansible#79783). cisco.aci ~~~~~~~~~ - Add Node Profile BGP Peer and Route Control Profile functionalities to aci_l3out_bgp_peer module - Add SVI auto state support (auto_state attribute) to aci_l3out_interface - Add aci_aaa_domain, aci_aaa_role and aci_custom_privilege modules - Add aci_fabric_pod_policy_group module - Add aci_interface_policy_leaf_profile_fex_policy_group module and add FEX support to aci_access_port_to_interface_policy_leaf_profile - Add aci_tenant_span_src_group_src module - Add action_groups for module_defaults - Add support for filter direction in aci_contract_subject and aci_contract_subject_to_filter - Update modules to assign roles and permissions to a user cisco.nxos ~~~~~~~~~~ - `nxos_acls` - Support ICMPv6 option. Please refer to module doc for all new options (ansible-collections/cisco.nxos#624). - `nxos_facts` - Update facts gathering logic to ensure that `gather_network_resources: all` does not fail for NX-OS on MDS switches. - `nxos_l2_interfaces` - Add new mode dot1q-tunnel (ansible-collections/cisco.nxos#600). community.crypto ~~~~~~~~~~~~~~~~ - get_certificate - adds ``ciphers`` option for custom cipher selection (ansible-collections/community.crypto#571). community.general ~~~~~~~~~~~~~~~~~ - dnsimple - set custom User-Agent for API requests to DNSimple (ansible-collections/community.general#5927). - flatpak_remote - add new boolean option ``enabled``. It controls, whether the remote is enabled or not (ansible-collections/community.general#5926). - gitlab_project - add ``releases_access_level``, ``environments_access_level``, ``feature_flags_access_level``, ``infrastructure_access_level``, ``monitor_access_level``, and ``security_and_compliance_access_level`` options (ansible-collections/community.general#5986). - jc filter plugin - added the ability to use parser plugins (ansible-collections/community.general#6043). - keycloak_group - add new optional module parameter ``parents`` to properly handle keycloak subgroups (ansible-collections/community.general#5814). - keycloak_user_federation - make ``org.keycloak.storage.ldap.mappers.LDAPStorageMapper`` the default value for mappers ``providerType`` (ansible-collections/community.general#5863). - ldap modules - add ``xorder_discovery`` option (ansible-collections/community.general#6045, ansible-collections/community.general#6109). - lxd_container - add diff and check mode (ansible-collections/community.general#5866). - mattermost, rocketchat, slack - replace missing default favicon with docs.ansible.com favicon (ansible-collections/community.general#5928). - modprobe - add ``persistent`` option (ansible-collections/community.general#4028, ansible-collections/community.general#542). - osx_defaults - include stderr in error messages (ansible-collections/community.general#6011). - proxmox - suppress urllib3 ``InsecureRequestWarnings`` when ``validate_certs`` option is ``false`` (ansible-collections/community.general#5931). - redfish_command - adding ``EnableSecureBoot`` functionality (ansible-collections/community.general#5899). - redfish_command - adding ``VerifyBiosAttributes`` functionality (ansible-collections/community.general#5900). - sefcontext - add support for path substitutions (ansible-collections/community.general#1193). community.grafana ~~~~~~~~~~~~~~~~~ - able to set `uid` for datasources in grafana via module grafana_datasource community.mongodb ~~~~~~~~~~~~~~~~~ - 491 mongodb_shell - Add feature to detect if mongo or mongosh is available. - 494 mongodb_auth - Removes module_defaults from role. - 494 mongodb_shutdown - Fix examples block. - 511 mongodb_auth - Adds support for deletion of users. - 514 mongodb_linux - Remove extended FQCN for pam_limits. - 524 mongodb_auth - Add supports for Amazon Linux 2. - 528 multiple roles - Use first ip address when multiple bind IPs provided. - 530 mongodb_role - Adds new module to manage MongoDB roles. - 536 mongodb_auth - Add user after enabling authentication. - 544 mongodb_replicaset - Module documentation improvements. - 547 mongodb_repository - Bump default of MongoDB to 6.0. community.mysql ~~~~~~~~~~~~~~~ - mysql_info - add ``connector_name`` and ``connector_version`` to returned values (ansible-collections/community.mysql#497). - mysql_role - enable auto_commit to avoid MySQL metadata table lock (ansible-collections/community.mysql#479). - mysql_user - add plugin_auth_string as optional parameter to use a specific pam service if pam/auth_pam plugin is used (ansible-collections/community.mysql#445). - mysql_user - add the ``session_vars`` argument to set session variables at the beginning of module execution (ansible-collections/community.mysql#478). - mysql_user - display a more informative invalid privilege exception. Changes the exception handling of the granting permission logic to show the query executed , params and the exception message granting privileges fails` (ansible-collections/community.mysql#465). - mysql_user - enable auto_commit to avoid MySQL metadata table lock (ansible-collections/community.mysql#479). - setup_mysql - update MySQL tarball URL (ansible-collections/community.mysql#491). community.vmware ~~~~~~~~~~~~~~~~ - vmware_guest_disk - Add support for IDE disk add, remove or reconfigure, and change to gather same VM disk info as in vmware_guest_disk_info (ansible-collections/community.vmware#1428). - vmware_guest_disk - Extend return value documentation for vmware_guest_disk (ansible-collections/community.vmware#1641) - vmware_guest_disk_info - Move gather VM disk info function to vm_device_helper.py (ansible-collections/community.vmware#1617) - vmware_vmotion - New parameter timeout in order to allow vmotions running longer than 1 hour (https://github.com/ansible-collections/community.vmware/pulls/1629). grafana.grafana ~~~~~~~~~~~~~~~ - Updated the return message in grafana.grafana.folder module hetzner.hcloud ~~~~~~~~~~~~~~ - hcloud_server - add private_networks_info containing name and private ip in responses - hcloud_server_info - add private_networks_info containing name and private ip in responses - inventory plugin - Add list of all private networks to server variables. - inventory plugin - Add new connect_with setting public_ipv6 to connect to discovered servers via public IPv6 address. - inventory plugin - Add public IPv6 address to server variables. - inventory plugin - Log warning instead of crashing when some servers do not work with global connect_with setting. inspur.ispim ~~~~~~~~~~~~ - Change the ansible-test.yml application file version. - Change the description of the edit_bios module file_url field. - Modify the description information of the backup module item field. - Modify the description of the media_attach, retry_count, and retry_time_interval fields of the edit_kvm module. - Modify the description of the secure_channel field of the edit_media_instance module. - Modify the description of the slot and vname fields of the add_ldisk module. - Modify the edit_ntp module example. - Modify the edit_snmp_trap module version field description information. - Modify the mode field description information of update_fw module. - Modify the name field description of the user_group module. - Modify the restore module example. - Modify the supporting properties and description information of the edit_ncsi module edit_ncsi field. - The edit_power_budget module adds the except_action field. kubernetes.core ~~~~~~~~~~~~~~~ - Adjust k8s_user_impersonation tests to be compatible with Kubernetes 1.24 (ansible-collections/kubernetes.core#520). - add support for dry run with kubernetes client version >=18.20 (ansible-collections/kubernetes.core#245). - added ignore.txt for Ansible 2.14 devel branch. - fixed module_defaults by removing routing hacks from runtime.yml (ansible-collections/kubernetes.core#347). - helm - add support for -set-file, -set-json, -set and -set-string options when running helm install (ansible-collections/kubernetes.core#533). - helm - add support for helm dependency update (ansible-collections/kubernetes.core#208). - helm - add support for post-renderer flag (ansible-collections/kubernetes.core#30). - helm - add support for timeout cli parameter to allow setting Helm timeout independent of wait (ansible-collections/kubernetes.core#67). - helm - add support for wait parameter for helm uninstall command. (https://github.com/ansible-collections/kubernetes/core/issues/33). - helm - support repo location for helm diff (ansible-collections/kubernetes.core#174). - helm - when ansible is executed in check mode, return the diff between what's deployed and what will be deployed. - helm, helm_plugin, helm_info, helm_plugin_info, kubectl - add support for in-memory kubeconfig. (ansible-collections/kubernetes.core#492). - helm_info - add hooks, notes and manifest as part of returned information (ansible-collections/kubernetes.core#546). - helm_info - add release state as a module argument (ansible-collections/kubernetes.core#377). - helm_info - added possibility to get all values by adding get_all_values parameter (ansible-collections/kubernetes.core#531). - helm_plugin - Add plugin_version parameter to the helm_plugin module (ansible-collections/kubernetes.core#157). - helm_plugin - Add support for helm plugin update using state=update. - helm_repository - Ability to replace (overwrite) the repo if it already exists by forcing (ansible-collections/kubernetes.core#491). - helm_repository - add support for pass-credentials cli parameter (ansible-collections/kubernetes.core#282). - helm_repository - added support for ``host``, ``api_key``, ``validate_certs``, and ``ca_cert``. - helm_repository - mark `pass_credentials` as no_log=True to silence false warning (ansible-collections/kubernetes.core#412). - helm_template - add name (NAME of release) and disable_hook as optional module arguments (ansible-collections/kubernetes.core#313). - helm_template - add show_only and release_namespace as module arguments (ansible-collections/kubernetes.core#313). - helm_template - add support for -set-file, -set-json, -set and -set-string options when running helm template (ansible-collections/kubernetes.core#546). - k8s - add no_proxy support to k8s* (ansible-collections/kubernetes.core#272). - k8s - add support for server_side_apply. (ansible-collections/kubernetes.core#87). - k8s - add support for user impersonation. (https://github.com/ansible-collections/kubernetes/core/issues/40). - k8s - allow resource definition using metadata.generateName (ansible-collections/kubernetes.core#35). - k8s lookup plugin - Enable turbo mode via environment variable (ansible-collections/kubernetes.core#291). - k8s, k8s_scale, k8s_service - add support for resource definition as manifest via. (ansible-collections/kubernetes.core#451). - k8s_cp - remove dependency with 'find' executable on remote pod when state=from_pod (ansible-collections/kubernetes.core#486). - k8s_drain - Adds ``delete_emptydir_data`` option to ``k8s_drain.delete_options`` to evict pods with an ``emptyDir`` volume attached (ansible-collections/kubernetes.core#322). - k8s_exec - select first container from the pod if none specified (ansible-collections/kubernetes.core#358). - k8s_exec - update deprecation warning for `return_code` (ansible-collections/kubernetes.core#417). - k8s_json_patch - minor typo fix in the example section (ansible-collections/kubernetes.core#411). - k8s_log - add the ``all_containers`` for retrieving all containers' logs in the pod(s). - k8s_log - added the `previous` parameter for retrieving the previously terminated pod logs (ansible-collections/kubernetes.core#437). - k8s_log - added the `tail_lines` parameter to limit the number of lines to be retrieved from the end of the logs (ansible-collections/kubernetes.core#488). - k8s_rollback - add support for check_mode. (https://github.com/ansible-collections/kubernetes/core/issues/243). - k8s_scale - add support for check_mode. (https://github.com/ansible-collections/kubernetes/core/issues/244). - kubectl - wait for dd command to complete before proceeding (ansible-collections/kubernetes.core#321). - kubectl.py - replace distutils.spawn.find_executable with shutil.which in the kubectl connection plugin (ansible-collections/kubernetes.core#456). netapp.ontap ~~~~~~~~~~~~ - na_ontap_aggregate - new option ``allow_flexgroups`` added. - na_ontap_cifs - new options ``access_based_enumeration``, ``change_notify``, ``encryption``, ``home_directory``, ``oplocks``, ``show_snapshot``, ``allow_unencrypted_access``, ``namespace_caching`` and ``continuously_available`` added in REST. - na_ontap_dns - ``skip_validation`` option requires 9.9.1 or later with REST and ignored for cluster DNS operations. - na_ontap_dns - support cluster scope for modify and delete. - na_ontap_interface - do not attempt to migrate FC interface if desired ``home_port``, ``home_node`` and ``current_port``, ``current_node`` are same. - na_ontap_license - support for NLF v2 license files. - na_ontap_nfs - new options ``root``, ``windows`` and ``security`` added in REST. - na_ontap_user_role - ``command_directory_name`` is required if ``privileges`` not set in REST. - na_ontap_user_role - ``path`` is required if ``privileges`` set in REST. - na_ontap_volume_efficiency - REST support for ``policy`` requires 9.7 or later, ``path`` requires 9.9.1 or later and ``volume_efficiency`` and ``start_ve_scan_old_data`` requires 9.11.1 or later. - na_ontap_volume_efficiency - ``schedule``, ``start_ve_scan_all``, ``start_ve_build_metadata``, ``start_ve_delete_checkpoint``, ``start_ve_queue_operation``, ``start_ve_qos_policy`` and ``stop_ve_all_operations`` options are not supported with REST. - na_ontap_volume_efficiency - new option ``volume_name`` added. - na_ontap_volume_efficiency - updated private cli with REST API. netbox.netbox ~~~~~~~~~~~~~ - nb_inventory - Add serial and asset tag to extracted attributes purestorage.flasharray ~~~~~~~~~~~~~~~~~~~~~~ - purefa_network - Added support for NVMe-RoCE and NVMe-TCP service types - purefa_user - Added Ops Admin role to choices - purefa_vlan - Added support for NVMe-TCP service type Breaking Changes / Porting Guide -------------------------------- hetzner.hcloud ~~~~~~~~~~~~~~ - inventory plugin - Python v3.5+ is now required. Deprecated Features ------------------- - Since the google.cloud collection seems to be maintained again, we `cancelled the removal process <https://github.com/ansible-collections/overview/blob/main/removal_from_ansible.rst#cancelling-removal-of-an-unmaintained-collection>`__. So contrary to an earlier announcement, this collection is NOT deprecated and will NOT be removed from Ansible 8 (ansible-community/community-topics#105). community.general ~~~~~~~~~~~~~~~~~ - gitlab_runner - the option ``access_level`` will lose its default value in community.general 8.0.0. From that version on, you have set this option to ``ref_protected`` explicitly, if you want to have a protected runner (ansible-collections/community.general#5925).
Andersson007
pushed a commit
that referenced
this pull request
Mar 21, 2023
…versions match expectation (#490) * Draft: Add a mariadb container * Add playbook to test connection to the server * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add default file for root necessary since using venv instead of docker * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add ansible-test integration inventory to .gitignore * Revert to old workflow to use ansible-test --venv It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497 * Cut target filtering * Fix comparison We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test. * Add path to default-file /root doesn't exist with --venv * Fix workflow unknown option container_name * Attempt GHA communication between container using "docker host network" https://docs.github.com/en/actions/using-containerized-services/about-service-containers I re-revert the workflow to use the new custom action. But I'm not sure it will works because I don't know how the container for ansible-test is started and if it will have access to the services containers. * Cut anchors currently unsupported by GHA * Disable healthcheck I want to first prove that this setup is possible before adding safety * Disable sanity, units and matrix to speed up tests in GHA * Further disable tests to speed up * Add mysql_client to the controller * Install mysql_client the correct way * Fix package name and missing apt cache * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Swap MariaDB with MySQL * De-duplicate the mysql_command alias * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Cut docker healthcheck unsupported by GHA * Fix replication server_id already in use * Add static test with replication containers * Fix database not selected * Fix replication due to usage of gateway_addr instead of localhost * Simplify version computation * Linting * Refactor setup_mysql into setup_controller * Fix test_mysql_role * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Add back a package to connect to MySQL 8+ * Linting * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Fix missing var * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Fix volume path * Fix volume path by adding a final / * Fix volume path using $(pwd) * Fix volume path using github.workspace var * Cut files from gitignore because it prevents ansible-test to copy them * Fix pre-test-cmd missing separators * Cut the newline added by lookup 'file' * Fix tailing newline by not created it in the first place * Disable tests to concentrate on the \n and quote issue with my files * Fix trailing newline and quote in db_engine_version * Re-enable integration tests to validate db_engine_version is fixed * lint * Cut unused file * Fix pre-test-cmd paste in wrong context * Re-enable service containers * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Add tmate to debug the server_id in replicas * Attempt to fix "invalid syntax" * Enclose command in quotes * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Re-activate all tests * Cut useless task * Use same variable as other target for consistency * Linting * Update version tested * Add options to the makefile * Add same variables as other target for consistency * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Attempt to set log-bin into docker * Reformat for readability * Document that full version is mandatory * Fix newline * Github complain it doesn't find python 3.1 !!! * Add option to run only a single target * Fix mysqlclient not supporting Python 3.9 * Enhance installation of mysql_client Initially I wanted to install mysql-client-5.7 to test mysql server 5.7 but this package is not available for Ubuntu 18+. I keep those changes because it allow us to specify the name of the package based on the Ubuntu version. * Linting * Add unique name to simplify debugging * Fix mysql_dump for MySQL 5.7 and MariaDB when using mysqldump 8 * Add unique name to simplify debugging * Deduplicate tasks * Lining * Add python script to recreate the test matrix from github workflow file * Fix dump with mysqldump 8 against mysql 5.7 * Disable test for replication with chanel for mysql 5.7 * Add better task name * Fix exclusion function * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Cut Ansible since we can't change what the user have on his computer * Add running make command for all tests of the matrix * Add unique test names * Document run_all_tests.py * Add unique test names * Add tmate to experiment with docker healthcheck * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add TODO verify that the version of mysql/mariadb is correct * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Fix healthcheck in GHA * Disable tests that fails only on MariaDB * Refactor to remove useless variables * Workaround for plugin role that fails with any MariaDB versions * Fix Python 3.10 beein run as 3.1 * Ensure replicas are healthy before rebooting them * Enable all tests * Add a virtualenv for ansible-test used locally * Simplify connector_name variables * Add PoC using custom ansible-test containers * Fix docker_container variable name * Cut forgotten comment * Fix error when using local registry by using quay.io * Change tag of test-containers to latest * Fix ansible-test unknown option I copied blindly https://github.com/ansible-collections/community.sops/blob/main/.github/workflows/ansible-test.yml#L195 and forgot what ansible-test was expecting * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Add manual test matrix (MariaDB 10.6, 10.7 and 10.8 missing) * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Reduce number of tests and adapt containers images * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Fix assertion for mariadb * Linting * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Cut playbook that are now handled by the test-containers * Change timeout from 10 to 30 seconds to let mysql/mariadb restart * Add connector information to the returned values I need to know what python library was used. I had a container with both mysqlclient and pymysql installed and tests used a different connector that what is advertised by the title of integration tests. We need to prevent that otherwise our tests are worth nothing. * Add a verify stage at setup of test to assert all version are correct * Attempt to build and publish an image on ghcr.io * Add latest release of actions and with a context * Add trigger on workflow file edit * Fix env not recognized in the 'on' clause * Add latest tag * Fix insufficient context * Add missing slash * Cut addition of tag 'latest' as GHA does it automatically * Add ghcr.io image for mariadb10.3 python3.8 mysqlclient2.0.1 * Change docker-image workflow to work on all images using matrix * Fix workflow title * Add support for version of mysqlclient * Fix context path * Workaround failed to push ghcr.io Error was: failed to copy: io: read/write on closed pipe * Add back all tests using ghcr.io images * Cut unused images * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Fix verify for mysqlclient second name MySQLdb * Rename variable for consistency * Fix container name * Add tag 'latest' to images * Cut filter for tests now that the right connector is used * Fix test of mysql/mariadb version in use * Fix python version lookup * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Document TODO * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix dict key lookup * Fix indentation * Cut tests that was excluded in previous matrix * Enable back sanity and unit tests * Refactor get_driver_version to display name while passing sanity tests * Fix variable name * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Fix image not found * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Fix verify ansible 'devel' for which the version is unknown Today 'devel' means 2.15, but in the future it will be something else. * Fix ansible version extraction for "devel" * Cut matrix from when build was done in a single workflow * Document fix container name * Add bold * Add option to let containers alive at end of testing * Enhance error handling and doc of get_driver_name and get_driver_version * Migrate tests documentations in their own file * Skip retry-on-error by default and add option to activate it on demand * Rename folder to better purpose * Enable back push and schedule workflow * Rename registry from fork to upstream * Cut Docker Image workflow's filter for branch from my fork * Add changelog fragment * Update supported versions * Rename file for clarity * Cut mariadb non long term releases * Add '-client' to the block title to better explain what it is * Update readme for tested versions of long term release of MariaDB * Attempt to add the workflow to the Action tab * Second attempt to add the workflow to the Action tab * Cut folder re-created by merge from main * Cut filter by branch GHA will build the image using the branch name as tag. So we can safely remove this filter. * Cut changelog item done in #497 * Attempt to fix upload of image under c.mysql instead of my fork * Add debug to buildkitd * Bump setup-buildy-action to latest * Cut dot in image name in attempt to fix buildx bad request 400 error * Sanitize the repository name using metadata-action https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#repository-name-must-be-lowercase * Document why we use optional checkout action * Cut debugging from setup-buildx-action * Fix workflow to work both on fork and c.mysql repository * Use apt-get instead of apt that not have a stable CLI interface * Use apt-get instead of apt * update docker image path to my personal repo I'm unable to publish under community.mysql. Either it's the dot in the name or I do something wrong with the GITHUB_TOKEN, but we need to test my PR, so I'll use docker images from my fork for now. * Fix test after merge of PR497 * Enhance testing documentation header * Fix installation of ansible venv ansible-test is included in ansible package. Also, on Fedora 37 with python 3.11, pip is missing. By using ensurepip we solve that issue. * Document usage of continue_on_errors * Fix versions used in examples * Add support for systems with unsupported python set as default * Fix cleanup task * Fix variable assignation to the include task * Add forgotten variable to handle unsupported python version * Fix user site-packages not visible in virtualenv * Fix test connection to the database and tasks names * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Reformat command multiline to oneline * Add deletion of anonymous volumes associated with the container * Comment unused variable * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat spelling Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Add better task name Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Move utility task files in their own folder * Refactor using reusable GHA workflows * Fix path to called workflow file * Fix path to use local workflow * Fix cannot specify version when calling local workflows * Attempt to use a fixed repo name in the image name My last attempts produced duplicates images under my name + repo name: laurent-indermuehle/community.mysql. Previously I had only my name. And none of the above are what we want. We want only community.mysql in the image name... * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collection. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]>
laurent-indermuehle
added a commit
to laurent-indermuehle/community.mysql
that referenced
this pull request
Mar 27, 2023
…versions match expectation (ansible-collections#490) * Draft: Add a mariadb container * Add playbook to test connection to the server * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add default file for root necessary since using venv instead of docker * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add ansible-test integration inventory to .gitignore * Revert to old workflow to use ansible-test --venv It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497 * Cut target filtering * Fix comparison We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test. * Add path to default-file /root doesn't exist with --venv * Fix workflow unknown option container_name * Attempt GHA communication between container using "docker host network" https://docs.github.com/en/actions/using-containerized-services/about-service-containers I re-revert the workflow to use the new custom action. But I'm not sure it will works because I don't know how the container for ansible-test is started and if it will have access to the services containers. * Cut anchors currently unsupported by GHA * Disable healthcheck I want to first prove that this setup is possible before adding safety * Disable sanity, units and matrix to speed up tests in GHA * Further disable tests to speed up * Add mysql_client to the controller * Install mysql_client the correct way * Fix package name and missing apt cache * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Swap MariaDB with MySQL * De-duplicate the mysql_command alias * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Cut docker healthcheck unsupported by GHA * Fix replication server_id already in use * Add static test with replication containers * Fix database not selected * Fix replication due to usage of gateway_addr instead of localhost * Simplify version computation * Linting * Refactor setup_mysql into setup_controller * Fix test_mysql_role * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Add back a package to connect to MySQL 8+ * Linting * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Fix missing var * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Fix volume path * Fix volume path by adding a final / * Fix volume path using $(pwd) * Fix volume path using github.workspace var * Cut files from gitignore because it prevents ansible-test to copy them * Fix pre-test-cmd missing separators * Cut the newline added by lookup 'file' * Fix tailing newline by not created it in the first place * Disable tests to concentrate on the \n and quote issue with my files * Fix trailing newline and quote in db_engine_version * Re-enable integration tests to validate db_engine_version is fixed * lint * Cut unused file * Fix pre-test-cmd paste in wrong context * Re-enable service containers * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Add tmate to debug the server_id in replicas * Attempt to fix "invalid syntax" * Enclose command in quotes * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Re-activate all tests * Cut useless task * Use same variable as other target for consistency * Linting * Update version tested * Add options to the makefile * Add same variables as other target for consistency * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Attempt to set log-bin into docker * Reformat for readability * Document that full version is mandatory * Fix newline * Github complain it doesn't find python 3.1 !!! * Add option to run only a single target * Fix mysqlclient not supporting Python 3.9 * Enhance installation of mysql_client Initially I wanted to install mysql-client-5.7 to test mysql server 5.7 but this package is not available for Ubuntu 18+. I keep those changes because it allow us to specify the name of the package based on the Ubuntu version. * Linting * Add unique name to simplify debugging * Fix mysql_dump for MySQL 5.7 and MariaDB when using mysqldump 8 * Add unique name to simplify debugging * Deduplicate tasks * Lining * Add python script to recreate the test matrix from github workflow file * Fix dump with mysqldump 8 against mysql 5.7 * Disable test for replication with chanel for mysql 5.7 * Add better task name * Fix exclusion function * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Cut Ansible since we can't change what the user have on his computer * Add running make command for all tests of the matrix * Add unique test names * Document run_all_tests.py * Add unique test names * Add tmate to experiment with docker healthcheck * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add TODO verify that the version of mysql/mariadb is correct * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Fix healthcheck in GHA * Disable tests that fails only on MariaDB * Refactor to remove useless variables * Workaround for plugin role that fails with any MariaDB versions * Fix Python 3.10 beein run as 3.1 * Ensure replicas are healthy before rebooting them * Enable all tests * Add a virtualenv for ansible-test used locally * Simplify connector_name variables * Add PoC using custom ansible-test containers * Fix docker_container variable name * Cut forgotten comment * Fix error when using local registry by using quay.io * Change tag of test-containers to latest * Fix ansible-test unknown option I copied blindly https://github.com/ansible-collections/community.sops/blob/main/.github/workflows/ansible-test.yml#L195 and forgot what ansible-test was expecting * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Add manual test matrix (MariaDB 10.6, 10.7 and 10.8 missing) * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Reduce number of tests and adapt containers images * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Fix assertion for mariadb * Linting * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Cut playbook that are now handled by the test-containers * Change timeout from 10 to 30 seconds to let mysql/mariadb restart * Add connector information to the returned values I need to know what python library was used. I had a container with both mysqlclient and pymysql installed and tests used a different connector that what is advertised by the title of integration tests. We need to prevent that otherwise our tests are worth nothing. * Add a verify stage at setup of test to assert all version are correct * Attempt to build and publish an image on ghcr.io * Add latest release of actions and with a context * Add trigger on workflow file edit * Fix env not recognized in the 'on' clause * Add latest tag * Fix insufficient context * Add missing slash * Cut addition of tag 'latest' as GHA does it automatically * Add ghcr.io image for mariadb10.3 python3.8 mysqlclient2.0.1 * Change docker-image workflow to work on all images using matrix * Fix workflow title * Add support for version of mysqlclient * Fix context path * Workaround failed to push ghcr.io Error was: failed to copy: io: read/write on closed pipe * Add back all tests using ghcr.io images * Cut unused images * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Fix verify for mysqlclient second name MySQLdb * Rename variable for consistency * Fix container name * Add tag 'latest' to images * Cut filter for tests now that the right connector is used * Fix test of mysql/mariadb version in use * Fix python version lookup * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Document TODO * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix dict key lookup * Fix indentation * Cut tests that was excluded in previous matrix * Enable back sanity and unit tests * Refactor get_driver_version to display name while passing sanity tests * Fix variable name * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Fix image not found * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Fix verify ansible 'devel' for which the version is unknown Today 'devel' means 2.15, but in the future it will be something else. * Fix ansible version extraction for "devel" * Cut matrix from when build was done in a single workflow * Document fix container name * Add bold * Add option to let containers alive at end of testing * Enhance error handling and doc of get_driver_name and get_driver_version * Migrate tests documentations in their own file * Skip retry-on-error by default and add option to activate it on demand * Rename folder to better purpose * Enable back push and schedule workflow * Rename registry from fork to upstream * Cut Docker Image workflow's filter for branch from my fork * Add changelog fragment * Update supported versions * Rename file for clarity * Cut mariadb non long term releases * Add '-client' to the block title to better explain what it is * Update readme for tested versions of long term release of MariaDB * Attempt to add the workflow to the Action tab * Second attempt to add the workflow to the Action tab * Cut folder re-created by merge from main * Cut filter by branch GHA will build the image using the branch name as tag. So we can safely remove this filter. * Cut changelog item done in ansible-collections#497 * Attempt to fix upload of image under c.mysql instead of my fork * Add debug to buildkitd * Bump setup-buildy-action to latest * Cut dot in image name in attempt to fix buildx bad request 400 error * Sanitize the repository name using metadata-action https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#repository-name-must-be-lowercase * Document why we use optional checkout action * Cut debugging from setup-buildx-action * Fix workflow to work both on fork and c.mysql repository * Use apt-get instead of apt that not have a stable CLI interface * Use apt-get instead of apt * update docker image path to my personal repo I'm unable to publish under community.mysql. Either it's the dot in the name or I do something wrong with the GITHUB_TOKEN, but we need to test my PR, so I'll use docker images from my fork for now. * Fix test after merge of PR497 * Enhance testing documentation header * Fix installation of ansible venv ansible-test is included in ansible package. Also, on Fedora 37 with python 3.11, pip is missing. By using ensurepip we solve that issue. * Document usage of continue_on_errors * Fix versions used in examples * Add support for systems with unsupported python set as default * Fix cleanup task * Fix variable assignation to the include task * Add forgotten variable to handle unsupported python version * Fix user site-packages not visible in virtualenv * Fix test connection to the database and tasks names * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Reformat command multiline to oneline * Add deletion of anonymous volumes associated with the container * Comment unused variable * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat spelling Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Add better task name Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Move utility task files in their own folder * Refactor using reusable GHA workflows * Fix path to called workflow file * Fix path to use local workflow * Fix cannot specify version when calling local workflows * Attempt to use a fixed repo name in the image name My last attempts produced duplicates images under my name + repo name: laurent-indermuehle/community.mysql. Previously I had only my name. And none of the above are what we want. We want only community.mysql in the image name... * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collection. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> (cherry picked from commit 6970aef)
laurent-indermuehle
added a commit
to laurent-indermuehle/community.mysql
that referenced
this pull request
Mar 27, 2023
…versions match expectation (ansible-collections#490) * Draft: Add a mariadb container * Add playbook to test connection to the server * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add default file for root necessary since using venv instead of docker * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add ansible-test integration inventory to .gitignore * Revert to old workflow to use ansible-test --venv It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497 * Cut target filtering * Fix comparison We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test. * Add path to default-file /root doesn't exist with --venv * Fix workflow unknown option container_name * Attempt GHA communication between container using "docker host network" https://docs.github.com/en/actions/using-containerized-services/about-service-containers I re-revert the workflow to use the new custom action. But I'm not sure it will works because I don't know how the container for ansible-test is started and if it will have access to the services containers. * Cut anchors currently unsupported by GHA * Disable healthcheck I want to first prove that this setup is possible before adding safety * Disable sanity, units and matrix to speed up tests in GHA * Further disable tests to speed up * Add mysql_client to the controller * Install mysql_client the correct way * Fix package name and missing apt cache * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Swap MariaDB with MySQL * De-duplicate the mysql_command alias * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Cut docker healthcheck unsupported by GHA * Fix replication server_id already in use * Add static test with replication containers * Fix database not selected * Fix replication due to usage of gateway_addr instead of localhost * Simplify version computation * Linting * Refactor setup_mysql into setup_controller * Fix test_mysql_role * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Add back a package to connect to MySQL 8+ * Linting * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Fix missing var * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Fix volume path * Fix volume path by adding a final / * Fix volume path using $(pwd) * Fix volume path using github.workspace var * Cut files from gitignore because it prevents ansible-test to copy them * Fix pre-test-cmd missing separators * Cut the newline added by lookup 'file' * Fix tailing newline by not created it in the first place * Disable tests to concentrate on the \n and quote issue with my files * Fix trailing newline and quote in db_engine_version * Re-enable integration tests to validate db_engine_version is fixed * lint * Cut unused file * Fix pre-test-cmd paste in wrong context * Re-enable service containers * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Add tmate to debug the server_id in replicas * Attempt to fix "invalid syntax" * Enclose command in quotes * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Re-activate all tests * Cut useless task * Use same variable as other target for consistency * Linting * Update version tested * Add options to the makefile * Add same variables as other target for consistency * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Attempt to set log-bin into docker * Reformat for readability * Document that full version is mandatory * Fix newline * Github complain it doesn't find python 3.1 !!! * Add option to run only a single target * Fix mysqlclient not supporting Python 3.9 * Enhance installation of mysql_client Initially I wanted to install mysql-client-5.7 to test mysql server 5.7 but this package is not available for Ubuntu 18+. I keep those changes because it allow us to specify the name of the package based on the Ubuntu version. * Linting * Add unique name to simplify debugging * Fix mysql_dump for MySQL 5.7 and MariaDB when using mysqldump 8 * Add unique name to simplify debugging * Deduplicate tasks * Lining * Add python script to recreate the test matrix from github workflow file * Fix dump with mysqldump 8 against mysql 5.7 * Disable test for replication with chanel for mysql 5.7 * Add better task name * Fix exclusion function * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Cut Ansible since we can't change what the user have on his computer * Add running make command for all tests of the matrix * Add unique test names * Document run_all_tests.py * Add unique test names * Add tmate to experiment with docker healthcheck * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add TODO verify that the version of mysql/mariadb is correct * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Fix healthcheck in GHA * Disable tests that fails only on MariaDB * Refactor to remove useless variables * Workaround for plugin role that fails with any MariaDB versions * Fix Python 3.10 beein run as 3.1 * Ensure replicas are healthy before rebooting them * Enable all tests * Add a virtualenv for ansible-test used locally * Simplify connector_name variables * Add PoC using custom ansible-test containers * Fix docker_container variable name * Cut forgotten comment * Fix error when using local registry by using quay.io * Change tag of test-containers to latest * Fix ansible-test unknown option I copied blindly https://github.com/ansible-collections/community.sops/blob/main/.github/workflows/ansible-test.yml#L195 and forgot what ansible-test was expecting * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Add manual test matrix (MariaDB 10.6, 10.7 and 10.8 missing) * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Reduce number of tests and adapt containers images * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Fix assertion for mariadb * Linting * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Cut playbook that are now handled by the test-containers * Change timeout from 10 to 30 seconds to let mysql/mariadb restart * Add connector information to the returned values I need to know what python library was used. I had a container with both mysqlclient and pymysql installed and tests used a different connector that what is advertised by the title of integration tests. We need to prevent that otherwise our tests are worth nothing. * Add a verify stage at setup of test to assert all version are correct * Attempt to build and publish an image on ghcr.io * Add latest release of actions and with a context * Add trigger on workflow file edit * Fix env not recognized in the 'on' clause * Add latest tag * Fix insufficient context * Add missing slash * Cut addition of tag 'latest' as GHA does it automatically * Add ghcr.io image for mariadb10.3 python3.8 mysqlclient2.0.1 * Change docker-image workflow to work on all images using matrix * Fix workflow title * Add support for version of mysqlclient * Fix context path * Workaround failed to push ghcr.io Error was: failed to copy: io: read/write on closed pipe * Add back all tests using ghcr.io images * Cut unused images * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Fix verify for mysqlclient second name MySQLdb * Rename variable for consistency * Fix container name * Add tag 'latest' to images * Cut filter for tests now that the right connector is used * Fix test of mysql/mariadb version in use * Fix python version lookup * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Document TODO * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix dict key lookup * Fix indentation * Cut tests that was excluded in previous matrix * Enable back sanity and unit tests * Refactor get_driver_version to display name while passing sanity tests * Fix variable name * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Fix image not found * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Fix verify ansible 'devel' for which the version is unknown Today 'devel' means 2.15, but in the future it will be something else. * Fix ansible version extraction for "devel" * Cut matrix from when build was done in a single workflow * Document fix container name * Add bold * Add option to let containers alive at end of testing * Enhance error handling and doc of get_driver_name and get_driver_version * Migrate tests documentations in their own file * Skip retry-on-error by default and add option to activate it on demand * Rename folder to better purpose * Enable back push and schedule workflow * Rename registry from fork to upstream * Cut Docker Image workflow's filter for branch from my fork * Add changelog fragment * Update supported versions * Rename file for clarity * Cut mariadb non long term releases * Add '-client' to the block title to better explain what it is * Update readme for tested versions of long term release of MariaDB * Attempt to add the workflow to the Action tab * Second attempt to add the workflow to the Action tab * Cut folder re-created by merge from main * Cut filter by branch GHA will build the image using the branch name as tag. So we can safely remove this filter. * Cut changelog item done in ansible-collections#497 * Attempt to fix upload of image under c.mysql instead of my fork * Add debug to buildkitd * Bump setup-buildy-action to latest * Cut dot in image name in attempt to fix buildx bad request 400 error * Sanitize the repository name using metadata-action https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#repository-name-must-be-lowercase * Document why we use optional checkout action * Cut debugging from setup-buildx-action * Fix workflow to work both on fork and c.mysql repository * Use apt-get instead of apt that not have a stable CLI interface * Use apt-get instead of apt * update docker image path to my personal repo I'm unable to publish under community.mysql. Either it's the dot in the name or I do something wrong with the GITHUB_TOKEN, but we need to test my PR, so I'll use docker images from my fork for now. * Fix test after merge of PR497 * Enhance testing documentation header * Fix installation of ansible venv ansible-test is included in ansible package. Also, on Fedora 37 with python 3.11, pip is missing. By using ensurepip we solve that issue. * Document usage of continue_on_errors * Fix versions used in examples * Add support for systems with unsupported python set as default * Fix cleanup task * Fix variable assignation to the include task * Add forgotten variable to handle unsupported python version * Fix user site-packages not visible in virtualenv * Fix test connection to the database and tasks names * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Reformat command multiline to oneline * Add deletion of anonymous volumes associated with the container * Comment unused variable * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat spelling Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Add better task name Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> * Move utility task files in their own folder * Refactor using reusable GHA workflows * Fix path to called workflow file * Fix path to use local workflow * Fix cannot specify version when calling local workflows * Attempt to use a fixed repo name in the image name My last attempts produced duplicates images under my name + repo name: laurent-indermuehle/community.mysql. Previously I had only my name. And none of the above are what we want. We want only community.mysql in the image name... * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collection. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <[email protected]> (cherry picked from commit 6970aef)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Add two returned values for the
mysql_infoplugin :connector_nameandconnector_version.You may have both mysqlclient and pymysql installed. This helps you know which one was used.
This feature is mandatory for #490 who will add a step that verify that all components have expected versions at the start of integrations tests.
ISSUE TYPE
COMPONENT NAME
mysql_info
ADDITIONAL INFORMATION