Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better testing of platform releases regarding plugin compatibility #54

Open
janpio opened this issue Nov 22, 2018 · 13 comments
Open

Better testing of platform releases regarding plugin compatibility #54

janpio opened this issue Nov 22, 2018 · 13 comments
Labels
enhancement New feature or request plugins

Comments

@janpio
Copy link
Member

janpio commented Nov 22, 2018

The 7.1.3 release of cordova-android had a breaking bug with multiple plugins that included app in their name, including our own cordova-plugin-inappbrowser.

The cordova-android tests did not catch that.

We should figure out a way to better test our platforms with plugins.

@janpio
Copy link
Member Author

janpio commented Nov 22, 2018

The plugin tests (run on CI via paramedic) themselves would probably have caught that, as they create a project and then see if functionality of the plugins work. But those tests are done with the released Cordova CLI and platforms.

We have the mobilespec project, but I am not exactly sure what that is and how it works.

@brodycj
Copy link

brodycj commented Nov 23, 2018

I just raised apache/cordova-mobile-spec#147 to document a few known quirks & issues:

@brodycj
Copy link

brodycj commented Nov 23, 2018

and not all of the automatic tests are passing for me. It seems to help to select 1 plugin at a time to run the tests on.

@janpio
Copy link
Member Author

janpio commented Nov 23, 2018

@brodycj

This comment has been minimized.

@janpio

This comment has been minimized.

@brodycj

This comment has been minimized.

@brodycj
Copy link

brodycj commented Nov 29, 2018

So mobilespec is one thing.

This and apache/cordova-mobile-spec#184 (comment) imply that there are some other ways to test platform releases and maybe some other package releases that I don't know about. Can you enlighten me and others?

@janpio
Copy link
Member Author

janpio commented Nov 29, 2018

So mobilespec is one thing.

This

That was the conclusion of us finding and understanding mobilespec as a way to test plugins compatibility on platforms in the comments before (which this topic is about).

apache/cordova-mobile-spec#184 (comment)

The comment you link to is about plugin testing, not platform releases.
Generally plugin testing is done via all the plugin tests, that are run via cordova-paramedic in CI. But that doesn't offer any (simple) options to control the platform being used.

But if you have such questions, please ask them in a new issue, Slack or dev mailing list thread and don't abuse one of my issues that has a clear topic or question - which is not about enlightening you.

@janpio
Copy link
Member Author

janpio commented Nov 29, 2018

Step back:

What would help to "test platform releases regarding plugin compatibility"? How can we find out if a new platform release breaks anything with plugins?

  1. Check if plugins install on the platform
  2. Check if normal app (without plugin usage) still works on platform with plugin installed
  3. Check if plugin tests app (cdvtests/index.html via cordova-plugin-test-framework) still works on platform with plugin installed
  4. Check if plugin Appium tests (currently only runnable on CI via cordova-paramedic) still work on platform with plugin installed

Other ideas?

@janpio
Copy link
Member Author

janpio commented Nov 29, 2018

Could we maybe somehow compare what a plugin installation does between two platform versions? That could catch if changes in our "copy files from a to b" logic (that caused problems in the past) will break a plugin without executing its code.

@brodycj
Copy link

brodycj commented Nov 29, 2018

I will need some time to evaluate these ideas. I think more comprehensive testing would be good on major releases and occasional minor releases. I would really favor a more straightforward testing procedure for hotfixes and other patch releases.

@janpio janpio added the enhancement New feature or request label Apr 10, 2019
@janpio janpio added the plugins label May 10, 2019
@janpio
Copy link
Member Author

janpio commented May 10, 2019

This exists:

https://github.com/alsorokin/cordova-periodic-build-android
https://github.com/alsorokin/cordova-periodic-build-ios

(Take a look at the Travis build history)

Broken right now, but could be a blueprint on how to create a repo that can test platforms with all plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugins
Projects
None yet
Development

No branches or pull requests

2 participants