ci: create baseline and experimental test app pipelines

[goaway]

Description: Creates distinct test app profiles for end-to-end testing of baseline and new/experimental features in CI on Swift and Kotlin. Previously only a baseline configuration was run in CI which resulted in blindspots, especially with respect to upstream changes.
Risk Level: Low
Testing: Local & CI

Signed-off-by: Mike Schore mike.schore@gmail.com

[Augustyniak]

I like the general idea of us testing how the engine behaves when all/some of its features are enabled.

So just to make sure that you know where I am coming from. As I understood that when we were talking about these changes (back when you proposed them originally) the idea behind them was to "continue to make sure that the engine continues to perform well even if all of its feature are enabled".

Some of the thoughts after a brief look at the PR:

1. It adds a lot of complexity/maintenance burden as we introduce another copy(ies) of filters and some other components and some of the upstream changes will require us to update all of these copies.
2. It may add to the confusion for people who want to contribute to Envoy Mobile. We have 3(?) example apps per platform now as opposed to one and that may lead to a lot of questions. If we decide to stick with this approach we should add some documents that would explain why we need all of these example apps (and to make people update them as they keep adding features?)
3. Couldn't we just add a few integration tests (one with all feature disabled/one with all features enabled) that would start the engine and perform a requests (or a few of them) to see whether EM works well? Wouldn't that give us exactly the same level of confidence as all of these newly introduced example apps do? At the same time, with integration tests approach we could cut significantly reduce the amount of added code (and using integration tests to verify things seems to be a more common practice than using custom example apps).

1. and 2) are the issues that I see with the currently proposed approach. 3) is a question regarding an alternative that I can see.

[goaway]

I think I maybe wasn't clear enough in that I consider this PR as a starting point, and not the ideal end state.

1. It adds a lot of complexity/maintenance burden as we introduce another copy(ies) of filters and some other components and some of the upstream changes will require us to update all of these copies.

The idea is that these wouldn't be copies long-term. They're starting that way because it doesn't substantially change the functionality of the tests from today. I can (and would) open a tracking issue to delete most of the "example" code from the test harnesses and most of the "test" code from the example apps - resulting in different and simpler setups that serve their respective purposes better.

2. It may add to the confusion for people who want to contribute to Envoy Mobile. We have 3(?) example apps per platform now as opposed to one and that may lead to a lot of questions. If we decide to stick with this approach we should add some documents that would explain why we need all of these example apps (and to make people update them as they keep adding features?)

I'd actually like to move the test harness apps into the /tests and leave only the example apps in /examples. I'd actually suggest it's more confusing to have code that exists purely for CI verification sitting in the example apps.

3. Couldn't we just add a few integration tests (one with all feature disabled/one with all features enabled) that would start the engine and perform a requests (or a few of them) to see whether EM works well? Wouldn't that give us exactly the same level of confidence as all of these newly introduced example apps do? At the same time, with integration tests approach we could cut significantly reduce the amount of added code (and using integration tests to verify things seems to be a more common practice than using custom example apps).

We have these tests because we've found historically that integration tests are not enough to catch everything. Envoy Mobile has a lot of layer-to-layer wiring, and the narrower scope of both Unit and Integration tests have historically missed issues that cross those boundaries.

[Augustyniak]

After talking with @goaway offline:

• @goaway is going to open a GH issue that will track the future work for simplifying (removing) code from new CI example apps. We are going to remove some http filters, simplify apps' ViewControllers etc.
• we are going to add some documentation that would explain what's the intent behind these new CI example apps and how they should be updated when somebody add a new feature to Envoy Mobile.
• new CI example apps are going to be moved out of examples directory

[Augustyniak]

Let's make sure that we do simplify newly added soon. I will advocate for removing these if we see that they add too much of maintenance burden.