Canary tests usage and results analyzing

[evolkova-syberry]

Hey guys !

I try to understand what canary tests are and how do they work.
I was able to run them following readme.

As far as I understood from code and code comments (I'm not fmiliar with Go), there is a test suite which

• executes different types of tests: TestSignalWorkflow, TestSanityWorkflow etc
• starts again every 30 seconds but finishes in 10 minutes (so when first job will be finishing I'll have appr 20 jobs running simultaneously)
• in config I can exclude certain packages but I can't configure test suite run frequency

What I don't undesrtand is this:

1. are tests results verified automatically and how can I see if test suite succeded or failed ?
2. if I want to run canary test every hour as it's suggested here, then should I change a value for CronSchedule in source code ?
3. test suite implies a number of workflows to complete (with different statuses including failed) so when I see canary container logs and there are workflows/activity errors and workflows_failed in metrics count so I'm confused whether it's expected behaviour or not ?
   here is my graph for canary workflows failed statistics over time:
   
4. is in general my undesrtanding of canary tests execution & results analysis correct ? (I want one test suite to be run every hour so I can monitor different metrics and ensure that there is same number of workflows succeded / failed)

Thanks in advance

[longquanzheng]

Hi I am improving it here: https://github.com/uber/cadence/pull/4447/files

I also pushed the code of the PR into an image if you want to try out earlier: ubercadence/cadence-canary:latest

Highlights of the changes:

• The binary is NOT starting canary test anymore. It only start workers.
• You can start the canary manually by one-off or cron, see the README in the PR
• Docs about the canary project

For your questsions:

executes different types of tests: TestSignalWorkflow, TestSanityWorkflow etc

SanityWorklfow is a special case. It's the whole suite of all other test cases

are tests results verified automatically and how can I see if test suite succeded or failed ?

If workflow completed, then it succeeds, if failed then it failed.
Please see the README in the PR

workflows_failed in metrics

The metrics shows that ONLY the archival test case is failing. very likely this feature is not enabled yet in your cluster.
Please see the README in the PR.

[evolkova-syberry]

thanks a lot
I'll check it out