Implement command line interface

[inikulin]

So, I would like to start bikeshedding regarding CLI. Here is how I see it:

 Usage: testcafe [command] [options]


  Commands:

    run [config-name]            run tests with given configuration and/or options
    add-config <config-name>     create run configuration for given options
    remove-config <config-name>  remove run configuration

  If [command] is not specified "run" command will be used as default

  Options:

    -h, --help                     output usage information
    -V, --version                  output the version number
    -lb, --list-browsers           output available browsers
    -f, --files <file1>,...        files or glob patterns to run
    -b, --browsers <browser1>,...  browsers to run tests in
    -r, --reporter <name>          specify the reporter to use
    -rp, --report-path <path>      output report to the file on the <path>
    -s, --screenshots <path>       enable screenshots and save them to the <path>
    -sf, --screenshots-on-fails    take screenshots on test fails
    -g, --grep <pattern>           only run tests matching <pattern>
    -f, --fgrep <string>           only run tests containing <string>
    -fg, --fixture-grep <pattern>  only run fixtures matching <pattern>
    -ff, --fixture-fgrep <string>  only run fixtures containing <string>
    -nc, --no-colors               force disabling of colors
    -gc, --global-config           use global configuration file

Regarding configurations: the idea is what you can save commonly used configuration with some alias, e.g.:

> testcafe add-config ie-grid -b ie -f test/**/*.js -fg ASPxGridView

Then this configuration will be saved to the .testcaferc file in the root of the project.
Optionally you can use -gc flag. In that case configuration will be saved to the global configuration file and can be reused in other projects.
It's no required for the configuration to have all options, so you can adopt it for different projects by passing additional options on run:

> testcafe add-config ie-chrome -b ie,chrome
> testcafe ie-chrome -f test/**/*.js

What bothers me:

• Is this configuration approach clear
• Is this usage is clear enough (we can add link to the detailed docs, so it's kinda not a big deal)
• Due to console limitations lists should not contains spaces, just ,. E.g.:

this is correct:

testcafe -b chrome,ie,firefiox

this is not:

testcafe -b chrome, ie, firefiox

I see it as a possible source for the user confusion.

Please, let me know what do you guys think and how we can make it better.
\cc @AlexanderMoskovkin @churkin @kirovboris @miherlosev @AlexanderMoskovkin @AndreyBelym @helen-dikareva @VasilyStrelyaev @MargaritaLoseva @arubtsov

[inikulin]

Additional questions:
should we use -f flag as a primary source of the files to run, or it should be used as the filter instead, so e.g. by default all .js files from test dir will be run and flag just filters found files. And which dir we should use by default: test, test/functional, test/automation, test/testcafe or use search in all of them?

[miherlosev]

looks good.
I think we need use the following syntax:
testcafe run - run all files in all configurations from the current directory
testcafe run -f test/**/*.js - run all found files in all configurations

[inikulin]

@miherlosev You've missed the idea of configuration. You can't run tests in all configurations, you need to choose one. Configurations are flexible: it can contain just information about browsers, so you need to specify files to run, or it can contain just file paths, and you need to specify browser to run in. It just the way to save some set of options and give it an alias.

[miherlosev]

ok.
testcafe run - run all files in default configurations from the current directory

[inikulin]

@miherlosev So, -f should work as filter?

[inikulin]

But which files we should run? All files in test directory? I bet there will be unit tests as well.

[AlexanderMoskovkin]

The configuration approach is quite clear for me.

Maybe the --files flag should be required? I can't find the answer for the question 'which files we should run by default'. I think it's specific for different projects

[helen-dikareva]

Now -f for --files and -f for --fgrep
And I think shortcuts for
-g, --grep only run tests matching
-f, --fgrep only run tests containing
-fg, --fixture-grep only run fixtures matching
-ff, --fixture-fgrep only run fixtures containing
is really easy to confuse

[inikulin]

TODOs:

• ports and hostname options
• remote browser connections option
• quarantine mode

[VasilyStrelyaev]

I like the configurations and they are clear enough to me. And quite useful, I should say.
Not sure that the --files flag used as a filter is something I could figure without reading the docs. Maybe it's because I'm totally unexperienced with CLIs. But still it's +1 to @AlexanderMoskovkin from me.
And, finally, regarding what @helen-dikareva noticed.
Does this look better?

   -g, --grep <pattern>           only run tests matching <pattern>
   -fg, --fgrep <string>           only run tests containing <string>
   -xg, --fixture-grep <pattern>  only run fixtures matching <pattern>
   -xf, --fixture-fgrep <string>  only run fixtures containing <string>

[inikulin]

What about making configurations even more flexible and allow to combine them, e.g.:
I have configuration for browsers and reporter:

> testcafe add-config all-browsers -b ie,chrome,ff

and configuration for target tests

> testcafe add-config ASPxGridView -f test/**/*.js -xf ASPxGridView
> testcafe add-config ASPxTreeList -f test/**/*.js -xf ASPxTreeList

The I can use them the following way:

> testcafe all-browsers ASPxGridView
> testcafe all-browser ASPxTreeList

Please, let me know what do you think

[AlexanderMoskovkin]

I like the idea, but will it be clear what the option value will be applied if some option will be in several configurations?

[inikulin]

I would like to summarize our discussion (both here and in person).

1. We decided to get rid of configuration concept. The reasons are:
   • It's not clear how merger options from multiple configurations: it's make sense to override in some cases (reporters, flags) and merge them in case of test files and browsers. This kind of disambiguation will be a reason for user confusion.
   • It's not clear where we should store configuration file. All possible options has significant disadvantages.
   • It's not clear how options should be used in the programmatically: they contain properties which affect construction of the TestCafe. There are now clear ways to achieve usage of both configuration files and API.
   • Configuration functionality can be easily implemented via shell scripts (or bat files on win) and aliases.
2. We haven't took into consideration when you need run browsers via path (our ASP farm case). The solution is: -b option should accept browser paths as well. We should try to recognize browser family from path and assign relevant browser info run options in browser-natives (task assigned to @AlexanderMoskovkin , @AndreyBelym )
3. Since we always require browsers and files to run task, they shouldn't be implemented as options (since they are not optional), command line will look this way:

testcafe [options] <comma-separated-browser-list> <file-or-glob...>

Here is the updated CLI usage:

  Usage: testcafe [options] <comma-separated-browser-list> <file-or-glob...>

  You can specify both browser alias and browser path in browser list.
  You can specify one or more file path or globbing pattern to run tests from.

  More info: http://http://testcafe.devexpress.com/Documentation/CLI

  Options:

    -h, --help                         output usage information
    -V, --version                      output the version number
    -b, --list-browsers                output available browser aliases
    -bc, --browser-connection <count>  create remote or local browser connections
    -r, --reporter <name>              specify the reporter to use
    -rp, --report-path <path>          output report to the file on the <path>
    -s, --screenshots <path>           enable screenshots and save them to the <path>
    -sf, --screenshots-on-fails        take screenshots on test fails
    -g, --grep <pattern>               only run tests matching <pattern>
    -f, --fgrep <string>               only run tests containing <string>
    -xg, --fixture-grep <pattern>      only run fixtures matching <pattern>
    -xf, --fixture-fgrep <string>      only run fixtures containing <string>
    -nc, --no-colors                   force disabling of colors
    -p1, --port1 <num>                 specify custom port1 number
    -p2, --port2 <num>                 specify custom port2 number
    -hs, --hostname <name>             specify hostname

Please, add final corrections, so I can start implementing it.

[VasilyStrelyaev]

lgtm so far

[VasilyStrelyaev]

Shouldn't we add a browser alias to run a task against all installed browsers?
I feel like that's a rare scenario but there's going to be a couple of folks who request it...

[inikulin]

@VasilyStrelyaev It's not a big deal to support:

testcafe * <files...>

[AndreyBelym]

LGTM except that it's more conventional to use -v (small v) as shortcut for --version. I can't remember any other CLI tool, that uses big V, except Mocha. And our other shortcuts have small letters first, so -V looks out of order a bit.

[inikulin]

@AndreyBelym I'll be happy too, but AFAIK it's not configurable in commander which I would like to use.

[AlexanderMoskovkin]

one question,
-s, --screenshots <path> enable screenshots and save them to the <path>
if the options is not set we don't save screenshots even when act.screenshot used?

[inikulin]

@AlexanderMoskovkin AFAIR we've decide to not make screenshots if screenshot path is not provided. We need to decide how to include screenshots in the report BTW, I remember we've got conversation about it with @kirovboris , but I don't remember the conclusion. Anyway, we'll resolve it context of #104

[inikulin]

Some additions:

• quarantine mode
• don't fail on JS errors

  Usage: testcafe [options] <comma-separated-browser-list> <file-or-glob...>

  You can specify both browser alias and browser path in browser list.
  You can specify one or more file path or globbing pattern to run tests from.

  More info: http://http://testcafe.devexpress.com/Documentation/CLI

  Options:

    -h, --help                         output usage information
    -V, --version                      output the version number
    -b, --list-browsers                output available browser aliases
    -bc, --browser-connection <count>  create remote or local browser connections
    -r, --reporter <name>              specify the reporter to use
    -rp, --report-path <path>          output report to the file on the <path>
    -s, --screenshots <path>           enable screenshots and save them to the <path>
    -sf, --screenshots-on-fails        take screenshots on test fails
    -q, --quarantine-mode              enable quarantine mode
    -E, --skip-js-errors               don't fail tests on JS errors
    -g, --grep <pattern>               only run tests matching <pattern>
    -f, --fgrep <string>               only run tests containing <string>
    -xg, --fixture-grep <pattern>      only run fixtures matching <pattern>
    -xf, --fixture-fgrep <string>      only run fixtures containing <string>
    -nc, --no-colors                   force disabling of colors
    -p1, --port1 <num>                 specify custom port1 number
    -p2, --port2 <num>                 specify custom port2 number
    -hs, --hostname <name>             specify hostname

[VasilyStrelyaev]

nice

[inikulin]

A little update: ports via single option, remotes now specified via browser list (we can have no alises specified, but browser list is required):

  Usage: testcafe [options] <comma-separated-browser-list> <file-or-glob ...>


    You can specify both browser alias and browser path in browser list.

    If you want use remote browser connection (e.g. connect mobile device)
    specify "remote" as browser alias. If you want to connect multiple devices
    then prefix alias with the number of browsers you want to connect (e.g. "3remote").


    You can specify one or more file path or globbing pattern to run tests from.

    More info: http://testcafe.devexpress.com/Documentation/CLI

  Options:

    -h, --help                     output usage information
    -V, --version                  output the version number
    -b, --list-browsers            output available browser aliases
    -r, --reporter <name>          specify the reporter to use
    -rp, --report-path <path>      output report to the file on the <path>
    -s, --screenshots <path>       enable screenshots and save them to the <path>
    -sf, --screenshots-on-fails    take screenshots on test fails
    -q, --quarantine-mode          enable quarantine mode
    -E, --skip-js-errors           don't fail tests on JS errors
    -g, --grep <pattern>           only run tests matching <pattern>
    -f, --fgrep <string>           only run tests containing <string>
    -xg, --fixture-grep <pattern>  only run fixtures matching <pattern>
    -xf, --fixture-fgrep <string>  only run fixtures containing <string>
    -p, --ports <port1,port2>      specify custom port numbers
    -hs, --hostname <name>         specify hostname

[VasilyStrelyaev]

ok then, if we need to get rid of two-letter flags:

    -h, --help                     output usage information
    -V, --version                  output the version number
    -b, --list-browsers            output available browser aliases
    -r, --reporter <name>          specify the reporter to use
    -p, --report-path <path>       output report to the file on the <path>
    -s, --screenshots <path>       enable screenshots and save them to the <path>
    -S, --screenshots-on-fails     take screenshots on test fails
    -q, --quarantine-mode          enable quarantine mode
    -e, --skip-js-errors           don't fail tests on JS errors
    -g, --grep <pattern>           only run tests matching <pattern>
    -f, --fgrep <string>           only run tests containing <string>
    -G, --fixture-grep <pattern>   only run fixtures matching <pattern>
    -F, --fixture-fgrep <string>   only run fixtures containing <string>
    --ports <port1,port2>          specify custom port numbers
    --hostname <name>              specify hostname

[inikulin]

Thinking of this recently I came to the following set of options:

    -h, --help                     output usage information
    -V, --version                  output the version number
    -b, --list-browsers            output available browser aliases
    -r, --reporter <name>          specify the reporter to use
    -p, --report-path <path>       output report to the file on the <path>
    -s, --screenshots <path>       enable screenshots and save them to the <path>
    -S, --screenshots-on-fails     take screenshots on test fails
    -q, --quarantine-mode          enable quarantine mode
    -e, --skip-js-errors           don't fail tests on JS errors
    -t, --test <name>              only run tests with <name>
    -T, --test-grep <pattern>      only run tests matching <pattern>
    -f, --fixture <name>           only run fixtures with <name>
    -F, --fixture-grep <pattern>   only run fixtures matching <pattern>
    --ports <port1,port2>          specify custom port numbers
    --hostname <name>              specify hostname

It's close to the @kirovboris ideas. IMHO we will have more memorizable flags. grep covers contains-functionality anyway. Please, let me know what do you think?
\cc everyone

[AndreyBelym]

Cool, I like it.

[VasilyStrelyaev]

flags are better this way, yeah

[AlexanderMoskovkin]

nice

[kirovboris]

I like it too

[miherlosev]

good

[lock]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.