Skip to content

Test Arg File

Jump to bottom
Vladislav Alekseev edited this page Jan 25, 2023 · 25 revisions

Test Arg File is a JSON file that describes a test plan, or a job. You pass it via --test-arg-file argument. It is required to:

  • perform a test discovery
  • schedule tests onto a shared queue

You can generate a sample test arg file and modify to your needs:

$ Emcee initTestArgFile --output /path/to/testargfile.json

Schema

JSON schema is defined by TestArgFile.swift file.

Root fields:

  • jobId: String - a unique job id. You can safely use some UUID for this field. Re-using the same job id will accumulate test results of all invocations with the same job id.
  • jobPriority: Int - a priority of a job, an Integer number with range from 0 to 999. Jobs (tests) with higher priority execute before the jobs with lower priority. This field is optional.
  • jobGroupId: String - a unique id of job group. This field is optional. You can logically group a set of jobId-s by assigning them the same jobGroupId. Grouping jobs allows to have a better control over the priority of job execution process.
  • jobGroupPriority: Int - a priority for this job group. Tests that are part of job group with higher job group priority will be executed before tests which are part of job group with lower job group priority. jobPriority controls the order of test execution within job group, and jobGroupPriority controls the order of test execution within the whole queue.
  • testDestinationConfigurations: Array - this is an array of objects that describe the outputs per test destination. More details below.
  • analyticsConfiguration: Object - defines an analytics configuration for this job. Read more how to configure it here.
  • entries: Array - a list of entries, each entry describes a set of tests to be executed. Usually you will have 1 or 2 entries: e.g. one for UI tests, and another one for unit tests.

entries

Each entry describe test bundle and tests that you'd like to execute. The fields are described below.

buildArtifacts

This is an object which describes build artifacts. Its content depends on type of tests you want to execute, you can read more here.

The fields are:

  • appBundle: the location of a ZIP file with application bundle (.app). This is optional field, you don't need it if you want to execute Logic tests

  • runner: the location of a ZIP file with XCTRunner app (e.g. TargetName-Runner.app). This is optional field, it is required only when you want to run UI tests.

  • xcTestBundle: the test bundle and test discovery mode for it. This is required field for all types of tests. This object is described by XcTestBundle:

{
    "xcTestBundle": {
        "testDiscoveryMode": "read more",
        "location": "http://host/file.zip#path/to/TestBundle.xctest"
    }
}
  • additionalApplicationBundles: this is an array of all possible apps your tests may use, e.g. they may launch these apps. This is required field for all types of tests, and if your tests do not deal with other apps, simply pass empty array here ([]). Otherwise, if your Xcode project has multiple buildable apps, and if your tests require them to be launched during test execution, you must pass here locations to ZIP archive for each required app. This field is similar to appBundle, it kind of extends it to support multiple apps launch during test run.

Below is a complete JSON for build artifacts that can be used to run UI tests:

{
    "appBundle": "http://example.com/MyApp.zip#MyApp.app",
    "runner": "http://example.com/MyAppUITests-Runner.zip#MyAppUITests-Runner.app",
    "xcTestBundle": {
        "testDiscoveryMode": "parseFunctionSymbols",
        "location": "http://example.com/UITests.xctest.zip#UITests.xctest"
    },
    "additionalApplicationBundles": []
}

Important note about URLs: any URL to any ZIP archive is expected to contain a reference to a file inside that archive. In fact, this URL is ResourceLocation entities. It is possible to append arbitrary HTTP headers as well, including for authentication. Please read more on URL Handling page.

environment

Defines test environment variables. Emcee does not read Xcode project or scheme settings, so you'll have to fill this field as needed.

This is a map from string keys to string values.

numberOfRetries

How many times failed tests should be attempted to run again. Allows to deal with flakiness. If you pass 0 (zero) here, Emcee will run test only once, that is, it will perform a single attempt to run test.

scheduleStrategy

Defines how array of tests will be split into buckets. Please read more here.

simulatorSettings

Object describes what settings should be applied to the simulators when running tests on them. Read more about this feature here.

testDestination

This object defines what device and runtime to use to run tests. The format is self-explanatory:

{"deviceType": "iPhone X", "runtime": "11.3"}

testTimeoutConfiguration

This object describes timeouts for each test in a job. The fields are:

  • singleTestMaximumDuration - maximum test duration in seconds. This is required field.

  • testRunnerMaximumSilenceDuration - maximum duration of silence of the runner process (xcodebuild) in seconds. For shorter tests (e.g. 5 minutes) you may set it equal to singleTestMaximumDuration, but if you have long running tests, you may want to set it to shorter amount. Thus, if runner process does not output anything to stdout/stderr, Emcee will fail the tests because of runner stall.

testType

Defines test type. The possible values are:

  • logicTest - these tests a being run under xctest process environment, they do not require app to be running
  • appTest - these tests are being run under your app environment, that is, the app itself becomes a test hosting app. This is suitable for gray/white box tests
  • uiTests - these tests are being run under XCTRunner.app process and they use IPC to communicate with main app. This is suitable for XC UI tests. Black box testing.

testsToRun

This array defines a list of predicates that will define a list of tests to run. You may pass the following values:

  • {"predicateType": "allDiscoveredTests"} - this will run all tests in xctest bundle. Usually you'll want to use this kind of predicate. You just pass an array with this single object in it.

  • {"predicateType": "singleTestName", "testName": "TestClassName/testMethod"} - this will run a single test TestClassName/testMethod. You can pass multiple objects of this kind and tell Emcee a concrete list of tests to execute.

This field allows you to setup a flexible test plan, e.g. you can repeat tests in it to make Emcee run a single test multiple times.

simulatorControlTool

Describes a tool to control simulators (create, boot, delete, shutdown) and where simulators are created. Possible values are:

  • {"location": "insideUserLibrary", "tool": {"toolType": "simctl"}} - tells Emcee to use xcrun simctl under DEVELOPER_DIR.

Note: fbsimctl is not supported anymore.

Emcee supports the following simulator location settings:

  • {"location": "insideUserLibrary"}: tells Emcee to create simulators in default location (~/Library/Developer/CoreSimulator/Devices). This is a preferred location.

  • {"location": "insideEmceeTempFolder"}: tells Emcee to create simulators in its temp folder (this mode is not compatible with xcodebuild test runner tool)

testRunnerTool

Describes a tool that allows Emcee to run tests on a booted simulator. Example values:

  • {"toolType": "xcodebuild"} - use xcodebuild to run tests.

Note: fbxctest is not supported anymore.

simulatorOperationTimeouts

Specifies simulator operation timeouts and controls automatic simulator shutdown feature. Example value:

{
    "simulatorOperationTimeouts": {
        "create": 20,
        "boot": 180,
        "shutdown": 20,
        "delete": 20,
        "automaticSimulatorShutdown": 600,
        "automaticSimulatorDelete": 600
    } 
}

These timeouts will be applied to a corresponding calls to xcrun simctl.

developerDir

Controls what Xcode environment should be used when running tests. Defining a correct Xcode environment is important for stabilizing tests. Emcee will use this setting to find Xcode. It will then append DEVELOPER_DIR environment to all invocations of xcrun, xcodebuild, etc.

Possible values are:

  • {"kind": "current"}: this value makes Emcee to use current Xcode version defined by xcode-select -p. This is okay if you have only a single Xcode installed on your workers, but always prefer to explicitly set Xcode version.

  • {"kind": "useXcode", "CFBundleShortVersionString": "11.5"}: this value makes Emcee to look for Xcode 11.5 (in /Applications) and fail test run if such Xcode does not exist.

testDestinationConfigurations

This setting defined specific outputs per test destination.

You may specify testDestinationConfigurations: [] if you don't want to have a specific outputs to be generated. Otherwise, specify a pair of test destination and report output objects. Emcee will iterate over test results and create a separate Junit for each test destination.

Example below tells Emcee to create 2 distinct Junit reports: one for tests executed on iPhone [email protected] simulator, and another one for tests executed on iPhone [email protected]:

{
    "testDestinationConfigurations": [
        {
            "testDestination": {"deviceType": "iPhone X", "runtime": "11.3"},
            "reportOutput": {"junit": "/iphone_x_11_3.junit.xml"}
        },
       {
            "testDestination": {"deviceType": "iPhone SE", "runtime": "10.3"},
            "reportOutput": {"junit": "/iphone_se_10_3.junit.xml"}
        }
    ]
}

pluginLocations

This is an array of URLs to ZIP archives with .emceeplugin bundles. Emcee will start plugins for this set of tests. Plugins will run on the workers. You can learn more about plugins.

{
    "pluginLocations": [
        "http://example.com/plugins/my_tms_plugin.zip#tms.emceeplugin"
    ]
}

If you don't have any plugins, just pass an empty array ([]).

Important note about URLs: any URL to any ZIP archive is expected to contain a reference to a file inside that archive. In fact, this URL is ResourceLocation entities. It is possible to append arbitrary HTTP headers as well, including for authentication. Please read more on URL Handling page.

Test Arg File Example

{
	"entries": [{
		"testsToRun": [
			{"predicateType": "allDiscoveredTests"}
		],
		"testDestination": {
			"deviceType": "iPhone 7", "runtime": "11.3"
		},
		"numberOfRetries": 5,
		"environment": {
			"SOME_ENV": "your tests will be able to read these envs via ProcessInfo.processInfo.environment"
		},
		"testType": "logicTest",
		"buildArtifacts": {
			"xcTestBundle": {
				"location": "http://example.com/file.zip#FunctionalTests.xctest",
				"testDiscoveryMode": "parseFunctionSymbols"
			},
			"additionalApplicationBundles": []
		},
		"simulatorControlTool": {
			"tool": {"toolType": "simctl"},
			"location": "insideUserLibrary"
		},
		"testRunnerTool": {
			"toolType": "xcodebuild"
		},
		"developerDir": {
			"kind": "useXcode",
			"CFBundleShortVersionString": "11.5"
		},
		"scheduleStrategy": "progressive",
		"simulatorSettings": {
			"simulatorLocalizationSettings": {
				"localeIdentifier": "ru_US",
				"keyboards":  ["ru_RU@sw=Russian;hw=Automatic", "en_US@sw=QWERTY;hw=Automatic"],
				"passcodeKeyboards": ["ru_RU@sw=Russian;hw=Automatic", "en_US@sw=QWERTY;hw=Automatic"],
				"languages": ["ru-US", "en", "ru-RU"],
				"addingEmojiKeybordHandled": true,
				"enableKeyboardExpansion": true,
				"didShowInternationalInfoAlert": true,
				"didShowContinuousPathIntroduction": true
			},
			"watchdogSettings": {
				"bundleIds": ["sample.app"],
				"timeout": 42
			}
		},
		"testTimeoutConfiguration": {
			"singleTestMaximumDuration": 20,
			"testRunnerMaximumSilenceDuration": 20
		},
		"simulatorOperationTimeouts": {
			"create": 20,
			"boot": 180,
			"shutdown": 20,
			"delete": 20,
			"automaticSimulatorShutdown": 600,
			"automaticSimulatorDelete": 600
		},
                "pluginLocations": []
	}],
	"priority": 500,
	"testDestinationConfigurations": []
}

Job Grouping and Priorities

Multiple jobs can be grouped into a logical job group by providing the same jobGroup value. Both jobIds and jpbGroupIds have priority field which controls how tests are performed in relation to each other.

Prioritization of scheduled tests works as follows:

  • Tests among all job groups with higher priority executed first

  • Tests within jobs with higher priority are executed first among other jobs

  • If jobs or job group have the same priority, the earlier created ones have higher priority over later created ones

Case Study

Imagine you have 2 test runs on your pull requests:

  • Run Unit Tests

  • Run e2e Tests

You use Pull Request ID as a group id when you schedule your tests.

Now, suppose two PRs are opened around the same time, PR_1 and PR_2.

Let's suppose the following jobs are scheduled in the order shown below:

  • PR_1 job group, Unit Tests job

  • PR_2 job group, Unit Tests job

  • PR_2 job group, e2e Tests job

  • PR_1 job group, e2e Tests job

Explanation:

  • PR_1 job group will have priority over PR_2 job group because it was created earlier

  • Unit Tests job within each job group will have priority over e1e Tests job because it was created earlier

So, Emcee will organise its queue as follows:

[Job Group PR_1: All tests of job Unit Tests]
[Job Group PR_1: All tests of job e2e Tests]
[Job Group PR_2: All tests of job Unit Tests]
[Job Group PR_2: All tests of job e2e Tests]

It means, even if workers will start executing PR_2 Unit Tests, if you schedule any tests into PR_1 job group, these tests will begin executing earlier than any other tests that queue has in it.

Also, it means that if at any moment of time you schedule tests with High Priority Job Group RELEASE: e2e tests, all current tests in the queue will be set aside, and workers will execute newly created High Priority Job Group RELEASE just because its priority is higher than one of Job Group PR_1 and Job Group PR_2.

Also, when tests are being retried due to flakiness, they are put into their job (and job group). This means their priority automatically inherits expected one, allowing them to be executed sooner that other tests that might have been submitted to the queue after these flaky tests have began executing.

Clone this wiki locally