Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions packages/imitation_game/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,3 +91,13 @@ An implementation has to follow these rules:
```

A single test run can report multiple numbers.

## Results
Date created: 2020-08-17 23:57:16.702500Z

- smiley
- flutter
- startupTime: 0.561475s
- uikit
- startupTime: 0.373102068901062s

Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import 'dart:async';
import 'dart:convert';
import 'dart:io';
import 'package:mustache/mustache.dart';
import 'package:imitation_game/README_template.dart';

const int _port = 4040;

Expand Down Expand Up @@ -33,6 +35,14 @@ Future<List<FileSystemEntity>> findFiles(Directory dir, {FileFilter where}) {
return completer.future;
}

String _makeMarkdownOutput(Map<String, dynamic> results) {
final Template template = Template(readmeTemplate, name: 'README.md');
final Map<String, dynamic> values = Map<String, dynamic>.from(results);
values['date'] = DateTime.now().toUtc();
final String output = template.renderString(values);
return output;
}

class _Script {
_Script({this.path});
String path;
Expand Down Expand Up @@ -75,6 +85,33 @@ class _ScriptRunner {
}
}

/// Recursively converts a map of maps to a map of lists of maps.
///
/// For example:
/// _map2List({'a': {'b': 123}}, ['foo', 'bar']) ->
/// {
/// 'foo':[
/// {
/// 'name': 'a',
/// 'bar': [{'name': 'b', 'value': 123}]
/// }
/// ]
/// }
Map<String, dynamic> _map2List(Map<String, dynamic> map, List<String> names) {
final List<Map<String, dynamic>> returnList = <Map<String, dynamic>>[];
final List<String> tail = names.sublist(1);
map.forEach((String key, dynamic value) {
final Map<String, dynamic> testResult = <String, dynamic>{'name': key};
if (tail.isEmpty) {
testResult['value'] = value;
} else {
testResult[tail.first] = _map2List(value, tail)[tail.first];
}
returnList.add(testResult);
});
return <String, dynamic>{names.first: returnList};
}

class _ImitationGame {
final Map<String, dynamic> results = <String, dynamic>{};
_ScriptRunner _scriptRunner;
Expand Down Expand Up @@ -163,8 +200,9 @@ Future<void> main() async {
keepRunning = await game.handleTimeout();
}
}
const JsonEncoder encoder = JsonEncoder.withIndent(' ');
final String jsonResults = encoder.convert(game.results);
print('$jsonResults');

final Map<String, dynamic> markdownValues =
_map2List(game.results, <String>['tests', 'platforms', 'measurements']);
File('README.md').writeAsStringSync(_makeMarkdownOutput(markdownValues));
await server.close(force: true);
}
108 changes: 108 additions & 0 deletions packages/imitation_game/lib/README_template.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
String readmeTemplate = """# Imitation Game

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultra nit: just let this be a .md file and read it to string for the Template call? Might be easier to read for contributors.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about it but reading from disk has some issues i can avoid by embedding the template into the binary. Unfortunately there doesn't seem to be a way to embed assets in dart that keeps it as a md file.


## Description

`imitation_game` is a platform for performing automated tests to compare the
performance of different UI frameworks. For example, how much memory does the
same app written in Flutter and UIKit take?

## Running all the tests

You need a mobile device plugged into your computer and setup for development.
The mobile device and the computer need to be on the same network, one that
allows communication between computers since that's how the mobile phone will
report its results to the computer.

```sh
dart imitation_game.dart
```

## Dependencies

In order to run the tests you will need the union of all the platforms being
tested. As new tests are added please add to this list:

### iOS

- Flutter
- Xcode
- [ios_deploy](https://github.com/ios-control/ios-deploy) - used to launch apps
on the attached iOS device.

## Example File Layout

```text
./
├─ imitation_game.dart
└─ tests/
├─ smiley/
│ ├─ README.md
│ ├─ flutter/
│ │ ├─ run_ios.sh
│ │ └─ <flutter project files>
│ └─ uikit/
│ ├─ run_ios.sh
│ └─ <uikit project files>
└─ memory/
├─ README.md
├─ flutter/
│ ├─ run_ios.sh
│ └─ <flutter project files>
└─ uikit/
├─ run_ios.sh
└─ <uikit project files>
```

Here there are 2 different tests with 2 different platform implementations. The
tests are named `smiley` and `memory`, they are both implemented on the
platforms `flutter` and `uikit`.

### Adding a test

Tests should comprise of implementations on one or more platform. The directory
for the test should be added to `./tests`. Inside that directory there should
be a directory of implementations and a `README.md` file that explains the test.

### Adding an implementation to a test

An implementation has to follow these rules:

- It needs to perform the same operations as the other implementations and
follow the description in the test's `README.md`.
- It needs to contain a `run_ios.sh` script that will build and launch the test
on the connected device.
- It should contain a file named `ip.txt` which will be overwritten by
`imitation_game.dart` with the ip address and port that should be used to
report results to.
- It needs to report its results to the ip and port in the `ip.txt` via an HTTP
POST of JSON data.

## Data format for results

```json
{
"test": "name_of_test",
"platform": "name_of_platform",
"results": {
"some_result_name": 1.23,
"some_result_name2": 4.56,
}
}
```

A single test run can report multiple numbers.

## Results
Date created: {{date}}

{{#tests}}
- {{name}}
{{#platforms}}
- {{name}}
{{#measurements}}
- {{name}}: {{value}}s
{{/measurements}}
{{/platforms}}
{{/tests}}

""";
3 changes: 3 additions & 0 deletions packages/imitation_game/pubspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
name: imitation_game
dependencies:
mustache: ^1.1.1