Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

[web] Support -j to use goma in felt build #13259

Merged
mdebbar merged 3 commits into from
Oct 21, 2019
Merged

[web] Support -j to use goma in felt build #13259

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2dc0f06
[web] Support -j in felt build
mdebbar Oct 18, 2019
128fb9b
Apply suggestions
mdebbar Oct 21, 2019
52cd526
Rewrite the README section
mdebbar Oct 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions lib/web_ui/dev/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ felt build --watch

If you don't want to add `felt` to your path, you can still invoke it using a relative path like `./web_ui/dev/felt <command>`

## Using goma
Goma is a great way to speed up your builds. In order to use goma, you first need to install it. Here are the instructions for [linux](https://g3doc.corp.google.com/devtools/goma/g3doc/how-to-use-goma/how-to-install-goma-linux.md) and [mac](https://g3doc.corp.google.com/devtools/goma/g3doc/how-to-use-goma/how-to-install-goma-mac.md).
Copy link
Contributor

@yjbanov yjbanov Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not put internal links in our public docs. Instead, we could have a very short sentence sending googlers to goma, something like:

For googlers: accelerate your builds with goma

If you are a Google employee you can use an internal instance of goma to speed up your builds.

Copy link
Contributor Author

@mdebbar mdebbar Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


Once goma is installed, you can take advantage of it in your builds:
```
felt build [-w] -j 100
```
Copy link
Contributor

@yjbanov yjbanov Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for derailing this again, but now that we've figured out that -j is available to everyone, how about we make this useful for everyone? Strawman:

Speeding up your builds

You can speed up your builds by using more CPU cores. Pass -j to specify the desired level of parallelism, like so:

felt build [-w] -j 100

If you are a Google employee you can use an internal instance of Goma to parallelize your builds. Because Goma compiles code on remote servers, this option is effective even on low-powered laptops.

Copy link
Contributor Author

@mdebbar mdebbar Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing this up 😃


## Running web engine tests
To run all tests:
```
Expand Down
29 changes: 25 additions & 4 deletions lib/web_ui/dev/build.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ class BuildCommand extends Command<bool> {
abbr: 'w',
help: 'Run the build in watch mode so it rebuilds whenever a change'
'is made.',
)
..addOption(
'goma',
abbr: 'j',
help: 'Enable parallelization through goma',
Copy link
Contributor

@yjbanov yjbanov Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this is a goma option and not ninja in general? I thought even without goma this option would speed up your ninja builds by parallelizing across multiple CPU cores.

Copy link
Contributor Author

@mdebbar mdebbar Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. You are right, this is a ninja parallelization option that it inherited from Make. I'll update the text and variable names accordingly.

);
}

Expand All @@ -31,12 +36,20 @@ class BuildCommand extends Command<bool> {

bool get isWatchMode => argResults['watch'];

int get gomaWorkers {
final String gomaWorkersArg = argResults['goma'];
if (gomaWorkersArg != null) {
return int.parse(gomaWorkersArg);
Copy link
Contributor

@jonahwilliams jonahwilliams Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will throw if I pass -jk, use tryParse instead?

Copy link
Contributor Author

@mdebbar mdebbar Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want it to throw if you pass a non-integer.

Copy link
Contributor Author

@mdebbar mdebbar Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you think it's better to print a warning, and continue building without goma workers?

Copy link
Contributor

@jonahwilliams jonahwilliams Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea, people tend to file bugs when they see stack traces, regardless of the content. If this is only for the flutter web team I wouldn't worry about it

}
return null;
}

@override
FutureOr<bool> run() async {
final FilePath libPath = FilePath.fromWebUi('lib');
final Pipeline buildPipeline = Pipeline(steps: <PipelineStep>[
gn,
ninja,
() => ninja(gomaWorkers),
]);
await buildPipeline.start();

Expand Down Expand Up @@ -67,11 +80,17 @@ Future<void> gn() {
}

// TODO(mdebbar): Make the ninja step interruptable in the pipeline.
Future<void> ninja() {
print('Running ninja...');
Future<void> ninja(int gomaWorkers) {
if (gomaWorkers == null) {
print('Running ninja (with no goma workers)...');
} else {
print('Running ninja (with $gomaWorkers goma workers)...');
}

return runProcess('ninja', <String>[
'-C',
environment.hostDebugUnoptDir.path,
if (gomaWorkers != null) ...['-j', '$gomaWorkers'],
]);
}

Expand Down Expand Up @@ -106,8 +125,10 @@ class Pipeline {
await _currentStepFuture;
}
status = PipelineStatus.done;
} catch (_) {
} catch (error, stackTrace) {
status = PipelineStatus.error;
print('Error in the pipeline: $error');
print(stackTrace);
} finally {
_currentStepFuture = null;
}
Expand Down