Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Build Web SDK in wasm build tree #36520

Merged
auto-submit merged 19 commits into from
Oct 14, 2022
Merged

Build Web SDK in wasm build tree #36520

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
543b02d
Build web sdk in wasm tree.
eyebrowsoffire Oct 7, 2022
07690ee
Use prebuilts to build tests.
eyebrowsoffire Oct 8, 2022
2259916
Don't create the copy target if `flutter_prebuilt_dart_sdk` is false
eyebrowsoffire Oct 8, 2022
d1a9f0a
Felt test without build.
eyebrowsoffire Oct 8, 2022
4650b9e
Remove debug statement I accidentally left in.
eyebrowsoffire Oct 8, 2022
1041f93
Refactored ddc and outline generation in gn files.
eyebrowsoffire Oct 10, 2022
e4f0c0d
Remove dependency on dart-sdk copy.
eyebrowsoffire Oct 10, 2022
81eff9c
Formatting stuff.
eyebrowsoffire Oct 10, 2022
82812b6
Use host_prebuilt_dart_sdk instead of just prebuilt_dart_sdk
eyebrowsoffire Oct 10, 2022
f3b6cfd
Don't make the flutter_web_sdk artifact on android builds, is isn't u…
eyebrowsoffire Oct 10, 2022
a9e7f3f
Change strategy for short-circuiting the flutter_web_sdk archive.
eyebrowsoffire Oct 11, 2022
9d1d6ae
Merge branch 'main' into web_sdk_build
eyebrowsoffire Oct 11, 2022
8bfb22f
Don't produce a pointless xcode/VS project when building wasm.
eyebrowsoffire Oct 11, 2022
094c678
Change up the way we are treating flags, and package canvaskit with t…
eyebrowsoffire Oct 13, 2022
d81efbd
Force `--build-canvaskit` until we update the recipes.
eyebrowsoffire Oct 13, 2022
a0d5345
Merge branch 'main' into web_sdk_build
eyebrowsoffire Oct 13, 2022
aeaa119
Oops, only build canvaskit in the web case.
eyebrowsoffire Oct 14, 2022
3d667d2
flutter_web_sdk need not be built on in linux_host_release.
eyebrowsoffire Oct 14, 2022
924697d
Change target for linux_host_engine drone.
eyebrowsoffire Oct 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 1 addition & 25 deletions BUILD.gn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,30 +44,6 @@ config("export_dynamic_symbols") {
}
}

if (flutter_prebuilt_dart_sdk) {
copy_trees("_copy_trees") {
sources = [
{
target = "copy_dart_sdk"
visibility = [ ":dart_sdk" ]
source = prebuilt_dart_sdk
dest = "$root_out_dir/dart-sdk"
ignore_patterns = "{}"
},
]
}
}

group("dart_sdk") {
if (build_engine_artifacts) {
if (flutter_prebuilt_dart_sdk) {
public_deps = [ ":copy_dart_sdk" ]
} else {
public_deps = [ "//third_party/dart:create_sdk" ]
}
}
}

group("flutter") {
testonly = true

Expand Down Expand Up @@ -96,7 +72,7 @@ group("flutter") {

if (build_engine_artifacts) {
public_deps += [
":dart_sdk",
"//flutter/dart_sdk",
"//flutter/flutter_frontend_server:frontend_server",

# This must be listed explicitly for desktop cross-builds since
Expand Down
2 changes: 1 addition & 1 deletion common/config.gni
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ if (flutter_prebuilt_dart_sdk) {
# There is no prebuilt Dart SDK targeting Fuchsia, but we also don't need
# one, so even when the build is targeting Fuchsia, use the prebuilt
# Dart SDK for the host.
if (current_toolchain == host_toolchain || is_fuchsia) {
if (current_toolchain == host_toolchain || is_fuchsia || is_wasm) {
prebuilt_dart_sdk = host_prebuilt_dart_sdk
prebuilt_dart_sdk_config = _host_prebuilt_dart_sdk_config
} else {
Expand Down
24 changes: 24 additions & 0 deletions dart_sdk/BUILD.gn
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import("//flutter/common/config.gni")

@zanderso zanderso Oct 14, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file needs a copyright header

@zanderso zanderso Oct 17, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things:

  1. What is the reason for moving this?
  2. GN files need a copyright header.

@eyebrowsoffire eyebrowsoffire Oct 17, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original reason for moving this was that the flutter/BUILD.gn file is not wasm friendly (it transitively pulls in a bunch of .gni files that have platform checks that choke when the target os/arch is wasm). Since I was trying to build the web sdk in the wasm build, it was pulling in the flutter/BUILD.gn file when it had a dependency on the dart-sdk target, so I separated it out and simplified it so that we wouldn't have to pull in the whole of flutter/BUILD.gn when building the web sdk.

After some additional changes though, the web sdk build now uses the prebuilt sdk in-place and so does not have a dependency on the dart_sdk target anymore, so I can actually move it back if this bothers you.

@yjbanov yjbanov Oct 17, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to allow Dart SDK hackers to test Dart SDK builds with local source changes, so we may end up with this dependency again. If it's better to have it split out into a standalone BUILD.gn I think I'd rather keep it.

@zanderso zanderso Oct 17, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to allow Dart SDK hackers to test Dart SDK builds with local source changes, so we may end up with this dependency again. If it's better to have it split out into a standalone BUILD.gn I think I'd rather keep it.

I don't follow how that's related. The existing way to hack on the Dart SDK in the engine tree is to make changes under third_party/dart, and to use the --no-prebuilt-dart-sdk flag to the gn script. Does that not work for some reason?

@eyebrowsoffire eyebrowsoffire Oct 18, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the dart-sdk target is in flutter/BUILD.gn, then the wasm build cannot consume or use it because the wasm build can't actually touch flutter/BUILD.gn without pulling in transitive dependencies that break under a wasm build. The transitive dependencies break even if you aren't building any of their targets, and they are dependencies we don't necessarily have control over.

That being said, we aren't actually using the dart-sdk target for the web build anymore, we just use prebuilts. I also don't actually see us adding a dependency on the dart-sdk target in the future either, and if we did it would be in a different manner that would probably build it with the non-default toolchain, since it's weird to pull host artifacts into the wasm build tree. So actually for our purposes I don't care anymore, and if you want me to refactor that target back into flutter/BUILD.gn I don't have any real practical objection, but I don't actually understand why that would be desirable. flutter/BUILD.gn is a bit of a kitchen sink and probably would be easier to work with if it were broken up into smaller logical components, and so having the dart-sdk separated into a different file seems better to me than keeping it in flutter/BUILD.gn. But again we don't really depend on either any more, so I don't mind either way you want to go with it.

import("//third_party/dart/build/dart/copy_tree.gni")

if (flutter_prebuilt_dart_sdk) {
copy_trees("_copy_trees") {
sources = [
{
target = "copy_dart_sdk"
visibility = [ ":dart_sdk" ]
source = prebuilt_dart_sdk
dest = "$root_out_dir/dart-sdk"
ignore_patterns = "{}"
},
]
}
}

group("dart_sdk") {
if (flutter_prebuilt_dart_sdk) {
public_deps = [ ":copy_dart_sdk" ]
} else {
public_deps = [ "//third_party/dart:create_sdk" ]
}
}
40 changes: 7 additions & 33 deletions lib/web_ui/dev/build.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
// found in the LICENSE file.

import 'dart:async';
import 'dart:io' show Directory, Platform;
import 'dart:io' show Directory;

import 'package:args/command_runner.dart';
import 'package:path/path.dart' as path;
Expand Down Expand Up @@ -43,14 +43,8 @@ class BuildCommand extends Command<bool> with ArgUtils<bool> {
final FilePath libPath = FilePath.fromWebUi('lib');
final List<PipelineStep> steps = <PipelineStep>[
GnPipelineStep(),
NinjaPipelineStep(target: environment.engineBuildDir),
NinjaPipelineStep(target: environment.wasmReleaseOutDir),
];
if (buildCanvasKit) {
steps.addAll(<PipelineStep>[
GnPipelineStep(target: 'canvaskit'),
NinjaPipelineStep(target: environment.wasmReleaseOutDir),
]);
}
final Pipeline buildPipeline = Pipeline(steps: steps);
await buildPipeline.run();

Expand All @@ -73,39 +67,19 @@ class BuildCommand extends Command<bool> with ArgUtils<bool> {
/// Not safe to interrupt as it may leave the `out/` directory in a corrupted
/// state. GN is pretty quick though, so it's OK to not support interruption.
class GnPipelineStep extends ProcessStep {
GnPipelineStep({this.target = 'engine'})
: assert(target == 'engine' || target == 'canvaskit');

@override
String get description => 'gn';

@override
bool get isSafeToInterrupt => false;

/// The target to build with gn.
///
/// Acceptable values: engine, canvaskit
final String target;

@override
Future<ProcessManager> createProcess() {
print('Running gn for $target...');
final List<String> gnArgs = <String>[];
if (target == 'engine') {
gnArgs.addAll(<String>[
'--unopt',
if (Platform.isMacOS) '--xcode-symlinks',
'--full-dart-sdk',
if (environment.isMacosArm) '--mac-cpu=arm64',
]);
} else if (target == 'canvaskit') {
gnArgs.addAll(<String>[
'--wasm',
'--runtime-mode=release',
]);
} else {
throw StateError('Target was not engine or canvaskit: $target');
}
print('Running gn...');
const List<String> gnArgs = <String>[
'--wasm',

@mdebbar mdebbar Oct 8, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this target is going to be used for all of flutter web (not just for wasm), does it make sense to change the name to flutter_web or similar?

@eyebrowsoffire eyebrowsoffire Oct 8, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there was some extensive discussion about this with the rest of the engine team, I'd like to ask @hterkelsen what their concerns were and why they landed on this. In any case, the --wasm can't be immediately deleted because the recipes use it, but if we are going to migrate to some different flag I should at least add it as an option here so the recipes can switch to it.

@harryterkelsen harryterkelsen Oct 13, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern with using --wasm to mean "build the Flutter Web SDK" is that the Flutter Web SDK bits aren't WASM (for now, anyway). The fact that we can invoke kernel compiler when using the WASM toolchain seems to just be a lucky coincidence. I think we should have a --web flag in GN which sets the default CPU to WASM and builds CanvasKit and the Flutter Web SDK.

@eyebrowsoffire eyebrowsoffire Oct 13, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. I think I get what you're going for. Let me make some changes and see what I can do.

'--runtime-mode=release',
];
return startProcess(
path.join(environment.flutterDirectory.path, 'tools', 'gn'),
gnArgs,
Expand Down
15 changes: 3 additions & 12 deletions lib/web_ui/dev/environment.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,17 @@ Environment? _environment;
class Environment {
factory Environment() {
final bool isMacosArm = ffi.Abi.current() == ffi.Abi.macosArm64;
final io.File dartExecutable = io.File(io.Platform.resolvedExecutable);
final io.File self = io.File.fromUri(io.Platform.script);

final io.Directory engineSrcDir = self.parent.parent.parent.parent.parent;
final io.Directory engineToolsDir =
io.Directory(pathlib.join(engineSrcDir.path, 'flutter', 'tools'));
final io.Directory outDir =
io.Directory(pathlib.join(engineSrcDir.path, 'out'));
final io.Directory engineBuildDir =
io.Directory(pathlib.join(
outDir.path,
isMacosArm ? 'host_debug_unopt_arm64' : 'host_debug_unopt'
));
final io.Directory wasmReleaseOutDir =
io.Directory(pathlib.join(outDir.path, 'wasm_release'));
final io.Directory dartSdkDir =
io.Directory(pathlib.join(engineBuildDir.path, 'dart-sdk'));
final io.Directory dartSdkDir = dartExecutable.parent.parent;
final io.Directory webUiRootDir = io.Directory(
pathlib.join(engineSrcDir.path, 'flutter', 'lib', 'web_ui'));

Expand All @@ -55,7 +51,6 @@ class Environment {
engineSrcDir: engineSrcDir,
engineToolsDir: engineToolsDir,
outDir: outDir,
engineBuildDir: engineBuildDir,
wasmReleaseOutDir: wasmReleaseOutDir,
dartSdkDir: dartSdkDir,
);
Expand All @@ -68,7 +63,6 @@ class Environment {
required this.engineSrcDir,
required this.engineToolsDir,
required this.outDir,
required this.engineBuildDir,
required this.wasmReleaseOutDir,
required this.dartSdkDir,
});
Expand All @@ -93,9 +87,6 @@ class Environment {
/// This is where you'll find the ninja output, such as the Dart SDK.
final io.Directory outDir;

/// The output directory for the host_debug_unopt build.
final io.Directory engineBuildDir;

/// The output directory for the wasm_release build.
///
/// We build CanvasKit in release mode to reduce code size.
Expand Down
1 change: 1 addition & 0 deletions tools/gn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,7 @@ def to_gn_wasm_args(args, gn_args):
gn_args['skia_canvaskit_enable_webgpu'] = False
is_profile_build = args.runtime_mode == 'profile' or args.runtime_mode == 'debug'
gn_args['skia_canvaskit_profile_build'] = is_profile_build
gn_args['flutter_prebuilt_dart_sdk'] = True


def parse_args(args):
Expand Down
5 changes: 4 additions & 1 deletion wasm/BUILD.gn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,8 @@

# This is the default target when building when the target CPU is WASM.
group("wasm") {
deps = [ "//third_party/skia/modules/canvaskit" ]
deps = [
"//flutter/web_sdk",
"//third_party/skia/modules/canvaskit",
]
}
8 changes: 4 additions & 4 deletions web_sdk/BUILD.gn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ template("_dartdevc") {
":web_engine_library",
":web_ui_library",
":web_ui_library_sources",
"//flutter:dart_sdk",
"//flutter/dart_sdk",
]
script = "//build/gn_run_binary.py"

Expand Down Expand Up @@ -184,7 +184,7 @@ template("_dartdevc") {
":web_engine_library",
":web_ui_library",
":web_ui_library_sources",
"//flutter:dart_sdk",
"//flutter/dart_sdk",
"//third_party/dart/pkg:pkg_files_stamp",
"//third_party/dart/utils/dartdevc:dartdevc_files_stamp",
"//third_party/dart/utils/dartdevc:dartdevc_sdk_patch_stamp",
Expand Down Expand Up @@ -215,7 +215,7 @@ template("_kernel_worker") {
":web_engine_library",
":web_ui_library",
":web_ui_library_sources",
"//flutter:dart_sdk",
"//flutter/dart_sdk",
]
script = "//build/gn_run_binary.py"

Expand Down Expand Up @@ -247,7 +247,7 @@ template("_kernel_worker") {
":web_engine_library",
":web_ui_library",
":web_ui_library_sources",
"//flutter:dart_sdk",
"//flutter/dart_sdk",
"//third_party/dart/pkg:pkg_files_stamp",
"//third_party/dart/utils/dartdevc:dartdevc_files_stamp",
"//third_party/dart/utils/dartdevc:dartdevc_sdk_patch_stamp",
Expand Down
10 changes: 0 additions & 10 deletions web_sdk/web_test_utils/lib/environment.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,21 +41,17 @@ class Environment {
required this.webUiRootDir,
required this.engineSrcDir,
required this.engineToolsDir,
required this.outDir,
});

static Environment _prepareEnvironmentFromEngineDir(
io.File self, io.Directory engineSrcDir) {
final io.Directory engineToolsDir =
io.Directory(pathlib.join(engineSrcDir.path, 'flutter', 'tools'));
final io.Directory outDir =
io.Directory(pathlib.join(engineSrcDir.path, 'out'));
final io.Directory webUiRootDir = io.Directory(
pathlib.join(engineSrcDir.path, 'flutter', 'lib', 'web_ui'));

for (final io.Directory expectedDirectory in <io.Directory>[
engineSrcDir,
outDir,
webUiRootDir
]) {
if (!expectedDirectory.existsSync()) {
Expand All @@ -68,7 +64,6 @@ class Environment {
webUiRootDir: webUiRootDir,
engineSrcDir: engineSrcDir,
engineToolsDir: engineToolsDir,
outDir: outDir,
);
}

Expand All @@ -84,11 +79,6 @@ class Environment {
/// Path to the engine's "tools" directory.
final io.Directory engineToolsDir;

/// Path to the engine's "out" directory.
///
/// This is where you'll find the ninja output, such as the Dart SDK.
final io.Directory outDir;

/// Path to where github.com/flutter/engine is checked out inside the engine workspace.
io.Directory get flutterDirectory =>
io.Directory(pathlib.join(engineSrcDir.path, 'flutter'));
Expand Down