Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rustc takes 3.5 GB RSS to check vtparse #81124

Open
jyn514 opened this issue Jan 17, 2021 · 3 comments
Open

Rustc takes 3.5 GB RSS to check vtparse #81124

jyn514 opened this issue Jan 17, 2021 · 3 comments
Labels
A-borrow-checker Area: The borrow checker C-bug Category: This is a bug. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-compiler-performance Working group: Compiler Performance

Comments

@jyn514
Copy link
Member

jyn514 commented Jan 17, 2021

I tried this code: wez/wezterm@fa4bbbd

I expected to see this happen: The compiler builds the code in a reasonable amount of memory (say, < 800 MB).

Instead, this happened: The compiler uses over 3 GB, causing docs.rs to kill the process: rust-lang/docs.rs#1247

Meta

rustc --version --verbose:

> rustc --version -v
rustc 1.51.0-nightly (a62a76047 2021-01-13)
binary: rustc
commit-hash: a62a76047ea24aad7639f14eb3ce0e620b77bdb7
commit-date: 2021-01-13
host: x86_64-unknown-linux-gnu
release: 1.51.0-nightly
-Z time-passes 
> cargo rustc --profile=check -- -Z time-passes
    Checking utf8parse v0.2.0
    Checking vtparse v0.4.0 (/home/joshua/test-rustdoc/wezterm/vtparse)
time: 0.002; rss: 52MB	parse_crate
time: 0.000; rss: 52MB	attributes_injection
time: 0.000; rss: 53MB	incr_comp_prepare_session_directory
time: 0.000; rss: 53MB	incr_comp_garbage_collect_session_directories
time: 0.000; rss: 53MB	recursion_limit
time: 0.000; rss: 53MB	plugin_loading
time: 0.000; rss: 53MB	plugin_registration
time: 0.000; rss: 53MB	pre_AST_expansion_lint_checks
time: 0.000; rss: 53MB	crate_injection
time: 0.000; rss: 56MB	pre_AST_expansion_lint_checks
time: 0.000; rss: 56MB	pre_AST_expansion_lint_checks
time: 0.174; rss: 83MB	expand_crate
time: 0.000; rss: 83MB	check_unused_macros
time: 0.174; rss: 83MB	macro_expand_crate
time: 0.000; rss: 83MB	maybe_building_test_harness
time: 0.002; rss: 83MB	AST_validation
time: 0.000; rss: 83MB	maybe_create_a_macro_crate
time: 0.002; rss: 88MB	complete_gated_feature_checking
time: 0.196; rss: 88MB	configure_and_expand
time: 0.000; rss: 88MB	prepare_outputs
time: 0.000; rss: 88MB	blocked_on_dep_graph_loading
time: 0.013; rss: 97MB	hir_lowering
time: 0.004; rss: 97MB	early_lint_checks
time: 0.000; rss: 98MB	setup_global_ctxt
time: 0.001; rss: 98MB	dep_graph_tcx_init
time: 0.002; rss: 98MB	create_global_ctxt
time: 0.000; rss: 103MB	looking_for_entry_point
time: 0.000; rss: 103MB	looking_for_plugin_registrar
time: 0.000; rss: 103MB	looking_for_derive_registrar
time: 0.009; rss: 108MB	misc_checking_1
time: 0.007; rss: 116MB	type_collecting
time: 0.000; rss: 116MB	impl_wf_inference
time: 0.000; rss: 150MB	unsafety_checking
time: 0.000; rss: 150MB	orphan_checking
time: 0.059; rss: 150MB	coherence_checking
time: 0.016; rss: 156MB	wf_checking
time: 0.572; rss: 221MB	item_types_checking
time: 0.064; rss: 245MB	item_bodies_checking
time: 0.719; rss: 245MB	type_check_crate
time: 0.002; rss: 245MB	match_checking
time: 0.004; rss: 245MB	liveness_and_intrinsic_checking
time: 0.006; rss: 245MB	misc_checking_2
time: 2.499; rss: 1910MB	MIR_borrow_checking
time: 0.761; rss: 3685MB	MIR_effect_checking
time: 0.000; rss: 3685MB	layout_testing
time: 0.002; rss: 3685MB	death_checking
time: 0.001; rss: 3685MB	unused_lib_feature_checking
time: 1.156; rss: 3393MB	crate_lints
time: 0.007; rss: 3393MB	module_lints
time: 1.164; rss: 3393MB	lint_checking
time: 0.006; rss: 3393MB	privacy_checking_modules
time: 1.176; rss: 3393MB	misc_checking_3
time: 0.016; rss: 3393MB	generate_crate_metadata
time: 0.004; rss: 3393MB	codegen_crate
time: 0.000; rss: 3393MB	assert_dep_graph
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::type_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::generics_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::predicates_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_const_qualif)
time: 0.020; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_for_ctfe)
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::optimized_mir)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::coverageinfo)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::covered_file_name)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::covered_code_regions)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::promoted_mir)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::unsafety_check_result)
time: 0.002; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::typeck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::diagnostic_only_typeck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::used_trait_imports)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_borrowck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::eval_to_allocation_raw)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::eval_to_const_value_raw)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::check_match)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::symbol_name)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::codegen_fn_attrs)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::codegen_fulfill_obligation)
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::specialization_graph_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::adt_drop_tys)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::unused_generic_params)
time: 0.026; rss: 3394MB	encode_query_results
time: 0.028; rss: 3394MB	incr_comp_serialize_result_cache
time: 0.028; rss: 3394MB	incr_comp_persist_result_cache
time: 0.000; rss: 3394MB	incr_comp_serialize_dep_graph
time: 0.002; rss: 3394MB	incr_comp_encode_serialized_dep_graph
time: 0.003; rss: 3394MB	incr_comp_encode_dep_graph
time: 0.003; rss: 3394MB	incr_comp_persist_dep_graph
time: 0.031; rss: 3394MB	serialize_dep_graph
time: 0.010; rss: 3330MB	free_global_ctxt
time: 0.000; rss: 3330MB	join_worker_thread
time: 0.001; rss: 3330MB	copy_all_cgu_workproducts_to_incr_comp_cache_dir
time: 0.001; rss: 3330MB	finish_ongoing_codegen
time: 0.000; rss: 3330MB	llvm_dump_timing_file
time: 0.000; rss: 3330MB	serialize_work_products
time: 0.000; rss: 3330MB	incr_comp_finalize_session_directory
time: 0.000; rss: 3330MB	link_binary_check_files_are_writeable
time: 0.000; rss: 3330MB	link_binary_remove_temps
time: 0.000; rss: 3330MB	link_binary
time: 0.000; rss: 3330MB	link_crate
time: 0.002; rss: 3330MB	link
time: 5.477; rss: 3330MB		total
    Finished check [unoptimized + debuginfo] target(s) in 6.10s

The main offenders that stand out are

time: 2.499; rss: 1910MB	MIR_borrow_checking
time: 0.761; rss: 3685MB	MIR_effect_checking

which spike up from time: 0.006; rss: 245MB misc_checking_2 before. It would be nice to reduce this somehow.
@jyn514 jyn514 added A-borrow-checker Area: The borrow checker T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. C-bug Category: This is a bug. WG-compiler-performance Working group: Compiler Performance labels Jan 17, 2021
@jyn514 jyn514 mentioned this issue Jan 17, 2021
Closed
@Aaron1011
Copy link
Member

It looks like the high memory usage is being driven by the MIR for the TRANSITIONS static: https://github.com/wez/wezterm/blob/fa4bbbd077837d4eafe26805079f95a51054c4d7/vtparse/src/transitions.rs#L300-L315

It has 33,826 (!!!) basic blocks.

@Aaron1011
Copy link
Member

Aaron1011 commented Jan 17, 2021 •
edited
Loading

Running cargo expand shows that the large number of basic blocks is caused by the initializer being enormous: https://gist.github.com/Aaron1011/73c3f35c6aa1b07bf5a5d2240bc71b19

@jyn514 jyn514 mentioned this issue Jan 19, 2021
Closed
@bugadani
Copy link
Contributor

In this concrete case the memory mostly goes to the dataflow framework. The dataflow framework allocates a vector of bitsets for the entire universe. This bitset vector is number of move paths wide and number of basic blocks long. This particular initializer has 33.826 basic blocks and 124559 move paths, which means a single dataflow pass (MaybeUninitializedPlaces, MaybeInitializedPlaces, EverInitializedPlaces) each allocate around 500 MB of memory.
I'm not sure why this memory is not released, or if it is, it's release is not reflected in the rss numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-borrow-checker Area: The borrow checker C-bug Category: This is a bug. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-compiler-performance Working group: Compiler Performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bugadani @Aaron1011 @jyn514