[CI/Build] CMakeLists: build all extensions' cmake targets at the same time

[dtrifiro]

Instead of building one target at a time, we can specify multiple --target flags simultaneously to cmake, this should speed up the build process a little.

[mgoin]

This seems straightforward to me, thanks! Only concern is possibly increasing memory requirements to build because of more targets in flight. Any thoughts @bnellnm ?

Edit: make sure to run format.sh so CI can run

[youkaichao]

I don't think this should be turned on by default. We tried very hard to not overwhelm user's machines:

vllm/setup.py

Lines 71 to 105 in 9197709

	#
	# Determine number of compilation jobs and optionally nvcc compile threads.
	#
	def compute_num_jobs(self):
	# `num_jobs` is either the value of the MAX_JOBS environment variable
	# (if defined) or the number of CPUs available.
	num_jobs = envs.MAX_JOBS
	if num_jobs is not None:
	num_jobs = int(num_jobs)
	logger.info("Using MAX_JOBS=%d as the number of jobs.", num_jobs)
	else:
	try:
	# os.sched_getaffinity() isn't universally available, so fall
	# back to os.cpu_count() if we get an error here.
	num_jobs = len(os.sched_getaffinity(0))
	except AttributeError:
	num_jobs = os.cpu_count()
	nvcc_threads = None
	if _is_cuda() and get_nvcc_cuda_version() >= Version("11.2"):
	# `nvcc_threads` is either the value of the NVCC_THREADS
	# environment variable (if defined) or 1.
	# when it is set, we reduce `num_jobs` to avoid
	# overloading the system.
	nvcc_threads = envs.NVCC_THREADS
	if nvcc_threads is not None:
	nvcc_threads = int(nvcc_threads)
	logger.info(
	"Using NVCC_THREADS=%d as the number of nvcc threads.",
	nvcc_threads)
	else:
	nvcc_threads = 1
	num_jobs = max(1, num_jobs // nvcc_threads)
	return num_jobs, nvcc_threads

If we compile all the targets by default, it will violate this.

We can make this an opt-in feature, and users can turn it on if they know what they are doing.

[bnellnm]

This seems straightforward to me, thanks! Only concern is possibly increasing memory requirements to build because of more targets in flight. Any thoughts @bnellnm ?

Edit: make sure to run format.sh so CI can run

I'm not too worried about overloading machines. Even with multiple targets the maximum number of jobs should be honored. The only problem I see with this is that each extension/target could conceivably have a different configuration. Currently, this is not the case but this change could cause problems if targets ever had different configs.

[dtrifiro]

@youkaichao

I'm not sure I follow. This should use max_jobs processes with the given number of nvcc threads, the only difference is that cmake --build will be only invoked once and will be aware of all of the targets that need to be built and not have "idle times" between finishing building all of the pieces of one extension and starting compiling the next one.

@mgoin

Regarding memory usage: I noticed a very slight increase in peak memory usage, but overall I think it's negligible and it's still limited by setting proper values for nvcc_threads and max_jobs. Empirically, I saw at most ~3.5GB memory for each nvcc threads, so, for something like MAX_JOBS=8 ( => -j 4) and NVCC_THREADS=2, I saw a peak of ~21GB of memory:

Here is an example of what a build looks like with these settings:

Without this patch, CPU usage (and memory as well) raises, stays at max for a while, then dwindles down until it finishes compiling one extension and then ramps up again for the next extension and so on (looks like the tail of the above cpu usage graph), resulting in some time intervals in which not all of the cores are being used. If I rember correctly, this patch improves build times by up to 20 minutes in some cases

[dtrifiro]

Any updates on this?

[mgoin]

Thank you @dtrifiro for the clear justification on why this patch improves CPU utilization. It is clear that if we run into peak memory consumption issues, we should continue to tune MAX_JOBS and NVCC_THREADS independently of compiling all targets at once.