Fix broken AsyncInferenceClient on [DONE] signal #6394

Job	Run time
build-windows (3.8)	5m 6s
build-ubuntu (3.8, Repository only)	1m 35s
build-ubuntu (3.8, Everything else)	2m 59s
build-ubuntu (3.8, torch)	1m 6s
build-ubuntu (3.11, Repository only)	1m 32s
build-ubuntu (3.11, Everything else)	3m 0s
build-ubuntu (3.11, torch)	1m 23s
build-ubuntu (3.11, lfs)	59s
build-ubuntu (3.8, fastai)	1m 16s
build-ubuntu (3.10, fastai)	1m 5s
build-ubuntu (3.8, tensorflow)	57s
build-ubuntu (3.10, tensorflow)	49s
	21m 47s

Provide feedback