Fix moving keys to device in ResultCollection #25899

Job	Run time
build-packages / init	5s
build-packages / build-packages (lightning)	25s
build-packages / build-packages (fabric)	34s
build-packages / build-packages (pytorch)	26s
install-pkg (ubuntu-22.04, fabric, 3.9)	1m 50s
install-pkg (ubuntu-22.04, fabric, 3.11)	1m 44s
install-pkg (ubuntu-22.04, pytorch, 3.9)	2m 0s
install-pkg (ubuntu-22.04, pytorch, 3.11)	1m 50s
install-pkg (ubuntu-22.04, lightning, 3.9)	8m 42s
install-pkg (ubuntu-22.04, lightning, 3.11)	2m 5s
install-pkg (ubuntu-22.04, notset, 3.9)	1m 49s
install-pkg (ubuntu-22.04, notset, 3.11)	1m 52s
install-pkg (macOS-12, fabric, 3.9)	1m 4s
install-pkg (macOS-12, fabric, 3.11)	1m 14s
install-pkg (macOS-12, pytorch, 3.9)	1m 22s
install-pkg (macOS-12, pytorch, 3.11)	1m 8s
install-pkg (macOS-12, lightning, 3.9)	1m 32s
install-pkg (macOS-12, lightning, 3.11)	1m 48s
install-pkg (macOS-12, notset, 3.9)	1m 54s
install-pkg (macOS-12, notset, 3.11)	1m 10s
install-pkg (windows-2022, fabric, 3.9)	2m 3s
install-pkg (windows-2022, fabric, 3.11)	2m 3s
install-pkg (windows-2022, pytorch, 3.9)	2m 1s
install-pkg (windows-2022, pytorch, 3.11)	2m 5s
install-pkg (windows-2022, lightning, 3.9)	6m 24s
install-pkg (windows-2022, lightning, 3.11)	2m 3s
install-pkg (windows-2022, notset, 3.9)	2m 26s
install-pkg (windows-2022, notset, 3.11)	2m 5s
	55m 44s

Provide feedback