Queries stuck in "FINISHING" state

[NMonnerSans]

Hello all!

I am experiencing an issue when trying to upgrade Trino from version 386 to 397. As stated in the issue title, some queries (specifically INSERT INTO queries) get stuck in FINISHING state indefinitely, and eventually collapse our metastore. This issue was also seen when trying version 395.

For reference, we are building Trino from the official Docker image, and using Nomad to create containers and scale our production clusters. The queries are INSERT INTO queries which affect Hive tables and create a single partition each time.

[raunaqmorarka]

It might help to use jstack on the coordinator when the query is stuck to collect stack traces and see which code paths we get stuck on.

[findepi]

cc @electrum for #14122 (v 396)
and @arhimondr for #13055 (v 391)

@NMonnerSans please attach the coordinator thread dump here