GCS High Performance Parallel Listing

[hanseaston]

Hello Martin,

I am currently a Google SWE intern, and I am working as a part of the GCS team.

We are thinking of optimizing the listing operation in GCSFS, and want to get your initial approval on this.

In particular, we are thinking of utilizing multiple processes (using the concurrency.future library) and an optional GCS Insights Service to speed up the listing operation. Currently, the listing operation takes about 300 seconds to list 1 million objects. With the optimization, the listing operation can speed up 10x using the same setup. We have already experimented internally.

The Insights Service will be an optional configuration that the client can pass into the GCSFS listing call, so everything should be backwards compatible.

Let me know your initial thoughts. I will be working on the codebase and should hopefully have a PR out in the next few weeks.

My email is open: [email protected], if you ever have anything you wish to discuss offline.