TFX 1.9.0 Issues

[rtg0795]

Please comment or link any issues you find with TFX 1.9.0

Thanks.

[JPXKQX]

Issue #5039

[EdwardCuiPeacock]

Still running into problems with Dataflow jobs being stuck and killed after 1 hour. TFX 1.9.1 with Apache Beam 2.40.0, using the TFX docker image tensorflow/tfx:1.9.1

Error message from Dataflow:

Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.

Related to #4902 (comment)

Edit: Found the solution. It turns out it is related to this known issue. Setting the flag --experiments=disable_worker_container_image_prepull helped resolve the issue. TFX image has been getting larger and larger (TFX 1.9.1 around 17GB uncompressed), which means we would need to keep using this workaround in the future releases.

[KimuraTian]

Running data_view components on Vertex AI throws the same error as the first issue at 4472. Image version: tensorflow/tfx:1.9.1

[singhniraj08]

Issue #5039

@JPXKQX, 0c3d5dd is already merged for unique artifact name. Thank you!

Running data_view components on Vertex AI throws the same error as the first issue at 4472. Image version: tensorflow/tfx:1.9.1

@KimuraTian, Please refer this TensorFlow Ranking Example to implement data view component in TFX pipeline. Thank you!

Closing this due to inactivity. Please feel free to open new issue here: tfx/issues. Thank you!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No