BigTable connector to TensorFlow 2.x

[pierreoberholzer]

System information

TensorFlow version (you are using): 2.x
Are you willing to contribute it (Yes/No): Potentially as a tester / documenter

Describe the feature and the current behavior/state.

On one hand TensorFlow is currently recognized as one of the main machine learning libraries [1], in particular for neural network families of models (including auto-encoders) that integrates with additional services (e.g. TFX [2]) available on GCP.
On the other hand, BigTable is an existing GCP product that handles tables with interesting properties [3]:

• Wide table up to million of columns
• High sparsity due to schema-free definition and zero-cost saving

Training a machine learning model on such raw data would require to reduce dimension, which is one specific use case where TensorFlow is good at. Precisely, the training of deep and WIDE model would be in scope [4].

Combining both technologies seems to be a great fit.

Will this change the current api? How?

Can't say

Who will benefit with this feature?

Data engineering/data science community using GCP and sourcing BigTable data inside TensorFlow, typically for dimensionality reduction and sparse data use cases (e.g fraud detection that leverages dynamically aggregated values [5]).

[1] https://www.upgrad.com/blog/top-deep-learning-frameworks/
[2] https://www.tensorflow.org/tfx
[3] https://cloud.google.com/bigtable/docs/schema-design
[4] https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html
[5] https://cloud.google.com/solutions/machine-learning/minimizing-predictive-serving-latency-in-machine-learning#precomputing_and_caching_predictions

[yongtang]

@pierreoberholzer wondering if there is any advantages using BigTable compared to BigQuery? BigQuery is also used widely in GCP and there is already integrations with tensorflow-io.

[pierreoberholzer]

@yongtang BigQuery has a limit of 10'000 columns that BigTable hasn't [1] (plus higher storage costs for sparse table I assume).

[1] https://stackoverflow.com/questions/65416119/bigquery-how-to-go-beyond-10000-columns/65553624#65553624

[yongtang]

PR #1286 adds a stub emulator setup for test. Implementation will be added later.

[sshrdp]

@yongtang, Is this something you intend to continue working on? We'd love to support if so.

[yongtang]

@sshrdp I have been mostly working on modular file systems recently due to the upcoming 2.5 release, and don't have a lot of time to work on bigtable support yet. Would be glad if someone else or community can work on this item in the meantime. I am happy to provide additional help if needed.

[vlasenkoalexey]

FYI, there was a BT connector implementation, also it was a broken for a while.
I managed to fix it #278, but got pushback from SREs to revert the fix. I didn't have time to figure out the proper fix for it.
Eventually implementation was removed since nobody was using it and code was not maintained.
If you need to support it, you can try pulling code from some old branch.

In my experience, BT is very low level and a bit clunky to work with especially for experimentation.
Also it has other benefits.

[yongtang]

This issue has been addressed in #1578