Add preliminary support of OpenVINO as Keras 3 backend

[rkazants]

Details: Support OpenVINO as Keras 3 backend. This is inference-only backend. In order to switch on this, define environment variable as follows: os.environ["KERAS_BACKEND"] = "openvino" or use set_backend.

Here is an example how it works:
Install OpenVINO pip install openvino -U

import os
os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras

# build a simple network
x1 = keras.Input(shape=(2,), dtype='float32')
x2 = keras.Input(shape=(2,), dtype='float32')
add = keras.layers.Add()([x1, x2])
sigmoid = keras.activations.sigmoid(add)
model = keras.Model(inputs={'x1': x1, 'x2': x2}, outputs={'sigmoid': sigmoid})
model.summary()

# infer using OpenVINO
out = model.predict({'x1': np.array([[1.0, 2.0]], dtype=np.float32),
                     'x2': np.array([[3.0, 4.0]], dtype=np.float32)})
print('openvino out = ', out)

Added support for operations that allow to infer mobilenet_v3_small and bert_base_en_uncased from Keras Hub.

Example 1:

# from here https://keras.io/api/keras_nlp/models/bert/bert_text_classifier/#from_preset-method
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
#import keras_hub

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}

# Pretrained classifier without preprocessing.
#classifier = keras_hub.models.BertTextClassifier.from_preset(
#    "bert_base_en_uncased",
#    num_classes=4,
#    preprocessor=None,
#)
#keras.saving.save_model(classifier, "bert_base_en_uncased.keras")
classifier = keras.saving.load_model("bert_base_en_uncased.keras")
#output = classifier.predict(x=features, batch_size=2)
output = classifier.predict(features)

Example 2:

# from here https://keras.io/api/keras_cv/models/backbones/mobilenet_v3/
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
import keras_cv

rng = np.random.default_rng(23345243)

#input_data = np.random.rand(8, 224, 224, 3)
input_data = rng.uniform(0.0, 1.0, [8, 224, 224, 3]).astype(np.float32)
model = keras.saving.load_model("mobilenet_v3_small.keras")
#model = keras_cv.models.MobileNetV3Backbone.from_preset("mobilenet_v3_small",)

output = model.predict(input_data)

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fchollet]

Thanks for the PR!

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0.19194% with 1040 lines in your changes missing coverage. Please review.

Project coverage is 80.42%. Comparing base (0078f24) to head (9162c15).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
keras/src/backend/openvino/numpy.py	0.00%	396 Missing ⚠️
keras/src/backend/openvino/core.py	0.00%	222 Missing ⚠️
keras/src/backend/openvino/nn.py	0.00%	180 Missing ⚠️
keras/src/backend/openvino/trainer.py	0.00%	116 Missing ⚠️
keras/src/backend/openvino/random.py	0.00%	40 Missing ⚠️
keras/src/backend/openvino/math.py	0.00%	28 Missing ⚠️
keras/src/backend/openvino/__init__.py	0.00%	24 Missing ⚠️
keras/src/backend/openvino/linalg.py	0.00%	12 Missing ⚠️
keras/src/backend/openvino/rnn.py	0.00%	7 Missing ⚠️
keras/src/backend/openvino/image.py	0.00%	4 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19727      +/-   ##
==========================================
- Coverage   82.17%   80.42%   -1.76%     
==========================================
  Files         515      526      +11     
  Lines       47956    49023    +1067     
  Branches     7500     7603     +103     
==========================================
+ Hits        39409    39427      +18     
- Misses       6734     7779    +1045     
- Partials     1813     1817       +4     

Flag	Coverage Δ
keras	80.27% <0.19%> (-1.75%)	⬇️
keras-jax	63.73% <0.09%> (-1.39%)	⬇️
keras-numpy	58.82% <0.19%> (-1.29%)	⬇️
keras-tensorflow	64.69% <0.09%> (-1.41%)	⬇️
keras-torch	63.67% <0.09%> (-1.38%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gbaned]

Hi @rkazants Can you please resolve the conflicts? Thank you!

[rkazants]

Hi @rkazants Can you please resolve the conflicts? Thank you!

I am working on this PR. I am trying to resolve comments.

Best regards,
Roman

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[rkazants]

I am just back from vacation. I plan to continue from the next week.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[rkazants]

Hi @rkazants Any update on this PR? Please. Thank you!

Hi @gbaned, please anticipate the update early this week. Sorry for the delay.

Thanks,
Roman

[rkazants]

Hi @fchollet, look forward to receive your review:)
I am ready to address your comments asap these days.

Thanks in advance,
Roman

[rkazants]

@fchollet, kindly reminder about review.

Thank you,
Roman

[rkazants]

Hi @fchollet, any update? Please inform me if any concerns.

Thanks,
Roman

[rkazants]

Hi @fchollet, @gbaned,

Could you please take a look? Let me know if any minimal scope of supported ops are required for this PR. I will add it here.

Thanks,
Roman

[fchollet]

Sorry, only seeing this now. It looks better! It seems the set of ops currently implemented is quite small, however. What kinds of end-to-end workflows are currently working? What level of op coverage do you think you can achieve?

[rkazants]

Sorry, only seeing this now. It looks better! It seems the set of ops currently implemented is quite small, however. What kinds of end-to-end workflows are currently working? What level of op coverage do you think you can achieve?

Hi @fchollet, currently we support predict method only for inference. evaluate is not supported because it requires to support loss function and run it on the same backend. That is not convenient to do with our backend because eager execution is absent.

In future, from my quick glance we will be able to cover solid number of operations because OpenVINO opset is quite well developed and support a lot of TF, PyTorch, ONNX models from different hubs (TF, ONNX, HF, PyTorch, etc,) for inference and we permanently develop it to cover TF, PyTorch, ONNX opsets. You can find all OV operations here: https://docs.openvino.ai/2024/documentation/openvino-ir-format/operation-sets/operation-specs.html

Let me know what amount of operations I should cover in this PR to get it merged. So we will have the base OpenVINO support here and continue ops coverage support in future PRs. For the base OpenVINO support, I propose to support operations required by BERT model without preprocessor from Keras Hub, for example, and demonstrate that it works. Example below. Will it be sufficient for merge?

import os
os.environ["KERAS_BACKEND"] = "openvino"

import keras_nlp

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}

# Load a BERT model.
classifier = keras_nlp.models.BertClassifier.from_preset(
    "bert_base_en_uncased", 
    num_classes=2,
    preprocessor=None
)

classifier.predict(features)

In the description of this PR, I demonstrated very simple model inference. Please check.

[fchollet]

Let me know what amount of operations I should cover in this PR to get it merged. So we will have the base OpenVINO support here and continue ops coverage support in future PRs. For the base OpenVINO support, I propose to support operations required by BERT model without preprocessor from Keras Hub, for example, and demonstrate that it works. Example below. Will it be sufficient for merge?

We will have to run only a very similar subset of unit tests on CI for OpenVINO since we cannot skip every test that isn't passing (too many). So we would have to create basically an OpenVINO-specific integration test that checks precisely the end-to-end workflows that are intended to be supported.

Are you going to add support for CV workflows? I would recommend also having support for conv/pooling ops.

[rkazants]

We will have to run only a very similar subset of unit tests on CI for OpenVINO since we cannot skip every test that isn't passing (too many). So we would have to create basically an OpenVINO-specific integration test that checks precisely the end-to-end workflows that are intended to be supported.

Are you going to add support for CV workflows? I would recommend also having support for conv/pooling ops.

Yes, I will add separate integration tests for OpenVINO to cover currently supported ops. For this PR, let me enable MobileNetV3 from KerasCV that will require to support conv/pooling operations.

Summarizing it, my to-do list for this PR looks as follows:

1. Enable BertClassifier model from KerasNLP
2. Enable MobileNet model from KerasCV
3. Add integration tests for OpenVINO to cover supported ops

@fchollet, will it be fine and sufficient for this PR merge?

Thanks a lot,
Roman

[fchollet]

@fchollet, will it be fine and sufficient for this PR merge?

Yes, that sounds good to me!

[rkazants]

Hi @fchollet,

I added support for Keras Hub Bert and MobileNet models and removed openvino-specific code from layer.py. Please do preliminary code-review for it. In the meantime I am working on tests and GHA configuration for them.

Example 1:

# from here https://keras.io/api/keras_nlp/models/bert/bert_text_classifier/#from_preset-method
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
#import keras_hub

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}

# Pretrained classifier without preprocessing.
#classifier = keras_hub.models.BertTextClassifier.from_preset(
#    "bert_base_en_uncased",
#    num_classes=4,
#    preprocessor=None,
#)
#keras.saving.save_model(classifier, "bert_base_en_uncased.keras")
classifier = keras.saving.load_model("bert_base_en_uncased.keras")
#output = classifier.predict(x=features, batch_size=2)
output = classifier.predict(features)

Example 2:

# from here https://keras.io/api/keras_cv/models/backbones/mobilenet_v3/
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
import keras_cv

rng = np.random.default_rng(23345243)

#input_data = np.random.rand(8, 224, 224, 3)
input_data = rng.uniform(0.0, 1.0, [8, 224, 224, 3]).astype(np.float32)
model = keras.saving.load_model("mobilenet_v3_small.keras")
#model = keras_cv.models.MobileNetV3Backbone.from_preset("mobilenet_v3_small",)

output = model.predict(input_data)

Best regards,
Roman