High performance about online prediction

[smartcatdog]

Using bert-inference to online prediction is ok, however, the running time of prob = self.sess.run([self.probabilities], feed_dict) in 『with self.graph.as_default() as g』 will take little seconds, which is too long for online prediction. And i want for your help to get direction how to optimization the code for online prediction.