Vague input error for trained TensorFlow model #269
Description
I am trying to retrain and deploy our team's previously-already-working TensorFlow model onto SageMaker... having no luck after several days of banging my head against it. Any help that you all can provide on how to massage this into SageMaker would be greatly appreciated... about to give up and just run an AMI
Every time that I try to make a prediction/inference request on my trained, evaluated, and deployed model, I get the error
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message ""
Similar to #67, and probably a duplicate of #99
Serving input function is basically identical to the TensorFlow tutorial:
CSV_COLUMNS = ['reading', 'temperature', 'hour', 'weekday', 'is_holiday', 'area', 'structure_area', 'has_pv']
LABEL_COLUMN = 'reading'
CSV_COLUMN_DEFAULTS = [[0.0], [0.0], [0], [0], [0], [0.0], [2000.0], [0]]
UNUSED_COLUMNS = set(CSV_COLUMNS) - {col.name for col in INPUT_COLUMNS} - \
{LABEL_COLUMN}
def estimator_fn(run_config, params):
... build and return DNNRegressor ...
def serving_input_fn():
csv_row = tf.placeholder(
shape=[None],
dtype=tf.string
)
features = parse_csv(csv_row)
features.pop(LABEL_COLUMN)
return tf.estimator.export.ServingInputReceiver(features, {'csv_row': csv_row})
def parse_csv(rows_string_tensor):
"""Takes the string input tensor and returns a dict of rank-2 tensors."""
# Takes a rank-1 tensor and converts it into rank-2 tensor
# Example if the data is ['csv,line,1', 'csv,line,2', ..] to
# [['csv,line,1'], ['csv,line,2']] which after parsing will result in a
# tuple of tensors: [['csv'], ['csv']], [['line'], ['line']], [[1], [2]]
row_columns = tf.expand_dims(rows_string_tensor, -1)
columns = tf.decode_csv(row_columns, record_defaults=CSV_COLUMN_DEFAULTS)
features = dict(zip(CSV_COLUMNS, columns))
# Remove unused columns
for col in UNUSED_COLUMNS:
features.pop(col)
return features
Logs:
17:04:12
Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke self.transformer.transform(content, input_content_type, requested_output_content_type) File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 281, in transform return self.transform_fn(data, content_type, accepts), accepts File "/usr/l
AllTraffic/i-0933877fa79ea8c2f
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke
self.transformer.transform(content, input_content_type, requested_output_content_type)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 281, in transform
return self.transform_fn(data, content_type, accepts), accepts
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 206, in f
prediction = self.predict_fn(input)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 221, in predict_fn
return self.proxy_client.request(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 66, in request
request_fn = self.request_fn_map[self.prediction_type]
17:04:12
KeyError: u'tensorflow/serving/regress'
AllTraffic/i-0933877fa79ea8c2f
KeyError: u'tensorflow/serving/regress'
17:04:12
[2018-06-29 17:04:11,861] ERROR in serving: u'tensorflow/serving/regress'
AllTraffic/i-0933877fa79ea8c2f
[2018-06-29 17:04:11,861] ERROR in serving: u'tensorflow/serving/regress'