Add AttentionOp #708
Add AttentionOp #708
Conversation
| dnf install -y \ | ||
| which wget gcc zlib-devel bzip2 bzip2-devel readline-devel sqlite \ | ||
| sqlite-devel xz xz-devel libffi-devel curl git ncurses-devel \ | ||
| openssh-clients libcudnn8-devel zip jq \ |
There was a problem hiding this comment.
cudnn8 conflicts with cudnn9 in the base container.
| //===----------------------------------------------------------------------===// | ||
|
|
||
| bool tensorrt::AttentionOp::isValidForTensorRTVersion( | ||
| int64_t trtMajorVersion) { |
There was a problem hiding this comment.
We should also check the minor version here.
| def TensorRT_AttentionNormalizationOpAttr : TensorRT_EnumAttr<TensorRT_AttentionNormalizationOp, "attention_normalization_op">{ | ||
| } | ||
|
|
||
| def TensorRT_DataType : TensorRT_I32EnumAttr< |
There was a problem hiding this comment.
We didn't already have datatype?
There was a problem hiding this comment.
We didn't have an op that explicitly requires data type as an input, for example cast op uses MLIR data types
There was a problem hiding this comment.
Can we do the same here? I'm assuming we could use the same helpers that cast uses?
There was a problem hiding this comment.
cast uses output tensor type to indicate the data type. e.g.
tensorrt.cast %arg0 : tensor<3xf16> to tensor<3xf32>
but in attention op, normalization_quantize_to_type parameter is i) optional and ii) represents an intermidiate quantization data type.
No description provided.