Added shape (U)INT8/BF16/FP32 oneDNN kernel

[jakpiase]

PR types

New features

PR changes

OPs

Describe

Added shape (U)INT8/BF16/FP32 OneDNN kernel. It should speed up some models because it will remove unnecessary external reorders. It can work with any datatype supported by OneDNN, because shape kernel works in the same way, no matter what kind of datatype does the Input tensor have. External reorders are removed because shape kernel always results in plain memory format tag (NCHW/ABCD), so native kernel that is executed afterwards won't need a reorder. It speeds up faster_rcnn_r50_fpn_1x by 1%

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[paddle-bot-old]

Sorry to inform you that d629217's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[lidanqing-intel]

LGTM

[jakpiase]

I will continue working on this PR after md-in-tensor related PR will be merged, because this PR has a dependency on that PR

[jczaja]

LGTM

[jczaja]

@baoachun Please start your review

[baoachun]

@lidanqing-intel Do we still support modifying the properties of native operators now?

[jczaja]

@baoachun Plan is (as executed in #36541) that the only new attribute to be added in native ops is "use_mkldnn" . That is becasue in training there are no passes, so we still need an attribute (set with env var FLAGS_use_mkldnn) so that PaddlePaddle runs oneDNN kerenels.

[jakpiase]

use_mkldnn attribute is necessary for running mkldnn kernels during training since there are not inference passes in that mode. This attribute is used to determine whether mkldnn kernel should be run or not and that behavior is checked in GetExpectedKernelType(). It is a common behavior for all mkldnn kernels and that attribute must be included in original operator version but I can create another onednn-based op and move mkldnn_data_type attribute into it, but that cannot be done with use_mkldnn attribute. Personally, I would like to keep both of these attributes in native op to allow common inference infrastructure usage but if that's a problem then I'll create a separate op for that.
Here's code for shape op's GetExpecterKernelType() which uses use_mkldnn in this->CanMKLDNNBeUsed() function

Paddle/paddle/fluid/operators/shape_op.cc

Lines 38 to 51 in c475532

	framework::OpKernelType GetExpectedKernelType(
	const framework::ExecutionContext &ctx) const override {
	auto input_data_type =
	framework::OperatorWithKernel::IndicateVarDataType(ctx, "Input");
	#ifdef PADDLE_WITH_MKLDNN
	if (this->CanMKLDNNBeUsed(ctx, input_data_type)) {
	return framework::OpKernelType(input_data_type, ctx.GetPlace(),
	framework::DataLayout::kMKLDNN,
	framework::LibraryType::kMKLDNN);
	}
	#endif
	return framework::OpKernelType(input_data_type, ctx.GetPlace());
	}

[jczaja]

@baoachun , @jakpiase
Training does no use passes. So Executor used for training is having a mechanism that when FLAGS_use_mkldnn=True then iteration through all ops is done and those having use_mkldnn attribute are having this attribute set to True. So that is the way how during training to let know Paddle to use oneDNN kernels.

Regarding GPU and its "use_cudnn". I looked to code and most of operators (older ones like, conv2d, softmax, gelu, activation, abs,....) are having "use_cudnn" defined. Probably due to API compatibility reasons.
Some newer ops e.g. Matmul . Are not having use_cudnn. But matmul CPU and GPU kernel are the same method and execution decision is made based on parameter Place e.g. CPU or GPU.

Some others like affine_channel are having their own kernel for GPU , different from CPU. And decision which run is made based on searching through registers kernels when Key is OpKernelType using following comparison:

Paddle/paddle/fluid/framework/op_kernel_type.cc

Lines 55 to 60 in 1252f4b

	bool OpKernelType::operator==(const OpKernelType& o) const {
	return platform::places_are_same_class(place_, o.place_) &&
	data_type_ == o.data_type_ && data_layout_ == o.data_layout_ &&
	library_type_ == o.library_type_ &&
	customized_type_value_ == o.customized_type_value_;

So you can see that Kernel is chosen if required type matches .e.g library, place, type .

For GPU kernels there is GPU place. So library plain and GPU place inidicate that GPU kerenels should be used.
For oneDNN kernels it is a bit harder as there is only CPU place which is used for CPU kernels and oneDNN kernels.
So to pick oneDNN kernels we also need to set library to MKLDNN. And to set library to MKLDNN we currently use "use_mkldnn".

What I'm trying to write is that if there were oneDNN place then we could do the same as for GPU e.g. discard use_mkldnn attribute. But since there is no oneDNN place and we use CPU place then we need some way to indicate
during the training that library should be MKLDNN and this is done using attribute use_mkldnn.

[baoachun]

LGTM