[gTests] Create new gTests for bn_fwd_train and bn_bwd without saved mean

[xinlipn]

Create gTests to cover cases without saved Mean and saved InvVar, equivalent to e.g. the command below

./bin/MIOpenDriver bnormbfp16 -n 1 -c 2 -H 24 -W 16 -m 1 --forw 1 -b 0 -s 0 -r 0 -t 1 --layout NCHW

[bghimireamd]

I do not see bn bwd cpu handle both "saved mean" and "no saved mean"

[xinlipn]

I do not see bn bwd cpu handle both "saved mean" and "no saved mean"

Hi @bghimireamd , that's a good catch. I have this change locally, but but bn_bwd doesn't seem to pass yet with it. I have just committed the change to ComputeCPUBNBwd

[bghimireamd]

can we comment here why the difference in initialization?

[xinlipn]

can we comment here why the difference in initialization?

Hi @bghimireamd , MIOpenDriver command uses InitHostData to initialize the values, which is equivalent to the newly added fill_tensor_with_random_values, which create different initial values upon being called. In comparison, the generate() method will initialize tensors with the same initial values, which could lead to issues for no saved mean as I observed. But for this particular, i.e. input tensor, it may be ok using input.generate()

MIOpen/driver/bn_driver.hpp

Lines 266 to 269 in ea65f9d

	bnScale.InitHostData(
	bnScale.GetTensor().desc.GetElementSize(),
	true,
	uniform_signed_initializer<TScaleBias>(2e-3 /*scale*/, 1000 /*range*/));

[xinlipn]

More details about the changes in how tensors are initialized.

MIOpen/test/gtest/bn_test_data.hpp

Lines 423 to 440 in 5676b97

	if(this->saveMeanVar)
	{
	bnScale.generate(
	uniform_signed_initializer<ScaleDataType>(2e-3 /*scale*/, 1000 /*range*/));
	bnBias.generate(
	uniform_signed_initializer<ScaleDataType>(2e-3 /*scale*/, 1000 /*range*/));
	savedMean.generate(
	uniform_signed_initializer<MeanVarDataType>(2e-3 /*scale*/, 1000 /*range*/));
	savedInvVar.generate(
	uniform_signed_initializer<MeanVarDataType>(2e-3 /*scale*/, 1000 /*range*/));
	}
	else
	{
	fill_tensor_with_random_values(bnScale, 2e-3 /*scale*/, 1000 /*range*/);
	fill_tensor_with_random_values(bnBias, 2e-3 /*scale*/, 1000 /*range*/);
	fill_tensor_with_random_values(savedMean, 2e-3 /*scale*/, 1000 /*range*/);
	fill_tensor_with_random_values(savedInvVar, 2e-3 /*scale*/, 1000 /*range*/);
	}

1. Previous method calls generate(...) to initialize tensors, which seeds the RNG using a hash based on tensor dimensions desc.GetLenghths(), total number of elements data.size(), and Number of dimensions desc.GetLengths().size() as below. Thus, every time generated() is called on a tensor with the same shape, it will produce the same initial values

MIOpen/test/tensor_holder.hpp

Lines 240 to 252 in 5676b97

	template <class G>
	void generate_impl(G g)
	{
	auto seed = std::accumulate(desc.GetLengths().begin(),
	desc.GetLengths().end(),
	std::size_t{521288629},
	[](auto x, auto y) {
	x ^= x << 1U;
	return x ^ y;
	});
	seed ^= data.size();
	seed ^= desc.GetLengths().size();
	prng::reset_seed(seed);

2. In comparison, the new fill_tensor_with_random_values(...) uses std::generate(...) with a random generator function and no seed is explicitly set, the current RNG state may be based on a) Global RNG seed, b) System time, or c) previously changed state from other PRNG calls. It's non-deterministic and this way different tensors will have different initial values even for the same shape.

[amd-hsivasun]

Imported to ROCm/rocm-libraries