Fix AWGN resampler test

[dshil]

The result of last rebase on @baranovmv changes.

     TEST   roc_audio
........................
src/tests/roc_audio/test_resampler.cpp:159: error: Failure in TEST(resampler, upscaling_twice_awgn)
	CHECK(buff[i] >= -60) failed

..!!......................
.
Errors (1 failures, 51 tests, 49 ran, 147101 checks, 2 ignored, 0 filtered out, 219 ms)

scons: *** [test/roc_audio] Error 1

[baranovmv]

Wow, that's difficult. especially this code works fine on PC.
I always add some code in tests or resampler to store the signal in a file so as to be able to look at it, but I don't have access to a Mac now. If I won't manage to get a Mac, we can leave it as is, until we meet (hope it'll happen this year).

[gavv]

We could add an option (e.g. ifdef) that dumps the necessary data to a file for such cases.

[gavv]

Since the hardware architecture and compiler are the same, I can only imagine a bug in memory management in our code or some difference is stdlib functions, like random or math.

[dshil]

ping @baranovmv,

I've simply put several print to debug a problem:

TEST(resampler, upscaling_twice_awgn) {
    enum { ChMask = 0x1 };

    MockReader reader;
    Resampler resampler(reader, buffer_pool, allocator, config, ChMask);

    CHECK(resampler.set_scaling(0.5f));

    const size_t sig_len = 2048;
    double buff[sig_len * 2];
    size_t i;

    for (size_t n = 0; n < InSamples; n++) {
        const sample_t s = (sample_t)AWGN_generator();
        reader.add(1, s);
    }

    // Put the spectrum of the resampled signal into buff.
    // Odd elements are magnitudes in dB, even elements are phases in radians.
    get_sample_spectrum1(resampler, buff, sig_len);

    for (i = 0; i < sig_len - 1; i += 2) {
        if (i <= sig_len * 0.90 / 2) {
            if (buff[i] < -60) {
                printf("i = %lu, buf[i] = %f\n", i, buff[i]);
            }
            /* CHECK(buff[i] >= -60); */
        } else if (i >= sig_len * 1.00 / 2) {
            if (buff[i] > -60) {
                printf("i = %lu, buf[i] = %f\n", i, buff[i]);
            }
            /* CHECK(buff[i] <= -60); */
        }
    }
}

And as a result:

i = 518, buf[i] = -74.226430

i = 1024, buf[i] = -59.682005
i = 1026, buf[i] = -59.800069
i = 1028, buf[i] = -59.847236
i = 1030, buf[i] = -59.815538
i = 1032, buf[i] = -59.871927
i = 1036, buf[i] = -59.982135

The whole buff

A glimmer of hope: Does it tell something to you ? :)

[baranovmv]

1. as this test was committed, then I suppose it passes on PCs under Linux, right? Then the issue relates somehow with compiler and/or standard libraries, e.g. different floating point math implementation, etc...
2. according to the numbers, the spectrum is a little bit above the border (~0.5 dB). The border -60 dB should have been chosen with some margin, so it might be a marker of some hidden troubles. Or it can be just a random difference in implementation.

I am not able to reproduce this bug, sorry. I suggest roc to hold this issue until someone will refactor the Resampler.

[gavv]

It appeared that this test was just broken in several aspects and the problem was not specific to macOS.

I was able to reproduce this on Linux too, by setting non-default random seed, e.g. adding srand(100500) to the beginning of the test. It just happened that it worked with the default seed on Linux and didn't work with the default seed on macOS.

The test has several problems:

• The test is not truly random because it always uses the default seed for rand().

• The test's requirement for the lower half of the spectrum to be above -60dB is not strictly met on any generated noise. The spectrum may have spikes if some frequencies are mostly missing, which happens from time to time when we use random seed. The spectrum value for such frequencies may be below -70dB, for example.

• The test's requirements for the upper half of the spectrum to be below -60dB is not always met as well. My tests showed that we can only rely on -50dB.

• The test's requirements for the cut-off border between the lower and the upper halves (90%) was too strict as well.

• The check for the upper half of the spectrum was incorrect. The condition if (i >= sig_len * 1.00 / 2) was never triggered.

What I've done (3f62f7a):

• Switched AWGN generator from stdlib rand() to Roc core::random(), which is guaranteed to have a random seed. After doing this, the test started failing on Linux frequently.

• Applied a median filter to the spectrum to remove spikes caused by missing frequencies.

• Relaxed and fixed test's requirements both for the magnitude and for the cut-off border.

• Refactored AWGN generator. Its code was overcomplicated and the comments were wrong.

The test is now stable both on Linux and macOS.

@baranovmv The 90% cut-off border and -60dB magnitude threshold were too optimistic for our resampler with this AWGN generator. I don't know where did the specific requirements come from. For now my goal is to make the test correct and stable. Then we can think whether the relaxed requirement are okay or we need some changes in the resampler or in the test.

UPD: after a short discussion, it seems that the relaxed requirements are not great. I think we need to perform more testing on different resampler profiles and inputs.

[gavv]

Here are a few examples of the spectrums that didn't fit the original test requirements.

Spikes in the lower half

Higher magnitude in the upper half

(spikes were removed using a median filter)

A bit shifted cut-off border

(spikes were removed using a median filter)

[gavv]

Closing this. We can discuss further action in #153.