Issue451 diy recognizer feedback #459

towsey · 2021-02-24T04:40:54Z

Changes to the call detection algorithm.

What is the purpose of this PR?
~~To incorporate changes into master.~~

To change the post-processing so that it works per-decibel threshold. See #451

Changes

Two major changes:

change to the detection/post-processing sequence.
changes to names of the post-processing parameters in the config file.

Issues

None as yet.

Visual Changes

The changes output more information about the effect of parameter values on the acceptance or rejection of events. THis will make it easier for the user to "tune" parameters.

Final Checklist

Assign reviewers if you have permission
Assign labels if you have permission
Link issues related to PR
Link any PRs or issues blocking this PR from being merged
Remove/Reduce warnings from edited files
Unit tests have been added for changes
Ensure CI build is passing

…riodicity Issue #451 Also write a new method to convert an array of real values to a string.

Issue #451 Add feedback about filtering events on syllable duration, bandwidth, sideband activity etc.

Issue #451 Add info concerning combining events and also add more to determination of side-band activity.

…ilters Issue #451 In particular provide more info on side-band activity filters. Also temporarily comment out calls to method that writes the results.csv file because it is not working.

Issue #451 These recognizers are either not needed or are unusable and would be extremely difficult to resurrect. Their development was discontinued around 2015 when they were in an embryonic state.

Issue #451 1: Add some booleans into the config file to control event filtering. 2: Edit the Call Recognizer Guide to reflect recent changes in the names of config parameters. 3: Remove a unit test for a discontinued call recognizer.

Issue #451 Rewrite the event detection/post-processing algorithm so that post-processing steps are performed for each level of decibel detection threshold. The guide.md file has been edited to explain this change and to make it consistent with changes to the config parameter names.

codecov · 2021-02-24T04:43:27Z

Codecov Report

Merging #459 (ef881a9) into master (bc0f263) will decrease coverage by 37.08%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #459       +/-   ##
==========================================
- Coverage   37.79%   0.70%   -37.09%     
==========================================
  Files         482     470       -12     
  Lines       69114   48626    -20488     
  Branches     8062    7662      -400     
==========================================
- Hits        26120     344    -25776     
- Misses      42904   48232     +5328     
+ Partials       90      50       -40

Impacted Files	Coverage Δ
...coustics.Shared/Extensions/EnumerableExtensions.cs	`0.00% <0.00%> (-67.41%)`	⬇️
...nalysisPrograms/Recognizers/Base/RecognizerBase.cs	`0.00% <ø> (-13.07%)`	⬇️
.../AnalysisPrograms/Recognizers/GenericRecognizer.cs	`0.00% <0.00%> (-76.38%)`	⬇️
src/AudioAnalysisTools/Events/EventCommon.cs	`0.00% <0.00%> (-85.72%)`	⬇️
src/AudioAnalysisTools/Events/EventFilters.cs	`0.00% <0.00%> (-85.29%)`	⬇️
...oAnalysisTools/Events/Types/EventPostProcessing.cs	`0.00% <0.00%> (-100.00%)`	⬇️
src/TowseyLibrary/DataTools.cs	`0.00% <0.00%> (-31.44%)`	⬇️
src/AcousticWorkbench/Api.cs	`0.00% <0.00%> (-100.00%)`	⬇️
src/Acoustics.Shared/Binary.cs	`0.00% <0.00%> (-100.00%)`	⬇️
src/AnalysisBase/AnalyzerConfig.cs	`0.00% <0.00%> (-100.00%)`	⬇️
... and 461 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc0f263...477a542. Read the comment docs.

Issue #451

atruskie

I'm not all convinced post-processing should be done per decibel threshold. If the purpose of multiple decibel threshold is to detect the same event at varying intensities, then they should be flattened into one layer. If the purpose is to detect different events, then they should be different profiles.

I actually think any event detected in the same spot in multiple threshold settings, should be flattened, and only the highest decibel event kept.

If the post-processing should be completely agnostic to algorithm used as well. What of algorithms that do not use decibel thresholds?

But assuming you have better reasons than you have explained here, these changes are still problematic.

Changing parameters constitutes a breaking contract
There are numerous code quality issues
the include multiple changes unrelated to Providing users of Call Recognizers with more diagnostic feedback #451
- including the deletion of recognisers
- changing of post-processing events

atruskie · 2021-02-24T13:30:19Z

docs/guides/generic_recognizers.md


 [!code-yaml[post_processing_filtering](./Ecosounds.NinoxBoobook.yml?start=34&end=62&highlight=20- "Post Processing: filtering")]

 Use the parameter `Duration` to filter out events that are too long or short.
 This filter removes events whose duration lies outside three standard deviations (SDs) of an expected value.

+- `FilterOnDuration` should be set true if you want to filter events on their duration, otherwise false.


This FilterOnDuration switch is not in the example config... and also it is verbose. We should remove the concept entirely.

If a user does not want to filter on duration then they should just not provide any parameters

atruskie · 2021-02-24T13:30:56Z

docs/guides/generic_recognizers.md

-Add a post processing section to you config file by adding the `PostProcessing` parameter and indenting the sub-parameters.
+### [PostProcessing](xref:AudioAnalysisTools.Events.Types.EventPostProcessing.PostProcessingConfig)
+
+Post-processing of events is performed after event detection. However it is important to understand that post-processing is performed once for each of the DecibelThresholds. As an example: suppose you have three decibel thresholds (6, 9 and 12 dB is a typical set of values) in each of two profiles. All the events detected at threshold 6 dB (by both profiles) will be collected together and subjected to the post processing steps. Typically some or all of the events may fail to be accepted as "true" events based on your post-processing parameters. Then all the events detected at 9 dB will be collected and independently subjected to post-processing. Then, likewise, all events detected at the 12 dB threshold will be post-processed. In other words, one round of post-processing is performed for each decibel threshold. This sequence of multiple post-processing steps gives rise to one or more temporally nested events. Think of them as Russion doll events! The final post-processing step is to remove all but the longest duration event in any nested set of events.


add line break please - just as you would for code... one long line makes diffs difficult to read

I won't mark other lines with this issue, but it applies to all the changes you made

I have inserted many line breaks.

atruskie · 2021-02-24T13:32:38Z

docs/guides/generic_recognizers.md


 [!code-yaml[post_processing](./Ecosounds.NinoxBoobook.yml#L34-L34 "Post Processing")]

-Post processing is optional. You may just want to combine or filter component events in your own code.
+Post processing is optional - you may decide to combine or filter the "raw" events using code you have written yourself. To add a post-processing section to your config file, insert the `PostProcessing` parameter and indent the sub-parameters. There are five post-processing possibilities each of which you may choose to use or not. Note that the post-processing steps are performed in this order which cannot be changed by the user:


Suggested change

Post processing is optional - you may decide to combine or filter the "raw" events using code you have written yourself. To add a post-processing section to your config file, insert the `PostProcessing` parameter and indent the sub-parameters. There are five post-processing possibilities each of which you may choose to use or not. Note that the post-processing steps are performed in this order which cannot be changed by the user:

Post processing is optional - you may decide to combine or filter the "raw" events using code you have written yourself. To add a post-processing section to your config file, insert the `PostProcessing` parameter and indent the sub-parameters. Each of the post-processing possibilities are optional. Note that the post-processing steps are performed in this order which cannot be changed by the user:

I have done a lot of work here to shift unnecessary documentation to other guides

atruskie · 2021-02-24T13:34:23Z

docs/guides/generic_recognizers.md


 Unlike overlapping events, if you want to combine a group of events (like syllables) that are near each other but not
 overlapping, then make use of the `SyllableSequence` parameter.

 [!code-yaml[post_processing_combining_syllables](./Ecosounds.NinoxBoobook.yml?start=34&end=51&highlight=10- "Post Processing: Combining syllables")]

-Set `CombinePossibleSyllableSequence` true where you want to combine possible syllable sequences. A typical example is
-a sequence of chirps in a honeyeater call.
+Combining syllable sequences is enabled by setting `CombinePossibleSyllableSequence` true. A typical example is a sequence of chirps in a honeyeater call.


Suggested change

Combining syllable sequences is enabled by setting `CombinePossibleSyllableSequence` true. A typical example is a sequence of chirps in a honeyeater call.

Combining syllable sequences is enabled by setting `CombinePossibleSyllableSequence` to `true`. A typical example is a sequence of chirps in a honeyeater call.

Fixed these where they occur.

atruskie · 2021-02-24T13:35:05Z

docs/guides/generic_recognizers.md

 Use the parameter `Bandwidth` to filter out events whose bandwidth is too small or large.
 This filter removes events whose bandwidth lies outside three standard deviations (SDs) of an expected value.

+- `FilterOnBandwidth` should be set true if you want to filter events on their bandwidth, otherwise false.


Ditto, remove docs and concept

atruskie · 2021-02-24T13:58:23Z

src/AudioAnalysisTools/Events/EventFilters.cs

+            // The following line does it all BUT it does not allow for feedback to the user.
+            //var filteredEvents = events.Where(ev => ((SpectralEvent)ev).EventDurationSeconds >= minimumDurationSeconds && ((SpectralEvent)ev).EventDurationSeconds <= maximumDurationSeconds).ToList();
+
+            var filteredEvents = new List<EventCommon>();
+
+            var count = 0;
+            foreach (var ev in events)
+            {
+                count++;
+                var duration = ((SpectralEvent)ev).EventDurationSeconds;
+                if ((duration > minimumDurationSeconds) && (duration < maximumDurationSeconds))
+                {
+                    Log.Debug($" Event{count} accepted: Actual duration = {duration:F3}s");
+                    filteredEvents.Add(ev);
+                }
+                else
+                {
+                    Log.Debug($" Event{count} rejected: Actual duration = {duration:F3}s");
+                    continue;
+                }
+            }
+
+            return filteredEvents;
+        }
+


As with the others, an incorrect assertion, revert the change

A new extension method has been written to take care of this.

atruskie · 2021-02-24T14:00:53Z

src/AudioAnalysisTools/Events/EventFilters.cs

+                string strArray = DataTools.Array2String(actualPeriodSeconds.ToArray());
+                Log.Debug($" Event{count} actual periods: {strArray}");


Use

audio-analysis/src/Acoustics.Shared/Extensions/EnumerableExtensions.cs

Lines 307 to 319 in 5baf2d9

public static string Join<T>(this IEnumerable<T> items, string delimiter = " ")

{

var result = new StringBuilder();

foreach (var item in items)

{

result.Append(item);

result.Append(delimiter);

}

// return one delimiter length less because we always add a delimiter on the end

return result.ToString(0, result.Length - delimiter.Length);

}

}

Suggested change

string strArray = DataTools.Array2String(actualPeriodSeconds.ToArray());

Log.Debug($" Event{count} actual periods: {strArray}");

Log.Debug($" Event{count} actual periods: {actualPeriodSeconds.Join(",")}");

New extension method has been written.

atruskie · 2021-02-24T14:04:30Z

src/AudioAnalysisTools/Events/Types/EventPostProcessing.cs

+            /// <summary>
+            /// Gets or sets a value indicating Whether or not to filter events on their duration.
+            /// </summary>
+            public bool FilterOnDuration { get; set; }


Suggested change

/// <summary>

/// Gets or sets a value indicating Whether or not to filter events on their duration.

/// </summary>

public bool FilterOnDuration { get; set; }

Property removed.

atruskie · 2021-02-24T14:04:44Z

src/AudioAnalysisTools/Events/Types/EventPostProcessing.cs

+            /// <summary>
+            /// Gets or sets a value indicating Whether or not to filter events on their band-width.
+            /// </summary>
+            public bool FilterOnBandwidth { get; set; }


Suggested change

/// <summary>

/// Gets or sets a value indicating Whether or not to filter events on their band-width.

/// </summary>

public bool FilterOnBandwidth { get; set; }

Property removed.

atruskie · 2021-02-24T14:07:25Z

src/TowseyLibrary/DataTools.cs

+        public static string Array2String(double[] array)
+        {
+            StringBuilder sb = new StringBuilder();
+            for (int i = 0; i < array.Length; i++)
+            {
+                sb.Append(array[i].ToString("F3") + ", ");
+            }
+
+            return sb.ToString();
+        }
+


Suggested change

public static string Array2String(double[] array)

{

StringBuilder sb = new StringBuilder();

for (int i = 0; i < array.Length; i++)

{

sb.Append(array[i].ToString("F3") + ", ");

}

return sb.ToString();

}

New extension method written.

Issue #451 Final changes to the use of post-processing parameters in config files.

Issue #451 Changes to config.yml files and to generic recognizer tests to accommodate changes to post-processing parameters.

Issue #451 Major changes were to allow some parameter values to be nullable to avoid the use of booleans.

Issue #451 Remove an unnecessary line inserted into a unit test.

Issue #451 All these changes are to do with post-processing of acoustic events.

atruskie

Okay, I can see you've made some changes - thankyou.

However, there are still multiple issues with the guide. Since this is such a big PR, I only have the option of accepting or rejecting it as a whole, and on those grounds alone I'm requiring more changes.

None of these changes are hard, but if they're not done in this contribution, then it represents work I'll need to do which I don't presently have time for.

Major issues:

still not enough line breaks (this includes line breaks you've removed from the master copy)
over explanation of config file parameters. Any detailed explanation of a parameter should be in the docs for it's config class, not in this guide
various syntax errors,
- with the NOTE: quotes (line break needed)
- with bullet lists (line break needed before the start of the list

atruskie · 2021-03-01T01:22:32Z

docs/guides/generic_recognizers.md

-profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in that case there
-would be profile for each target syllable or acoustic event.
+Each algorithm is designed to detect a syllable type. Thus to make a generic recognizer there should be at least one (1)
+profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in which case there will be profile for each target syllable or acoustic event.


Suggested change

profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in which case there will be profile for each target syllable or acoustic event.

profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in which case there

will be a profile for each target syllable or acoustic event.

'a' has been inserted

atruskie · 2021-03-01T01:37:08Z

docs/guides/generic_recognizers.md

+Post-processing of events is performed after event detection. Post-processing is performed once for each of the DecibelThresholds. As an example: suppose you have three decibel thresholds (6, 9 and 12 dB is a typical set of values) in each of two profiles. There will be three rounds of post-processing:
+- All the events detected at threshold 6 dB (by both profiles) will be collected together and subjected to the post-processing steps. Typically some or all of the events may fail to be accepted as "true" events based on your post-processing parameters.
+- Next all the events detected at 9 dB will be collected and independently subjected to post-processing.
+- Next all events detected at the 12 dB threshold will be post-processed. 
+
+This sequence of multiple post-processing steps gives rise to one or more temporally nested events. Think of them as Russion doll events! The final post-processing step is to remove all but the longest duration event in any nested set of events.


Suggested change

Post-processing of events is performed after event detection. Post-processing is performed once for each of the DecibelThresholds. As an example: suppose you have three decibel thresholds (6, 9 and 12 dB is a typical set of values) in each of two profiles. There will be three rounds of post-processing:

- All the events detected at threshold 6 dB (by both profiles) will be collected together and subjected to the post-processing steps. Typically some or all of the events may fail to be accepted as "true" events based on your post-processing parameters.

- Next all the events detected at 9 dB will be collected and independently subjected to post-processing.

- Next all events detected at the 12 dB threshold will be post-processed.

This sequence of multiple post-processing steps gives rise to one or more temporally nested events. Think of them as Russion doll events! The final post-processing step is to remove all but the longest duration event in any nested set of events.

Post-processing of events is performed after event detection. Post-processing is performed once for each of the DecibelThresholds.

As an example: suppose you have three decibel thresholds (6, 9 and 12 dB is a typical set of values) in each of two profiles.

There will be three rounds of post-processing:

- All the events detected at threshold 6 dB (by both profiles) will be collected together and subjected to the

post-processing steps. Typically some or all of the events may fail to be accepted as "true" events based on your

post-processing parameters.

- Next all the events detected at 9 dB will be collected and independently subjected to post-processing.

- Next all events detected at the 12 dB threshold will be post-processed.

This sequence of multiple post-processing steps gives rise to one or more temporally nested events. Think of them as Russion doll events!

The final post-processing step is to remove all but the longest duration event in any nested set of events.

I have rewritten parts of this but not clear what the issue was/is.

docs/guides/generic_recognizers.md

src/AnalysisConfigFiles/RecognizerConfigFiles/Towsey.NinoxBoobook.yml

src/AnalysisPrograms/Recognizers/GenericRecognizer.cs

atruskie · 2021-03-01T01:49:49Z

src/AnalysisPrograms/Recognizers/GenericRecognizer.cs

+            // Add a spacer for easier reading of the debug output.
+            Log.Debug($" ");
+            Log.Debug($"Event count BEFORE removing enclosed events = {postEvents.Count}.");
+            results.NewEvents = CompositeEvent.RemoveEnclosedEvents(postEvents);


This comment remains unanswered?

atruskie · 2021-03-01T01:53:58Z

src/AnalysisPrograms/Recognizers/GenericRecognizer.cs

+            // Add a spacer for easier reading of the debug output.
+            Log.Debug($" ");
+            Log.Debug($"Event count BEFORE removing enclosed events = {postEvents.Count}.");
+            results.NewEvents = CompositeEvent.RemoveEnclosedEvents(postEvents);


I think I understand, this is not flattening nested events, or even flattening Russian dolls, rather this is removing any event that is completely overlapped in time/frequency bounds by another event covering the same area.

If my new understanding is correct, then no change is required, other than updating the log message.

The terminology nested is wrong here since there is no parent-child relationship. Hence the confusion.

src/AnalysisPrograms/Recognizers/GenericRecognizer.cs

src/AudioAnalysisTools/Events/EventFilters.cs

Issue #451

Issue #451 Changes to documentation involved shifted some documentation to other guides and to class summaries.

Also added a safe overload for the new JoinFormatted method

Issue #451 A few further changes to make a parameter available to the user with changes accordingly to the documentation.

…utEcoacoustics/audio-analysis into Issue451_DIYRecognizerFeedback

Issue #451 Fix conflicts

Also fixed the example config file and added aside styling for breakout text

Issue #451 Made some final tweaks to the guide.

towsey added 7 commits February 19, 2021 10:32

Add feedback to user about results of filtering events on syllable pe…

0c2d451

…riodicity Issue #451 Also write a new method to convert an array of real values to a string.

Add feedback to user about results of filtering events

53945d0

Issue #451 Add feedback about filtering events on syllable duration, bandwidth, sideband activity etc.

Add more info concerning post-processing of events

94b33d4

Issue #451 Add info concerning combining events and also add more to determination of side-band activity.

Provide more information to user about the effect o post-processing f…

343c4ad

…ilters Issue #451 In particular provide more info on side-band activity filters. Also temporarily comment out calls to method that writes the results.csv file because it is not working.

Removal of incomplete and unresurrectable recognizers

bee4e67

Issue #451 These recognizers are either not needed or are unusable and would be extremely difficult to resurrect. Their development was discontinued around 2015 when they were in an embryonic state.

Add more parameters to control post-processing steps

c7ccc06

Issue #451 1: Add some booleans into the config file to control event filtering. 2: Edit the Call Recognizer Guide to reflect recent changes in the names of config parameters. 3: Remove a unit test for a discontinued call recognizer.

Fix broken tests after changes to recognizer config file

cabcefb

Issue #451

atruskie requested changes Feb 24, 2021

View reviewed changes

towsey added 5 commits February 25, 2021 16:37

Changes to the use of parameters in the config files

dcc9559

Issue #451 Final changes to the use of post-processing parameters in config files.

Changes to accommodate changed post-processing parameters

c75a8a8

Issue #451 Changes to config.yml files and to generic recognizer tests to accommodate changes to post-processing parameters.

Fix most issues addressed by Anthony in PR response

5a24efc

Issue #451 Major changes were to allow some parameter values to be nullable to avoid the use of booleans.

Update GenericRecognizerTests.cs

380e65c

Issue #451 Remove an unnecessary line inserted into a unit test.

Change docs and config file to be consistent with code changes

be61baa

Issue #451 All these changes are to do with post-processing of acoustic events.

atruskie requested changes Mar 1, 2021

View reviewed changes

towsey and others added 9 commits March 1, 2021 17:41

Changes as requested by Anthony.

a92319b

Issue #451

Changes to code and documentation requested by Anthony

7822272

Issue #451 Changes to documentation involved shifted some documentation to other guides and to class summaries.

Fixed formatting in generic recognizer guide

e699645

Also added a safe overload for the new JoinFormatted method

Final set of editing based on Anthony's requests.

5a9c950

Issue #451 A few further changes to make a parameter available to the user with changes accordingly to the documentation.

Merge branch 'Issue451_DIYRecognizerFeedback' of https://github.com/Q…

ef881a9

…utEcoacoustics/audio-analysis into Issue451_DIYRecognizerFeedback

Update generic_recognizers.md

ce858da

Issue #451 Fix conflicts

Reapplied markdown formatting changes

17c7931

Also fixed the example config file and added aside styling for breakout text

Update generic_recognizers.md

405ac24

Issue #451 Made some final tweaks to the guide.

Cleanup generic recogniser docs again

68983e6

atruskie approved these changes Mar 3, 2021

View reviewed changes

Resolved merge conflict

477a542

atruskie merged commit 477a542 into master Mar 3, 2021

atruskie deleted the Issue451_DIYRecognizerFeedback branch March 3, 2021 04:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue451 diy recognizer feedback #459

Issue451 diy recognizer feedback #459

towsey commented Feb 24, 2021 •

edited by atruskie

Loading

codecov bot commented Feb 24, 2021 •

edited

Loading

atruskie left a comment

atruskie Feb 24, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie Feb 24, 2021

towsey Mar 2, 2021

atruskie left a comment

atruskie Mar 1, 2021

towsey Mar 2, 2021

atruskie Mar 1, 2021

towsey Mar 2, 2021

atruskie Mar 1, 2021

atruskie Mar 1, 2021

	Combining syllable sequences is enabled by setting `CombinePossibleSyllableSequence` true. A typical example is a sequence of chirps in a honeyeater call.
	Combining syllable sequences is enabled by setting `CombinePossibleSyllableSequence` to `true`. A typical example is a sequence of chirps in a honeyeater call.

		string strArray = DataTools.Array2String(actualPeriodSeconds.ToArray());
		Log.Debug($" Event{count} actual periods: {strArray}");

	public static string Join<T>(this IEnumerable<T> items, string delimiter = " ")
	{
	var result = new StringBuilder();
	foreach (var item in items)
	{
	result.Append(item);
	result.Append(delimiter);
	}

	// return one delimiter length less because we always add a delimiter on the end
	return result.ToString(0, result.Length - delimiter.Length);
	}
	}

	profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in which case there will be profile for each target syllable or acoustic event.
	profile in the `Profiles` list. A config file may target more than one syllable or acoustic event, in which case there
	will be a profile for each target syllable or acoustic event.

Issue451 diy recognizer feedback #459

Issue451 diy recognizer feedback #459

Conversation

towsey commented Feb 24, 2021 • edited by atruskie Loading

Changes to the call detection algorithm.

Changes

Issues

Visual Changes

Final Checklist

codecov bot commented Feb 24, 2021 • edited Loading

Codecov Report

atruskie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atruskie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

towsey commented Feb 24, 2021 •

edited by atruskie

Loading

codecov bot commented Feb 24, 2021 •

edited

Loading