Add deseasonality in SrCnnEntireAnomalyDetect by guinao · Pull Request #5202 · dotnet/machinelearning

guinao · 2020-06-03T10:14:07Z

We are excited to review your PR.

So we can do the best job, please check:

There's a descriptive title that will make sense to other developers some time from now.
There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
You have included any necessary tests in the same PR.

Description

In time series data, seasonality is the presence of variations that occur at specific regular intervals less than a year, such as weekly, monthly, or quarterly. In time series anomaly detection, it is essential to remove the effect of seasonality before detecting anomalies.

In this PR, we add the de-seasonality process in SrCnnEntireAnomalyDetect.

A new SrCnnEntireAnomalyDetect API is introduced.

        /// <summary>
        /// Create <see cref="SrCnnEntireAnomalyDetector"/>, which detects timeseries anomalies for entire input using SRCNN algorithm.
        /// </summary>
        /// <param name="catalog">The AnomalyDetectionCatalog.</param>
        /// <param name="input">Input DataView.</param>
        /// <param name="outputColumnName">Name of the column resulting from data processing of <paramref name="inputColumnName"/>.
        /// The column data is a vector of <see cref="System.Double"/>. The length of this vector varies depending on <paramref name="options.DetectMode"/>.</param>
        /// <param name="inputColumnName">Name of column to process. The column data must be <see cref="System.Double"/>.</param>
        /// <param name="options">Defines the settings of the load operation.</param>
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
        /// [!code-csharp[DetectEntireAnomalyBySrCnn](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/TimeSeries/DetectEntireAnomalyBySrCnn.cs)]
        /// ]]>
        /// </format>
        /// </example>
        public static IDataView DetectEntireAnomalyBySrCnn(this AnomalyDetectionCatalog catalog, IDataView input, string outputColumnName, string inputColumnName, SrCnnEntireAnomalyDetectorOptions options)
            => new SrCnnEntireAnomalyDetector(CatalogUtils.GetEnvironment(catalog), input, outputColumnName, inputColumnName, options);

In the options parameter, if Period is set to a positive integer, de-seasonality will be enabled before detection.

Three de-seasonality methods are introduced.

    public enum SrCnnDeseasonalityMode
    {
        /// <summary>
        /// In this mode, the stl decompose algorithm is used to de-seasonality.
        /// </summary>
        Stl = 0,

        /// <summary>
        /// In this mode, the mean value of points in the same position in a period is substracted to de-seasonality.
        /// </summary>
        Mean = 1,

        /// <summary>
        /// In this mode, the median value of points in the same position in a period is substracted to de-seasonality.
        /// </summary>
        Median = 2
    }

Changes in this PR includes.

Add new constructor for SrCnnEntireAnomalyDetector class that accepts an options class as parameter.
Implement three de-seasonality functions, namely the STL algorithm, the mean method and the median method. The STL algorithm can be referred to at http://www.nniiem.ru/file/news/2016/stl-statistical-model.pdf. The mean/median method substracts the mean/median value of points belong to the same bucket as de-seasonality value.
Add unit tests in TimeSeriesDirectApi file.

…ogic

guinao · 2020-06-03T12:55:14Z

Not used?

I don't think this is used (or should be used)?

Removed this file. #Resolved

guinao · 2020-06-03T12:55:42Z

Not used?

Removed this file #Resolved

src/Microsoft.ML.TimeSeries/Microsoft.ML.TimeSeries.csproj

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs

Microsoft.ML.sln

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs

guinao · 2020-06-04T12:55:24Z

@harishsk is added to the review. #Closed

guinao · 2020-06-04T12:57:33Z

@conhua is added to the review. #Closed

guinao · 2020-06-04T12:58:04Z

@jinhua is added to the review. #Closed

src/Microsoft.ML.TimeSeries/PolynomialModel/AbstractPolynomialModel.cs

src/Microsoft.ML.TimeSeries/STL/FastLoess.cs

src/Microsoft.ML.TimeSeries/STL/InnerStl.cs

src/Microsoft.ML.TimeSeries/PolynomialModel/AbstractPolynomialModel.cs

src/Microsoft.ML.TimeSeries/Deseasonality/MeanDeseasonality.cs

src/Microsoft.ML.TimeSeries/STL/LoessBasicParameters.cs

kere-nel · 2020-06-18T22:15:37Z

src/Microsoft.ML.TimeSeries/STL/LeastSquares.cs

+
+            Contracts.CheckParam(weights.Count == _length, nameof(weights));
+            if (weights.Count != _length)
+                throw Contracts.Except("the weight vector is not equal length to the data points");


Suggestion: "The length of the weight vector is not equal to the length of the data points." #Resolved

Updated, thanks.

In reply to: 442531436 [](ancestors = 442531436)

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs

…omments

guinao · 2020-06-22T03:41:02Z

I updated a description for this PR in the first comment.

In reply to: 646231552 [](ancestors = 646231552)

yaeldekel · 2020-06-22T08:41:42Z

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs

+                else // if (deseasonalityMode == SrCnnDeseasonalityMode.Median)
+                {
+                    _deseasonalityFunction = new MedianDeseasonality();
+                }


This can be changed to a switch statement. #Resolved

In the default case, instead of writing deseasonalityMode == SrCnnDeseasonalityMode.Median you can add

Contracts.Assert(deseasonalityMode == SrCnnDeseasonalityMode.Median);

(that way, if in the future a value is added to the SrCnnDeseasonalityMode enum, it will be easier to detect that this code needs to be updated).

In reply to: 443404193 [](ancestors = 443404193)

yaeldekel · 2020-06-22T08:57:34Z

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs

+                for (int i = 0; i < values.Length; ++i)
+                {
+                    results[i][3] = values[i] - residual[i];
+                }


Can you call GetExpectedValuePeriod instead? #Resolved

yaeldekel · 2020-06-22T08:59:30Z

test/Microsoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs

+        [Theory, CombinatorialData]
+        public void TestSrCnnAnomalyDetectorWithSeasonalData(
+            [CombinatorialValues(SrCnnDeseasonalityMode.Stl, SrCnnDeseasonalityMode.Mean, SrCnnDeseasonalityMode.Median)] SrCnnDeseasonalityMode mode
+        )


) [](start = 8, length = 1)

nit: move this to the line above. #Resolved

yaeldekel · 2020-06-22T09:00:15Z

test/Microsoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs

+            [CombinatorialValues(SrCnnDeseasonalityMode.Stl, SrCnnDeseasonalityMode.Mean, SrCnnDeseasonalityMode.Median)] SrCnnDeseasonalityMode mode
+        )
+        {
+            var ml = new MLContext(1);


MLContext [](start = 25, length = 9)

The base class has an MLContext field. #Resolved

Could you point out the MLContext field? As I find in this class, other tests are creating MLContext by themselves.

In reply to: 443415149 [](ancestors = 443415149)

Sorry, I was wrong. The MLContext is not in BaseTestClass, it is in BaseTestBaseline, which is derived from BaseTestClass.

In reply to: 443532513 [](ancestors = 443532513,443415149)

yaeldekel · 2020-06-22T09:00:50Z

test/Microsoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs

+            [CombinatorialValues(SrCnnDeseasonalityMode.Stl, SrCnnDeseasonalityMode.Mean, SrCnnDeseasonalityMode.Median)] SrCnnDeseasonalityMode mode
+        )
+        {
+            var ml = new MLContext(1);


MLContext [](start = 25, length = 9)

Use the MLContext of the base class. #Resolved

Same as above comment, the base class don't have this field.

In reply to: 443415470 [](ancestors = 443415470)

yaeldekel · 2020-06-22T09:01:21Z

using System.Collections.Generic;

Add header. #Resolved

Refers to: src/Microsoft.ML.TimeSeries/STL/FastLoess.cs:1 in 816479a. [](commit_id = 816479a, deletion_comment = False)

yaeldekel · 2020-06-22T09:02:01Z

using System.Collections.Generic;

For the rest of the new files as well.

In reply to: 647384981 [](ancestors = 647384981)

Refers to: src/Microsoft.ML.TimeSeries/STL/FastLoess.cs:1 in 816479a. [](commit_id = 816479a, deletion_comment = False)

yaeldekel · 2020-06-23T04:56:27Z

src/Microsoft.ML.TimeSeries/STL/FastLoess.cs

+        }
+
+        /// <summary>
+        /// The estimated y values. This is the very cool smoothing method.


This is the very cool smoothing method. [](start = 36, length = 39)

What does this comment mean? #Resolved

Removed, original, it's refer to loess(local weighted regression)

In reply to: 443960923 [](ancestors = 443960923)

harishsk

harishsk

yuyi@microsoft.com added 6 commits June 1, 2020 16:51

add seasonality detect and stl decompose in srcnn

78961c7

optimizations in performance

468d444

Add parameter period in SrCnn interface, remove inner period detect l…

45b6971

…ogic

add periodical data

245731f

add test

604084a

Remove unused files

7c6734f

guinao requested a review from a team as a code owner June 3, 2020 10:14

yuyi@microsoft.com added 3 commits June 3, 2020 18:19

further remove unused codes

8ea07f6

remove unused functions

b398fdb

update

cce849d

yuyi@microsoft.com added 3 commits June 4, 2020 15:51

optimize WeightedRegression; clean code; add null checks

10935c3

recover

b524957

reduce file numbers

d40440b

guinao commented Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/Microsoft.ML.TimeSeries.csproj Outdated Show resolved Hide resolved

restore

92c6aab

mengaims reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

Microsoft.ML.sln Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

move stl related codes to a subfolder

f6e8566

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Jun 4, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

kere-nel reviewed Jun 18, 2020

View reviewed changes

harishsk reviewed Jun 18, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/PolynomialModel/AbstractPolynomialModel.cs Outdated Show resolved Hide resolved

kere-nel reviewed Jun 18, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/Deseasonality/MeanDeseasonality.cs Outdated Show resolved Hide resolved

harishsk reviewed Jun 18, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/STL/LoessBasicParameters.cs Outdated Show resolved Hide resolved

kere-nel reviewed Jun 18, 2020

View reviewed changes

src/Microsoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs Outdated Show resolved Hide resolved

yuyi@microsoft.com added 2 commits June 22, 2020 11:15

refactor polynomial model class and deseasonality functions, refine c…

c3f74d5

…omments

refine

816479a

yaeldekel reviewed Jun 22, 2020

View reviewed changes

update comment

a239a89

yaeldekel reviewed Jun 22, 2020

View reviewed changes

yuyi@microsoft.com added 2 commits June 22, 2020 20:48

updates

1446115

update some wordings

8b10cc0

yaeldekel reviewed Jun 23, 2020

View reviewed changes

update comments

a817829

harishsk approved these changes Jun 25, 2020

View reviewed changes

guinao closed this Jun 26, 2020

guinao reopened this Jun 26, 2020

yuyi@microsoft.com added 2 commits June 26, 2020 18:48

update some comments

8fac10e

wording

2cad48f

harishsk approved these changes Jun 29, 2020

View reviewed changes

harishsk merged commit 33f5f32 into dotnet:master Jun 29, 2020

michaelgsharp mentioned this pull request Sep 27, 2021

DetectEntireAnomalyBySrCnn - No anomalies detected as of version 1.5.4 or greater #5891

Open

ghost locked as resolved and limited conversation to collaborators Mar 18, 2022

Conversation

guinao commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

guinao commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guinao commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guinao commented Jun 4, 2020

Uh oh!

guinao commented Jun 4, 2020

Uh oh!

guinao commented Jun 4, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kere-nel Jun 18, 2020 • edited by harishsk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guinao Jun 22, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guinao commented Jun 22, 2020

Uh oh!

yaeldekel Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 22, 2020

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guinao Jun 22, 2020

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 23, 2020

Choose a reason for hiding this comment

Uh oh!

yaeldekel Jun 22, 2020 • edited by harishsk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guinao Jun 23, 2020

Choose a reason for hiding this comment

Uh oh!

yaeldekel commented Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guinao commented Jun 3, 2020 •

edited

Loading

guinao commented Jun 3, 2020 •

edited

Loading

guinao commented Jun 3, 2020 •

edited

Loading

kere-nel Jun 18, 2020 •

edited by harishsk

Loading

yaeldekel Jun 22, 2020 •

edited

Loading

yaeldekel Jun 22, 2020 •

edited

Loading

yaeldekel Jun 22, 2020 •

edited

Loading

yaeldekel Jun 22, 2020 •

edited

Loading

yaeldekel Jun 22, 2020 •

edited by harishsk

Loading

yaeldekel commented Jun 22, 2020 •

edited

Loading

yaeldekel Jun 23, 2020 •

edited by harishsk

Loading