Skip to content

Fixes #4571. About memory leak when using FeaturizeText. #4576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antoniovs1029 merged 21 commits into from
Dec 18, 2019
Merged

Fixes #4571. About memory leak when using FeaturizeText. #4576

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1182b91
Proposal to change NormStr
antoniovs1029 Dec 13, 2019
19a2d0a
Proposal to change TextNormalizingTransformer.Mapper
antoniovs1029 Dec 13, 2019
d2f6528
Added Proposal 3 - Modifying NormStr.Pool.Get
antoniovs1029 Dec 13, 2019
20cb8de
Revert "Proposal to change NormStr"
antoniovs1029 Dec 16, 2019
c3a1174
Revert "Proposal to change TextNormalizingTransformer.Mapper"
antoniovs1029 Dec 16, 2019
d36ea8e
Added flag to control creation of a new str
antoniovs1029 Dec 16, 2019
eca5b13
Added Benchmark Test
antoniovs1029 Dec 16, 2019
62af85b
Added minor comment to benchmark
antoniovs1029 Dec 16, 2019
e82485b
Nit - Fixed references to PRs and Issues
antoniovs1029 Dec 16, 2019
3dfa0b9
Nit - renamed flag in NormStr.Pool.Get()
antoniovs1029 Dec 16, 2019
198caed
Added space
antoniovs1029 Dec 16, 2019
48892ea
Added duplicateStr prameter to NormStr.Pool.Add method
antoniovs1029 Dec 17, 2019
ba5dadd
Changed threshold in FeaturizeText Benchmark to be 400MB instead of 1GB
antoniovs1029 Dec 17, 2019
94b2424
Typo
antoniovs1029 Dec 17, 2019
a5eb6c8
Changed threshold to 240MB
antoniovs1029 Dec 17, 2019
ce181b6
Added Benchmark output
antoniovs1029 Dec 17, 2019
25d6250
Removed Assertion from Benchmark
antoniovs1029 Dec 17, 2019
70886fe
Added test with memory usage assertion
antoniovs1029 Dec 17, 2019
b5766fb
Updated test comment
antoniovs1029 Dec 17, 2019
c94a293
Comment out test Console.WriteLine
antoniovs1029 Dec 17, 2019
2998b05
Removing test
antoniovs1029 Dec 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Core/Utilities/NormStr.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ internal sealed class NormStr
private NormStr(ReadOnlyMemory<char> str, int id, uint hash)
{
Contracts.Assert(id >= 0 || id == -1 && str.IsEmpty);
Value = str;
Value = new ReadOnlyMemory<char>(str.ToString().ToCharArray());
Comment thread
antoniovs1029 marked this conversation as resolved.
Outdated
Id = id;
_hash = hash;
}
Expand Down Expand Up @@ -136,7 +136,7 @@ public NormStr Get(ReadOnlyMemory<char> str, bool add = false)
}
Contracts.Assert(ins == -1);

return add ? AddCore(str, hash) : null;
return add ? AddCore(str.ToString().AsMemory(), hash) : null;
Comment thread
antoniovs1029 marked this conversation as resolved.
Outdated
}

/// <summary>
Expand Down
3 changes: 2 additions & 1 deletion src/Microsoft.ML.Transforms/Text/TextNormalizing.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,8 @@ private void NormalizeSrc(in ReadOnlyMemory<char> src, ref ReadOnlyMemory<char>
if (min == 0)
{
Host.Assert(src.Length == len);
dst = src;
buffer.AppendSpan(span);
dst = buffer.ToString().AsMemory();
}
Comment thread
antoniovs1029 marked this conversation as resolved.
else
{
Expand Down