-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Move DataFrame to machinelearning #5641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 53 commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
df1661b
Change namespace to Microsoft.Data.Analysis (#2773)
f09c30c
APIs for reversed binary operators (#2769)
afe3e61
Throw for incompatible inPlace (#2778)
5c4b1b4
Version, Tags and Description for Nuget (#2779)
5de343c
Flags for release (#2781)
81f3d42
Fix the Description method to not throw (#2786)
c6eb2f7
Use dataTypes if it passed in to LoadCsv (#2791)
7cee9d9
Creating a `Rows` property, similar to `Columns` (#2794)
e64cbad
DataFrame.LoadCsv throws an exception on projects targeting < netcore…
eerhardt 303ba62
Params constructor on DataFrame (#2800)
dc4f9b0
Remove `T : unmanaged` constraint from DataFrameColumn.BinaryOperatio…
838350b
Bump version to 0.2.0 (#2803)
0fa210d
Add Apply<TResult>method to PrimitiveDataFrameColumn (#2807)
zHaytam 430ac09
Add additional tests for ReadCsv (#2811)
jwood803 70bb9e9
Added static factory methods to DataFrameColumn (#2808)
MgSam 82c315f
Append rows to a DataFrame (#2823)
afdbc5b
Move corefxlab to arcade (#2795)
355d3fb
Rename Sort to OrderBy (#2814)
jwood803 9e10004
Explode column types and generate converters (#2857)
1544c23
Address remaining concerns from the 2nd DataFrame API Review (#2861)
8d7fb66
Add binary operations and operators on the exploded columns (#2867)
7ef10ba
Add APIs to get the strongly typed columns from a DataFrame (#2878)
4072f96
Enable xml docs for Data.Analysis (#2882)
a6c34d0
Add Apply to ArrowStringDataFrameColumn (#2889)
9c80608
Support for Exploded columns types in Arrow and IO scenarios (#2885)
d120982
Bump version (#2890)
59df417
Fix versioning to allow for individual stable packages (#2891)
d79dd2f
Bump Microsoft.Data.Analysis version to 0.4.0 (#2892)
eerhardt 28140bd
Fix https://github.com/dotnet/corefxlab/issues/2906 (#2907)
5c3ac8b
Improve LoadCsv to handle null values when deducing the column types …
0bef531
Create a 0.4.0 package (#2918)
b215eb4
Revert "Create a 0.4.0 package (#2918)" (#2919)
3b4aafa
Produce a 0.4.0 build (#2920)
8d08434
Default Length for StringDataFrameColumn (#2921) (#2923)
rhysparry 7dcf184
Increment version and stop producing stable packages (#2922)
881886b
Add DataFrame object formatter. (#2931)
dcostea 6e60307
Fix a bug in InsertColumn
RamonWill 6c2d800
Add Microsoft.Data.Analysis.nuget project (#2933)
dcostea 7ebe8bc
ColumnName indexer on DataFrame (#2959)
54633a2
Implement FillNulls() for ArrowStringDataFrameColumn with inPlace: fa…
RamonWill 4e6d801
Prevent DataFrame.Sample() method from returning duplicated rows (#2939)
RamonWill 81d0ba5
Add WriteCsv plus unit tests. (#2947)
dcostea db5c49e
Missing values default to a `StringDataFrameColumn` (#2982)
cb7ab00
Update FromArrowRecordBatches for dotnet-spark (#2978)
cff30e3
Implement DataFrame.LoadCsvFromString (#2988)
e7a9c42
Part 1 of porting the csv reader (#2997)
881c619
Merge branch 'port' of ../corefxlab into DataFrame_1
1c1c3a8
Move to the test folder
fea6bd2
Suppress warnings
0b8541a
Move extensions reference out of props
bf82179
Address feedback
9d74a83
Address feedback
fa39b74
Default to preview version
216554a
Update nuget.config
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
155 changes: 155 additions & 0 deletions
155
src/Microsoft.Data.Analysis.Interactive/DataFrameKernelExtension.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
| // See the LICENSE file in the project root for more information. | ||
|
|
||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.Linq; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.AspNetCore.Html; | ||
| using Microsoft.DotNet.Interactive; | ||
| using Microsoft.DotNet.Interactive.Formatting; | ||
| using static Microsoft.DotNet.Interactive.Formatting.PocketViewTags; | ||
|
|
||
| namespace Microsoft.Data.Analysis.Interactive | ||
| { | ||
| public class DataFrameKernelExtension : IKernelExtension | ||
| { | ||
| public Task OnLoadAsync(Kernel kernel) | ||
| { | ||
| RegisterDataFrame(); | ||
|
|
||
| return Task.CompletedTask; | ||
| } | ||
|
|
||
| public static void RegisterDataFrame() | ||
| { | ||
| Formatter<DataFrame>.Register((df, writer) => | ||
| { | ||
| const int MAX = 10000; | ||
| const int SIZE = 10; | ||
|
|
||
| var uniqueId = DateTime.Now.Ticks; | ||
|
|
||
| var header = new List<IHtmlContent> | ||
| { | ||
| th(i("index")) | ||
| }; | ||
| header.AddRange(df.Columns.Select(c => (IHtmlContent)th(c.Name))); | ||
|
|
||
| if (df.Rows.Count > SIZE) | ||
| { | ||
| var maxMessage = df.Rows.Count > MAX ? $" (showing a max of {MAX} rows)" : string.Empty; | ||
| var title = h3[style: "text-align: center;"]($"DataFrame - {df.Rows.Count} rows {maxMessage}"); | ||
|
|
||
| // table body | ||
| var maxRows = Math.Min(MAX, df.Rows.Count); | ||
| var rows = new List<List<IHtmlContent>>(); | ||
| for (var index = 0; index < maxRows; index++) | ||
| { | ||
| var cells = new List<IHtmlContent> | ||
| { | ||
| td(i((index))) | ||
| }; | ||
| foreach (var obj in df.Rows[index]) | ||
| { | ||
| cells.Add(td(obj)); | ||
| } | ||
| rows.Add(cells); | ||
| } | ||
|
|
||
| //navigator | ||
| var footer = new List<IHtmlContent>(); | ||
| BuildHideRowsScript(uniqueId); | ||
|
|
||
| var paginateScriptFirst = BuildHideRowsScript(uniqueId) + GotoPageIndex(uniqueId, 0) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptFirst]("⏮")); | ||
|
|
||
| var paginateScriptPrevTen = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, -10, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptPrevTen]("⏪")); | ||
|
|
||
| var paginateScriptPrev = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, -1, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptPrev]("◀️")); | ||
|
|
||
| footer.Add(b[style: "margin: 2px;"]("Page")); | ||
| footer.Add(b[id: $"page_{uniqueId}", style: "margin: 2px;"]("1")); | ||
|
|
||
| var paginateScriptNext = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, 1, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptNext]("▶️")); | ||
|
|
||
| var paginateScriptNextTen = BuildHideRowsScript(uniqueId) + UpdatePageIndex(uniqueId, 10, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptNextTen]("⏩")); | ||
|
|
||
| var paginateScriptLast = BuildHideRowsScript(uniqueId) + GotoPageIndex(uniqueId, (maxRows - 1) / SIZE) + BuildPageScript(uniqueId, SIZE); | ||
| footer.Add(button[style: "margin: 2px;", onclick: paginateScriptLast]("⏭️")); | ||
|
|
||
| //table | ||
| var t = table[id: $"table_{uniqueId}"]( | ||
| caption(title), | ||
| thead(tr(header)), | ||
| tbody(rows.Select(r => tr[style: "display: none"](r))), | ||
| tfoot(tr(td[colspan: df.Columns.Count + 1, style: "text-align: center;"](footer))) | ||
| ); | ||
| writer.Write(t); | ||
|
|
||
| //show first page | ||
| writer.Write($"<script>{BuildPageScript(uniqueId, SIZE)}</script>"); | ||
| } | ||
| else | ||
| { | ||
| var rows = new List<List<IHtmlContent>>(); | ||
| for (var index = 0; index < df.Rows.Count; index++) | ||
| { | ||
| var cells = new List<IHtmlContent> | ||
| { | ||
| td(i((index))) | ||
| }; | ||
| foreach (var obj in df.Rows[index]) | ||
| { | ||
| cells.Add(td(obj)); | ||
| } | ||
| rows.Add(cells); | ||
| } | ||
|
|
||
| //table | ||
| var t = table[id: $"table_{uniqueId}"]( | ||
| thead(tr(header)), | ||
| tbody(rows.Select(r => tr(r))) | ||
| ); | ||
| writer.Write(t); | ||
| } | ||
| }, "text/html"); | ||
| } | ||
|
|
||
| private static string BuildHideRowsScript(long uniqueId) | ||
| { | ||
| var script = $"var allRows = document.querySelectorAll('#table_{uniqueId} tbody tr:nth-child(n)'); "; | ||
| script += "for (let i = 0; i < allRows.length; i++) { allRows[i].style.display='none'; } "; | ||
| return script; | ||
| } | ||
|
|
||
| private static string BuildPageScript(long uniqueId, int size) | ||
| { | ||
| var script = $"var page = parseInt(document.querySelector('#page_{uniqueId}').innerHTML) - 1; "; | ||
| script += $"var pageRows = document.querySelectorAll(`#table_{uniqueId} tbody tr:nth-child(n + ${{page * {size} + 1 }})`); "; | ||
| script += $"for (let j = 0; j < {size}; j++) {{ pageRows[j].style.display='table-row'; }} "; | ||
| return script; | ||
| } | ||
|
|
||
| private static string GotoPageIndex(long uniqueId, long page) | ||
| { | ||
| var script = $"document.querySelector('#page_{uniqueId}').innerHTML = {page + 1}; "; | ||
| return script; | ||
| } | ||
|
|
||
| private static string UpdatePageIndex(long uniqueId, int step, long maxPage) | ||
| { | ||
| var script = $"var page = parseInt(document.querySelector('#page_{uniqueId}').innerHTML) - 1; "; | ||
| script += $"page = parseInt(page) + parseInt({step}); "; | ||
| script += $"page = page < 0 ? 0 : page; "; | ||
| script += $"page = page > {maxPage} ? {maxPage} : page; "; | ||
| script += $"document.querySelector('#page_{uniqueId}').innerHTML = page + 1; "; | ||
| return script; | ||
| } | ||
| } | ||
| } |
18 changes: 18 additions & 0 deletions
18
src/Microsoft.Data.Analysis.Interactive/Microsoft.Data.Analysis.Interactive.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>netcoreapp3.1</TargetFramework> | ||
| <IsPackable>false</IsPackable> | ||
| <NoWarn>$(NoWarn);MSML_ParameterLocalVarName;SA1028</NoWarn> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.DotNet.Interactive" Version="$(MicrosoftDotNetInteractiveVersion)" /> | ||
| <PackageReference Include="Microsoft.DotNet.Interactive.Formatting" Version="$(MicrosoftDotNetInteractiveFormattingVersion)" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\Microsoft.Data.Analysis\Microsoft.Data.Analysis.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tabs hurt my eyes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ugh, this setting must be mis-matched locally on my machine between VSCode and VS :\ Will fix in a bit
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of curiosity, how did you see tabs in this file in GH? I changed all tabs to spaces (no idea how tabs occurred in the first place) in 4 csprojs in the PR, but unfortunately there is no "Change all tabs to spaces in all files" in VS I think. For now, a solution wide regex search yields no tabs in source code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looked too indented then I saw it while trying to select white space.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I use CodeFlow to review PRs, and you can tell it to show whitespace. Arrows are tabs, dots are spaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Has codeflow been fixed to work better with Github comment threads? That was a deal breaker in the past.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it still has some issues with comment threads. Typically I only create new conversations from CodeFlow. And respond to threads in GH. I usually have them both open on 2 different windows. I just like seeing the full file, searching across the change, seeing the tree view of files, etc in CodeFlow. It's a much better experience for me.