[HOPSWORKS-3125] feature view (#1089)

* [HOPSWORKS-3125] Feature view (#923) * [HOPSWORKS-2945] [FeatureView] Implement activity endpoints (#829) * init * big fixes * Update ActivityResource.java * Update FeatureViewController.java * Update FeatureViewController.java * add logging of activity for FV * Update feature_store_activity_spec.rb * Update feature_store_activity_spec.rb * Update featurestore_helper.rb * Update featurestore_helper.rb * Update FeatureViewController.java * add statistics * [HOPSWORKS-2947] [ModelFeature] Implement PrepareStatementResource (#824) * init * improvements * small changes to the backend * temp * Update FeatureViewController.java * Update preparedstatements_spec.rb * standardize * Update featurestore_helper.rb * small updates * small changes * Update preparedstatements_spec.rb * addressing feedback * change endpoint to lower case * [HOPSWORKS-2943] Add query resource (#834) * implement batch query endpoint * handle empty start or end time * throw feature store exception * return original query * check nested join * fix NPE for join * add feature group to feature * return features properly. * add IT for batch query * fix batch query test * set commit time to query * add timetravel condition * reuse get_query from TrainingDatasetController * pass withLabel and isHiveEngine when constructing batch query * reformat * test get query * add test for get query * remove object variables * fix indentation * refactor getJoinsSorted * fix format * reformat * use getSingleByNameVersionAndFeatureStore * add query filter to FV * refactor event time filter, add unit test * fix resourcerequest NPE * add featureview to trainingdatasetfilter * add query filter to IT Co-authored-by: Kenneth Mak <[email protected]> * [HOPSWORKS-2944] Implement keyword related endpoints (#842) * needs inode to function * add inode * Update FeatureView.java * Update FeatureViewController.java * Update ProjectController.java * changes with inode * Update FeatureViewController.java * inode work * Update FeatureViewController.java * Update FeatureViewController.java * Update FeatureViewController.java * Update FeatureViewController.java * Update ProjectController.java * permissions * small changes * tests * Update featureview_keywords_spec.rb * some feedback addressed * add featureView xattr * Update FeatureViewBuilder.java * Update FeatureViewController.java * Update FeatureViewController.java * Update FeatureViewController.java * save hopsfs connector * Update ProjectController.java * Update FeatureViewController.java * Update FeatureViewController.java * add features to featureViewDTO * Update FeatureViewBuilder.java * Update HopsFSProvenanceController.java * removing fv dir * Update FeaturestoreController.java * path changes for fv * restructure fv path * Update HopsFSProvenanceController.java * Update HopsworksJAXBContext.java * Update featureview_keywords_spec.rb * some of the comments addressed * remove keyword controller duplication * make createFeatureView use and return FeatureView instead of FeatureViewDTO * Update FeatureViewBuilder.java * Update FeaturestoreKeywordResource.java * change feature view path * Update featureview_keywords_spec.rb * Update featureview_keywords_spec.rb * [HOPSWORKS-2946] Add transformation resource and statistics resource (#856) * get transformation function * statistics resource * fix statistics dto forTransformation * fix transformation uri * add IT for transformation * change statistics folder name * remove td from feature view statistics * delete statistics along with feature view * add IT for feature view statistics * Revert "add IT for feature view statistics" This reverts commit db49dd0971fe1cdc7929f8fd0c13a5249c451ff0. * Revert "delete statistics along with feature view" This reverts commit 697768d51b186c14e2d451858f0552df1d8ffa77. * Revert "change statistics folder name" This reverts commit 922c17f94c14c50a771b7d70f04154712305ab19. * Revert "fix statistics dto forTransformation" This reverts commit 3143984906d6cc8eced6fb3349d24c4a0a4df360. * Revert "statistics resource" This reverts commit 7361bd8ff06ffff239285bfb8360f25ba14ef68a. * fix access control * refactor uri * [FeatureView] return original query as expansion (#866) * [HOPSWORKS-2942] Implement TagResource (#864) * init * Update TagResource.java * Update TagResource.java * bug fixing/standardization * tests * generic tags resource * fix for the abstract tags resource * some feedback addressed * Update TagsResource.java Co-authored-by: Alex Ormenisan <[email protected]> * [HOPSWORKS-3064] [FeatureView] Allow update of metadata in FeatureView resource (#895) * init * Update FeatureViewResource.java * bug fixing * Update featureview_spec.rb * Update featureview_spec.rb * [HOPSWORKS-2941] Implement Training Dataset resource (#845) * implement batch query endpoint * throw feature store exception * return features properly. * add IT for batch query * fix batch query test * reuse get_query from TrainingDatasetController * create td * compute td * get td * delete td * delete td data only * fix compile * reformat * set feature view * set featurestore and project * implement statistics resource * handle hopsfstrainingdataset is null * return all td * fix create and get td * do not return query in dto * skip feature in dto * set query in job config * add td IT * add td IT * fix internal td it * external td IT * add external td test * rename create td method * reformat * move query related methods to QueryController * revert unintended changes * refactor get FV * fix failed tests * fix comments * check name of training dataset in IT * check name of feature view against td * keep td folder when deleting data only * fix failed test * fix failed test * remove extra update * return features after creating fv * fix td exist error * create feature view td job * do not assume a split if splits is empty * remove extra get annotation * set default td name = fv name + version * return batch query as object * set start time and end time to td * return feature group in correct type * throw exception if feature not found * fix test * fix feature to featuredto map * fix query test * [Append] training dataset resource remove redundant lines in test * rebase * [Hopsworks-3063] [FeatureView] Allow update of metadata in TrainingDataset resource (#921) * init * Update featurestore_helper.rb * Update featureview_trainingdataset_spec.rb * remove label from feature view table (#924) * [HOPSWORKS-3073] [FeatureView] Remove ResourceRequest from FeatureViewController (#925) * init small changes also on other endpoints to standardize * addressing the feedback * [HOPSWORKS-2941] [Append] td resource (#930) * fix dataset * fix infinite loop when getting all feature groups * change event time type * use date as event time in batch query * add in memory td * fix prepared statement comment * refactor commit time * fix keyword resource * fix transformation * remove unused import * assign storage connector to in-memory td * fix integration test Co-authored-by: Ralf <[email protected]> Co-authored-by: Kenneth Mak <[email protected]> Co-authored-by: Alex Ormenisan <[email protected]> (cherry picked from commit af28c64) # Conflicts: # featurestore-ee/src/main/java/io/hops/hopsworks/featurestore/KeywordController.java # hopsworks-IT/src/test/ruby/spec/ee_tags_spec.rb # hopsworks-IT/src/test/ruby/spec/helpers/tag_helper.rb # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/featureview/FeatureViewResource.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/featureview/FeatureViewService.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/featureview/FeatureViewTagResource.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/preparestatement/PreparedStatementResource.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/query/QueryResource.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/trainingdataset/TrainingDatasetResource.java # hopsworks-api/src/main/java/io/hops/hopsworks/api/featurestore/transformation/TransformationResource.java * [HOPSWORKS-3125] [append] Feature view fix end time error #944 (cherry picked from commit 3c064ff)
logicalclocks · May 19, 2022 · 5eef6ac · 5eef6ac
1 parent cd97302
commit 5eef6ac
Show file tree

Hide file tree

Showing 65 changed files with 3,900 additions and 588 deletions.
diff --git a/hopsworks-IT/src/test/ruby/spec/feature_store_activity_spec.rb b/hopsworks-IT/src/test/ruby/spec/feature_store_activity_spec.rb
@@ -206,5 +206,37 @@
       expect(activity["items"][0]["type"]).to eql("METADATA")
       expect(activity["items"][0]["metadata"]).to eql("The training dataset was created")
     end
+
+    it "should be able to retrieve feature view creation event" do
+      featurestore_id = get_featurestore_id(@project[:id])
+      json_result, _ = create_cached_featuregroup(@project[:id], featurestore_id)
+      expect_status_details(201)
+
+      parsed_json = JSON.parse(json_result)
+      fg_id = parsed_json["id"]
+      # create queryDTO object
+      query = {
+        leftFeatureGroup: {
+          id: fg_id
+        },
+        leftFeatures: ['testfeature'].map do |feat_name|
+          {name: feat_name}
+        end,
+        joins: []
+      }
+
+      json_result, _ = create_feature_view(@project.id, featurestore_id, query)
+      parsed_json = JSON.parse(json_result)
+      expect_status(201)
+
+      feature_view_name = parsed_json["name"]
+      feature_view_version = parsed_json["version"]
+      json_result = get "#{ENV['HOPSWORKS_API']}/project/#{@project.id}/featurestores/#{featurestore_id}/featureview/#{feature_view_name}/version/#{feature_view_version}/activity"
+      expect_status_details(200)
+
+      activity = JSON.parse(response.body)
+      expect(activity["items"][0]["type"]).to eql("METADATA")
+      expect(activity["items"][0]["metadata"]).to eql("The feature view was created")
+    end
   end
 end
diff --git a/hopsworks-IT/src/test/ruby/spec/featureview_query.rb b/hopsworks-IT/src/test/ruby/spec/featureview_query.rb
@@ -0,0 +1,138 @@
+# This file is part of Hopsworks
+# Copyright (C) 2022, Logical Clocks AB. All rights reserved
+#
+# Hopsworks is free software: you can redistribute it and/or modify it under the terms of
+# the GNU Affero General Public License as published by the Free Software Foundation,
+# either version 3 of the License, or (at your option) any later version.
+#
+# Hopsworks is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
+# without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License along with this program.
+# If not, see <https://www.gnu.org/licenses/>.
+
+require 'json'
+
+describe "On #{ENV['OS']}" do
+  after(:all) { clean_all_test_projects(spec: "featureviewquery") }
+
+  describe "feature view query" do
+    describe "internal" do
+      context 'with valid project, featurestore service enabled' do
+        before :all do
+          with_valid_project
+        end
+
+        it "should be able to create batch query" do
+          featurestore_id = get_featurestore_id(@project.id)
+          featurestore_name = get_featurestore_name(@project.id)
+          featuregroup_suffix = short_random_id
+          project_name = @project.projectname.downcase
+          query = make_sample_query(@project, featurestore_id, featuregroup_suffix: featuregroup_suffix)
+          json_result, _ = create_feature_view(@project.id, featurestore_id, query)
+          parsed_json = JSON.parse(json_result)
+          expect_status(201)
+
+          feature_view_name = parsed_json["name"]
+          feature_view_version = parsed_json["version"]
+          query_result = get "#{ENV['HOPSWORKS_API']}/project/#{@project.id}/featurestores/#{featurestore_id}/featureview/#{feature_view_name}/version/#{feature_view_version}/query/batch?start_time=1234&end_time=4321"
+          expect_status_details(200)
+
+          parsed_query_result = JSON.parse(query_result)
+          expect(parsed_query_result["featureStoreId"]).to eql(featurestore_id)
+          expect(parsed_query_result["featureStoreName"]).to eql(featurestore_name)
+          expect(parsed_query_result["leftFeatureGroup"]["id"]).to eql(query[:leftFeatureGroup][:id])
+          expect(parsed_query_result["leftFeatures"][0]["name"]).to eql(query[:leftFeatures][0][:name])
+          expect(parsed_query_result["leftFeatures"][1]["name"]).to eql(query[:leftFeatures][1][:name])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatureGroup"]["id"]).to eql(query[:joins][0][:query][:leftFeatureGroup][:id])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatures"][0]["name"]).to eql(query[:joins][0][:query][:leftFeatures][1][:name])
+          # a_testfeature1 > 0
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["feature"]["name"]).to eql(query[:filter][:leftFilter][:feature][:name])
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["condition"]).to eql(query[:filter][:leftFilter][:condition])
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["value"]).to eql(query[:filter][:leftFilter][:value])
+          # ts <= 1234
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["feature"]["name"]).to eql("ts")
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["condition"]).to eql("GREATER_THAN_OR_EQUAL")
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["value"]).to eql("1234")
+          # ts >= 4321
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["feature"]["name"]).to eql("ts")
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["condition"]).to eql("LESS_THAN_OR_EQUAL")
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["value"]).to eql("4321")
+        end
+
+        it "should be able to retrieve original query" do
+          featurestore_id = get_featurestore_id(@project.id)
+          featurestore_name = get_featurestore_name(@project.id)
+          project_name = @project.projectname.downcase
+          featuregroup_suffix = short_random_id
+          query = make_sample_query(@project, featurestore_id, featuregroup_suffix: featuregroup_suffix)
+          json_result, _ = create_feature_view(@project.id, featurestore_id, query)
+          parsed_json = JSON.parse(json_result)
+          expect_status(201)
+
+          feature_view_name = parsed_json["name"]
+          feature_view_version = parsed_json["version"]
+          query_result = get "#{ENV['HOPSWORKS_API']}/project/#{@project.id}/featurestores/#{featurestore_id}/featureview/#{feature_view_name}/version/#{feature_view_version}/query"
+          expect_status_details(200)
+          parsed_query_result = JSON.parse(query_result)
+          expect(parsed_query_result["featureStoreId"]).to eql(featurestore_id)
+          expect(parsed_query_result["featureStoreName"]).to eql(featurestore_name)
+          expect(parsed_query_result["leftFeatureGroup"]["id"]).to eql(query[:leftFeatureGroup][:id])
+          expect(parsed_query_result["leftFeatures"][0]["name"]).to eql(query[:leftFeatures][0][:name])
+          expect(parsed_query_result["leftFeatures"][1]["name"]).to eql(query[:leftFeatures][1][:name])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatureGroup"]["id"]).to eql(query[:joins][0][:query][:leftFeatureGroup][:id])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatures"][0]["name"]).to eql(query[:joins][0][:query][:leftFeatures][1][:name])
+          expect(parsed_query_result["filter"]["leftFilter"]["feature"]["name"]).to eql(query[:filter][:leftFilter][:feature][:name])
+          expect(parsed_query_result["filter"]["leftFilter"]["condition"]).to eql(query[:filter][:leftFilter][:condition])
+          expect(parsed_query_result["filter"]["leftFilter"]["value"]).to eql(query[:filter][:leftFilter][:value])
+        end
+
+        it "should be able to create batch query using retrieved query" do
+          featurestore_id = get_featurestore_id(@project.id)
+          featurestore_name = get_featurestore_name(@project.id)
+          project_name = @project.projectname.downcase
+          featuregroup_suffix = short_random_id
+          query = make_sample_query(@project, featurestore_id, featuregroup_suffix: featuregroup_suffix)
+          json_result, _ = create_feature_view(@project.id, featurestore_id, query)
+          expect_status(201)
+          parsed_json = JSON.parse(json_result)
+
+          feature_view_name = parsed_json["name"]
+          feature_view_version = parsed_json["version"]
+          query_result = get "#{ENV['HOPSWORKS_API']}/project/#{@project.id}/featurestores/#{featurestore_id}/featureview/#{feature_view_name}/version/#{feature_view_version}/query"
+          expect_status_details(200)
+          parsed_query_result = JSON.parse(query_result)
+          json_result, _ = create_feature_view(@project.id, featurestore_id, parsed_query_result)
+          expect_status(201)
+          parsed_json_new = JSON.parse(json_result)
+          feature_view_version_new = parsed_json_new["version"]
+
+          query_result = get "#{ENV['HOPSWORKS_API']}/project/#{@project.id}/featurestores/#{featurestore_id}/featureview/#{feature_view_name}/version/#{feature_view_version_new}/query/batch?start_time=1234&end_time=4321"
+          expect_status_details(200)
+
+          parsed_query_result = JSON.parse(query_result)
+          expect(parsed_query_result["featureStoreId"]).to eql(featurestore_id)
+          expect(parsed_query_result["featureStoreName"]).to eql(featurestore_name)
+          expect(parsed_query_result["leftFeatureGroup"]["id"]).to eql(query[:leftFeatureGroup][:id])
+          expect(parsed_query_result["leftFeatures"][0]["name"]).to eql(query[:leftFeatures][0][:name])
+          expect(parsed_query_result["leftFeatures"][1]["name"]).to eql(query[:leftFeatures][1][:name])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatureGroup"]["id"]).to eql(query[:joins][0][:query][:leftFeatureGroup][:id])
+          expect(parsed_query_result["joins"][0]["query"]["leftFeatures"][0]["name"]).to eql(query[:joins][0][:query][:leftFeatures][1][:name])
+          # a_testfeature1 > 0
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["feature"]["name"]).to eql(query[:filter][:leftFilter][:feature][:name])
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["condition"]).to eql(query[:filter][:leftFilter][:condition])
+          expect(parsed_query_result["filter"]["leftLogic"]["leftLogic"]["leftFilter"]["value"]).to eql(query[:filter][:leftFilter][:value])
+          # ts <= 1234
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["feature"]["name"]).to eql("ts")
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["condition"]).to eql("GREATER_THAN_OR_EQUAL")
+          expect(parsed_query_result["filter"]["leftLogic"]["rightLogic"]["leftFilter"]["value"]).to eql("1234")
+          # ts >= 4321
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["feature"]["name"]).to eql("ts")
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["condition"]).to eql("LESS_THAN_OR_EQUAL")
+          expect(parsed_query_result["filter"]["rightLogic"]["leftFilter"]["value"]).to eql("4321")
+        end
+      end
+    end
+  end
+end
diff --git a/hopsworks-IT/src/test/ruby/spec/featureview_spec.rb b/hopsworks-IT/src/test/ruby/spec/featureview_spec.rb
@@ -0,0 +1,68 @@
+# This file is part of Hopsworks
+# Copyright (C) 2022, Logical Clocks AB. All rights reserved
+#
+# Hopsworks is free software: you can redistribute it and/or modify it under the terms of
+# the GNU Affero General Public License as published by the Free Software Foundation,
+# either version 3 of the License, or (at your option) any later version.
+#
+# Hopsworks is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
+# without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License along with this program.
+# If not, see <https://www.gnu.org/licenses/>.
+
+require 'json'
+
+describe "On #{ENV['OS']}" do
+  after(:all) {clean_all_test_projects(spec: "featureview")}
+
+  describe "feature view" do
+    describe "internal" do
+      context 'with valid project, featurestore service enabled' do
+        before :all do
+          with_valid_project
+        end
+
+        it "should not be able to update feature view that doesnt exist" do
+          featurestore_id = get_featurestore_id(@project.id)
+
+          json_data = {
+            name: "feature_view_name",
+            version: 1,
+            description: "testfeatureviewdescription"
+          }
+
+          update_feature_view(@project.id, featurestore_id, json_data)
+          expect_status(404)
+        end
+
+        it "should be able to update the description of a feature view" do
+          featurestore_id = get_featurestore_id(@project.id)
+
+          json_result, fg_name = create_cached_featuregroup(@project.id, featurestore_id, online:true)
+          parsed_json = JSON.parse(json_result)
+
+          json_result, _ = create_feature_view_from_feature_group(@project.id, featurestore_id, parsed_json)
+          parsed_json = JSON.parse(json_result)
+          expect_status(201)
+          featureview_name = parsed_json["name"]
+          featureview_version = parsed_json["version"]
+
+          new_description = "new_testfeatureviewdescription"
+          json_data = {
+            name: featureview_name,
+            version: featureview_version,
+            description: new_description
+          }
+
+          json_result = update_feature_view(@project.id, featurestore_id, json_data)
+          parsed_json = JSON.parse(json_result)
+          expect_status(200)
+
+          expect(parsed_json["description"]).to eql(new_description)
+        end
+      end
+    end
+  end
+end