-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/support cosmos #145
base: main
Are you sure you want to change the base?
Commits on Dec 12, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 48394f1 - Browse repository at this point
Copy the full SHA 48394f1View commit details -
Configuration menu - View commit details
-
Copy full SHA for b6b48a3 - Browse repository at this point
Copy the full SHA b6b48a3View commit details
Commits on Dec 18, 2022
-
Configuration menu - View commit details
-
Copy full SHA for fa5c9c4 - Browse repository at this point
Copy the full SHA fa5c9c4View commit details -
* Adding tests/environment folder to store datasets and bicep templates for test sources * Added scripts to create databricks jobs and a notebook to mount storage on Databricks * Making test environments more consistent across notebooks (secret scope, environment variables) * Handle of tests were modified to correct mistakes not caught in source controlled versions * Added documentation for testing environment including what secrets are used and what they look like * Adding requirements.txt file for environment deployment * Hive tests should run without additional intervention (i.e. use CREATE IF NOT EXISTS) * Removing production env deployment * Remove the wasbs with parameters test * After updating all jobdefs to be ready for upload, the run-tests script needed to look at .name instead of .settings.name * Unfortunately, when calling the jobs API, it returns a .settings.name which must be used
Configuration menu - View commit details
-
Copy full SHA for 9c71e45 - Browse repository at this point
Copy the full SHA 9c71e45View commit details
Commits on Jan 6, 2023
-
Resolving race condition based on app settings being deployed while m…
…s deploy is going and event hub authorization rules deploying at the same time. Co-authored-by: hmoazam <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 73e656e - Browse repository at this point
Copy the full SHA 73e656eView commit details
Commits on Feb 12, 2023
-
Bump cryptography from 38.0.4 to 39.0.1 in /tests/environment
Bumps [cryptography](https://github.com/pyca/cryptography) from 38.0.4 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@38.0.4...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 612f4c0 - Browse repository at this point
Copy the full SHA 612f4c0View commit details
Commits on Apr 28, 2023
-
Added unit test and integration test for Azure MySQL, and updated LIM…
…ITATIONS.md to indicate MySQL support (#149)
Configuration menu - View commit details
-
Copy full SHA for b74ad2b - Browse repository at this point
Copy the full SHA b74ad2bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f42108 - Browse repository at this point
Copy the full SHA 6f42108View commit details -
Configuration menu - View commit details
-
Copy full SHA for e83d8ac - Browse repository at this point
Copy the full SHA e83d8acView commit details -
* Adding bicep file * Added unit and integration tests for Postgres. Updated LIMITATIONS.md to describe support and limitations
Configuration menu - View commit details
-
Copy full SHA for 7099edd - Browse repository at this point
Copy the full SHA 7099eddView commit details -
* Implemented support for kusto by updating mappings. * Implemented Kusto support and added unit and integration tests
Configuration menu - View commit details
-
Copy full SHA for 6296607 - Browse repository at this point
Copy the full SHA 6296607View commit details -
Refactoring SelectReturnEntity to be more clear
* checking for resource set types as its own method * checking for blob or data lake file path or filesystem types as its own method * refactored checking for adf relationships for blob/data lake file path types to its own method * Behavior Change for blob/data lake file path types that match: follow the insert into first position of valid entities rather than return a list with only the entity * Renamed ResourceSet_QualifiedNames_Match to QualifiedNames_Match_After_Normalizing and added the trim of the trailing slash for comaprisons * Using QualifiedNames_Match_After_Normalizing when comparing as the dfs vs blob comparison was preventing folder paths from matching if they're using wasbs but scanned for azure data lake gen2 (abfss)
Configuration menu - View commit details
-
Copy full SHA for f59a347 - Browse repository at this point
Copy the full SHA f59a347View commit details -
Refactoring SelectReturnEntity to reflect we only accept entities wit…
…h matching fully qualified names By pulling out and putting first the QualifiedNames_Match_After_Normalizing, it's clear that all search result entities must have a matching FQN. This puts a stronger emphasis on the Build_Searchable_QualifiedName and its ability to standardize the name.
Configuration menu - View commit details
-
Copy full SHA for dbccd6a - Browse repository at this point
Copy the full SHA dbccd6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2636d40 - Browse repository at this point
Copy the full SHA 2636d40View commit details -
Configuration menu - View commit details
-
Copy full SHA for 45f65db - Browse repository at this point
Copy the full SHA 45f65dbView commit details -
Refactoring validentity to validEntitiesAfterFiltering to make it mor…
…e clear that filtering has occured
Configuration menu - View commit details
-
Copy full SHA for 79410ce - Browse repository at this point
Copy the full SHA 79410ceView commit details -
Removing unncessary column mapping comments and Validate_Resource_Set…
… method which is never referenced
Configuration menu - View commit details
-
Copy full SHA for 61ea232 - Browse repository at this point
Copy the full SHA 61ea232View commit details -
Refactoring PurviewIngestion to remove unused methods and refactor ou…
…r the Log method in favor of using the ILogger directly
Configuration menu - View commit details
-
Copy full SHA for d53f45f - Browse repository at this point
Copy the full SHA d53f45fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a947969 - Browse repository at this point
Copy the full SHA a947969View commit details -
Refactoring SendToPurview in PurviewIngestion for each loop's variabl…
…e from a simple entity to purviewEntityToBeUpdated to make it more clear what's going to happen to this entity and not confuse it with other references to the word 'entity'
Configuration menu - View commit details
-
Copy full SHA for f061826 - Browse repository at this point
Copy the full SHA f061826View commit details -
Refactoring PurviewIngestion naming conventions to clarify entities t…
…hat may be deleted and adding comments to walk through relationship attribute extraction and replacement
Configuration menu - View commit details
-
Copy full SHA for d3dd7af - Browse repository at this point
Copy the full SHA d3dd7afView commit details -
Adding a field for originalQualifiedName and removing unused methods …
…simpleEntity, AddToTable, FindQualifiedNameInPurview, CleanUnusedCustomEntities, and Name_To_Search
Configuration menu - View commit details
-
Copy full SHA for d732ef7 - Browse repository at this point
Copy the full SHA d732ef7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 48dc0c6 - Browse repository at this point
Copy the full SHA 48dc0c6View commit details -
ColParser should optionally take in a mapping of original dataset nam…
…es to fully qualified names discovered during search and PurviewIngestion needs to keep track of these mappings
Configuration menu - View commit details
-
Copy full SHA for 50b0bf5 - Browse repository at this point
Copy the full SHA 50b0bf5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 48af630 - Browse repository at this point
Copy the full SHA 48af630View commit details -
Configuration menu - View commit details
-
Copy full SHA for ada23c9 - Browse repository at this point
Copy the full SHA ada23c9View commit details -
Updating Limitations and Readme to better reflect current state and s…
…upport of column lineage
Configuration menu - View commit details
-
Copy full SHA for 0c68b4a - Browse repository at this point
Copy the full SHA 0c68b4aView commit details -
Handle Azure Data Factory Job Names (#137)
Truncate Azure Data Factory job name guid to prevent creating duplicate jobs / tasks only differntiated by a guid / pipeline id Job name should be ADF_factoryName_pipelineName Task name should be ADF_factoryName_pipelineName_activityName ADF Regex pattern should not ignore case
Configuration menu - View commit details
-
Copy full SHA for eddfa25 - Browse repository at this point
Copy the full SHA eddfa25View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3fd940d - Browse repository at this point
Copy the full SHA 3fd940dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f4b166b - Browse repository at this point
Copy the full SHA f4b166bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 811a7ae - Browse repository at this point
Copy the full SHA 811a7aeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 014183b - Browse repository at this point
Copy the full SHA 014183bView commit details -
Configuration menu - View commit details
-
Copy full SHA for fbabc92 - Browse repository at this point
Copy the full SHA fbabc92View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e65747 - Browse repository at this point
Copy the full SHA 5e65747View commit details -
Configuration menu - View commit details
-
Copy full SHA for 39dcd36 - Browse repository at this point
Copy the full SHA 39dcd36View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c58203 - Browse repository at this point
Copy the full SHA 2c58203View commit details -
Fix Library Definitions in Job Tasks to prevent deserialization error…
… from JSON to class models
Configuration menu - View commit details
-
Copy full SHA for b6b1e87 - Browse repository at this point
Copy the full SHA b6b1e87View commit details -
OlToPurviewMapping Quality of Dev Improvements
Python script to remove spaces and added as an artifact to Build and Release (Closes #183) Python script to update the arm template with the OlToPurviewMapping in a stringified json format (Closes #184) Updated the arm template with the standardized response from the new python script to avoid conflicts later on
Configuration menu - View commit details
-
Copy full SHA for bc40c97 - Browse repository at this point
Copy the full SHA bc40c97View commit details -
Configuration menu - View commit details
-
Copy full SHA for 46ae5b9 - Browse repository at this point
Copy the full SHA 46ae5b9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 05a54b2 - Browse repository at this point
Copy the full SHA 05a54b2View commit details -
Fix Mappings for Mount Points with Subdirectories
Mount points that have a source with a subdirectory were failing to include the subdirectories Introduced a new namespaceBodyPartsJoinedBySlashFrom mappings variable to capture any parts from a certain position and concatenate them with a forward slash.
Configuration menu - View commit details
-
Copy full SHA for 6fd492e - Browse repository at this point
Copy the full SHA 6fd492eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7502ed5 - Browse repository at this point
Copy the full SHA 7502ed5View commit details -
Configuration menu - View commit details
-
Copy full SHA for c0f65d1 - Browse repository at this point
Copy the full SHA c0f65d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for c9f0189 - Browse repository at this point
Copy the full SHA c9f0189View commit details -
Configuration menu - View commit details
-
Copy full SHA for 51678a0 - Browse repository at this point
Copy the full SHA 51678a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d8df7d - Browse repository at this point
Copy the full SHA 6d8df7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2bf960 - Browse repository at this point
Copy the full SHA c2bf960View commit details -
Rebased to include updates to newdeploymenttemp.json
Configuration menu - View commit details
-
Copy full SHA for c73398b - Browse repository at this point
Copy the full SHA c73398bView commit details -
Cosmos integration test added, and test-env README updated. TODO: Upd…
…ate LIMITATIONS, add integration test notebooks, and add integration test for additional inputs + cosmos
Configuration menu - View commit details
-
Copy full SHA for ba6c4aa - Browse repository at this point
Copy the full SHA ba6c4aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7f24fa2 - Browse repository at this point
Copy the full SHA 7f24fa2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9f3714c - Browse repository at this point
Copy the full SHA 9f3714cView commit details -
Configuration menu - View commit details
-
Copy full SHA for b67bbe1 - Browse repository at this point
Copy the full SHA b67bbe1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 50f3129 - Browse repository at this point
Copy the full SHA 50f3129View commit details -
Configuration menu - View commit details
-
Copy full SHA for 00b18d4 - Browse repository at this point
Copy the full SHA 00b18d4View commit details -
Updated UnitTestData, test CompleteNoOutputsInputsFullMessage expecte…
…d result to True, as now considering COMPLETE events with only outputs
Configuration menu - View commit details
-
Copy full SHA for 247e56a - Browse repository at this point
Copy the full SHA 247e56aView commit details -
Configuration menu - View commit details
-
Copy full SHA for c3bbfdd - Browse repository at this point
Copy the full SHA c3bbfddView commit details -
Configuration menu - View commit details
-
Copy full SHA for e008feb - Browse repository at this point
Copy the full SHA e008febView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2da877e - Browse repository at this point
Copy the full SHA 2da877eView commit details -
Configuration menu - View commit details
-
Copy full SHA for edd5657 - Browse repository at this point
Copy the full SHA edd5657View commit details