Feature/support cosmos #145
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adding tests/environment folder to store datasets and bicep templates for test sources * Added scripts to create databricks jobs and a notebook to mount storage on Databricks * Making test environments more consistent across notebooks (secret scope, environment variables) * Handle of tests were modified to correct mistakes not caught in source controlled versions * Added documentation for testing environment including what secrets are used and what they look like * Adding requirements.txt file for environment deployment * Hive tests should run without additional intervention (i.e. use CREATE IF NOT EXISTS) * Removing production env deployment * Remove the wasbs with parameters test * After updating all jobdefs to be ready for upload, the run-tests script needed to look at .name instead of .settings.name * Unfortunately, when calling the jobs API, it returns a .settings.name which must be used
…s deploy is going and event hub authorization rules deploying at the same time. Co-authored-by: hmoazam <[email protected]>
hmoazam
force-pushed
the
feature/supportCosmos
branch
from
cd3bcdf
to
6cf20d5
Compare
Closed
Bumps [cryptography](https://github.com/pyca/cryptography) from 38.0.4 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@38.0.4...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>
hmoazam
force-pushed
the
feature/supportCosmos
branch
from
894914d
to
636b13e
Compare
…ITATIONS.md to indicate MySQL support (#149)
* Adding bicep file * Added unit and integration tests for Postgres. Updated LIMITATIONS.md to describe support and limitations
* Implemented support for kusto by updating mappings. * Implemented Kusto support and added unit and integration tests
* checking for resource set types as its own method
* checking for blob or data lake file path or filesystem types as its own method
* refactored checking for adf relationships for blob/data lake file path types to its own method
* Behavior Change for blob/data lake file path types that match: follow the insert into first position of valid entities rather than return a list with only the entity
* Renamed ResourceSet_QualifiedNames_Match to QualifiedNames_Match_After_Normalizing and added the trim of the trailing slash for comaprisons
* Using QualifiedNames_Match_After_Normalizing when comparing as the dfs vs blob comparison was preventing folder paths from matching if they're using wasbs but scanned for azure data lake gen2 (abfss)
…h matching fully qualified names By pulling out and putting first the QualifiedNames_Match_After_Normalizing, it's clear that all search result entities must have a matching FQN. This puts a stronger emphasis on the Build_Searchable_QualifiedName and its ability to standardize the name.
…e clear that filtering has occured
… method which is never referenced
…r the Log method in favor of using the ILogger directly
…e from a simple entity to purviewEntityToBeUpdated to make it more clear what's going to happen to this entity and not confuse it with other references to the word 'entity'
…hat may be deleted and adding comments to walk through relationship attribute extraction and replacement
…simpleEntity, AddToTable, FindQualifiedNameInPurview, CleanUnusedCustomEntities, and Name_To_Search
… from JSON to class models
Python script to remove spaces and added as an artifact to Build and Release (Closes #183) Python script to update the arm template with the OlToPurviewMapping in a stringified json format (Closes #184) Updated the arm template with the standardized response from the new python script to avoid conflicts later on
Mount points that have a source with a subdirectory were failing to include the subdirectories Introduced a new namespaceBodyPartsJoinedBySlashFrom mappings variable to capture any parts from a certain position and concatenate them with a forward slash.
Rebased to include updates to newdeploymenttemp.json
…ate LIMITATIONS, add integration test notebooks, and add integration test for additional inputs + cosmos
…d result to True, as now considering COMPLETE events with only outputs
hmoazam
force-pushed
the
feature/supportCosmos
branch
from
636b13e
to
edd5657
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updating implementation to support cosmos (data source v2) events.