…resource sets

* Adding tests/environment folder to store datasets and bicep templates for test sources
* Added scripts to create databricks jobs and a notebook to mount storage on Databricks
* Making test environments more consistent across notebooks (secret scope, environment variables)
* Handle of tests were modified to correct mistakes not caught in source controlled versions
* Added documentation for testing environment including what secrets are used and what they look like
* Adding requirements.txt file for environment deployment
* Hive tests should run without additional intervention (i.e. use CREATE IF NOT EXISTS)
* Removing production env deployment
* Remove the wasbs with parameters test
* After updating all jobdefs to be ready for upload, the run-tests script needed to look at .name instead of .settings.name
  * Unfortunately, when calling the jobs API, it returns a .settings.name which must be used

…s deploy is going

and event hub authorization rules deploying at the same time.

Co-authored-by: hmoazam <[email protected]>

Bumps [cryptography](https://github.com/pyca/cryptography) from 38.0.4 to 39.0.1.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@38.0.4...39.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

…ITATIONS.md to indicate MySQL support (#149)

* Adding bicep file

* Added unit and integration tests for Postgres. Updated LIMITATIONS.md to describe support and limitations

* Implemented support for kusto by updating mappings.

* Implemented Kusto support and added unit and integration tests

* checking for resource set types as its own method
* checking for blob or data lake file path or filesystem types as its own method
* refactored checking for adf relationships for blob/data lake file path types to its own method
* Behavior Change for blob/data lake file path types that match: follow the insert into first position of valid entities rather than return a list with only the entity
* Renamed ResourceSet_QualifiedNames_Match to QualifiedNames_Match_After_Normalizing and added the trim of the trailing slash for comaprisons
    * Using QualifiedNames_Match_After_Normalizing when comparing as the dfs vs blob comparison was preventing folder paths from matching if they're using wasbs but scanned for azure data lake gen2 (abfss)

…h matching fully qualified names

By pulling out and putting first the QualifiedNames_Match_After_Normalizing, it's clear that all search result entities must have a matching FQN.

This puts a stronger emphasis on the Build_Searchable_QualifiedName and its ability to standardize the name.

…e clear that filtering has occured

… method which is never referenced

…r the Log method in favor of using the ILogger directly

…r verifying

…e from a simple entity to purviewEntityToBeUpdated to make it more clear what's going to happen to this entity and not confuse it with other references to the word 'entity'

…hat may be deleted and adding comments to walk through relationship attribute extraction and replacement

…simpleEntity, AddToTable, FindQualifiedNameInPurview, CleanUnusedCustomEntities, and Name_To_Search

…es to fully qualified names discovered during search and PurviewIngestion needs to keep track of these mappings

…d in PurviewOut

…s entities

…upport of column lineage

Truncate Azure Data Factory job name guid to prevent creating duplicate jobs / tasks only differntiated by a guid / pipeline id
Job name should be ADF_factoryName_pipelineName
Task name should be ADF_factoryName_pipelineName_activityName
ADF Regex pattern should not ignore case

Closes #156

… from JSON to class models

Python script to remove spaces and added as an artifact to Build and Release (Closes #183)
Python script to update the arm template with the OlToPurviewMapping in a stringified json format (Closes #184)

Updated the arm template with the standardized response from the new python script to avoid conflicts later on

Mount points that have a source with a subdirectory were failing to include the subdirectories

Introduced a new namespaceBodyPartsJoinedBySlashFrom mappings variable to capture any parts from
a certain position and concatenate them with a forward slash.

Rebased to include updates to newdeploymenttemp.json

…ate LIMITATIONS, add integration test notebooks, and add integration test for additional inputs + cosmos

…d result to True, as now considering COMPLETE events with only outputs

… code