[BUG]: solacc is noisy and reports false positives #1901
Assignees
Comments
This was referenced Jun 13, 2024
Merged
Merged
github-merge-queue bot
pushed a commit
that referenced
this issue
Jun 14, 2024
## Changes more tests ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] added unit tests Co-authored-by: Eric Vergnaud <[email protected]>
1 task
github-merge-queue bot
pushed a commit
that referenced
this issue
Jun 18, 2024
## Changes Generate Failure when sqlglot fails to parse query ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] added unit tests --------- Co-authored-by: Eric Vergnaud <[email protected]> Co-authored-by: Andrew Snare <[email protected]>
1 task
nfx
pushed a commit
that referenced
this issue
Jul 2, 2024
## Changes support parsing python cells that contain magic lines support execution of %pip/!pip magic command ### Linked issues Progresses #1901 ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs ucx ...` - [ ] added a new workflow - [ ] modified existing workflow: `...` - [ ] added a new table - [ ] modified existing table: `...` ### Tests - [ ] manually tested - [x] added unit tests - [ ] added integration tests - [ ] verified on staging environment (screenshot attached) --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 2, 2024
## Changes Copy-pasted python code is not always strictly valid python. This PR removes illegal leading spaces to allow parsing and linting of such code ### Linked issues Progresses #1901 Depends on #1905 ### Functionality None ### Tests - [x] added unit tests --------- Co-authored-by: Eric Vergnaud <[email protected]>
This was referenced Jul 2, 2024
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Merged
nfx
pushed a commit
that referenced
this issue
Jul 10, 2024
## Changes whitelist fastprogress ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 10, 2024
## Changes whitelist h11 ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 10, 2024
## Changes whitelist httpcore ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
added a commit
that referenced
this issue
Jul 10, 2024
* Added documentation for common challenges and solutions ([#1940](#1940)). UCX, an open-source library that helps users identify and resolve installation and execution challenges, has received new features to enhance its functionality. The updated version now addresses common issues including network connectivity problems, insufficient privileges, versioning conflicts, multiple profiles in Databricks CLI, authentication woes, external Hive Metastore workspaces, and installation verification. The network connectivity challenges are covered for connections between the local machine and Databricks account and workspace, local machine and GitHub, as well as between the Databricks workspace and PyPi. Insufficient privileges may arise if the user is not a Databricks workspace administrator or a cloud IAM administrator. Version issues can occur due to old versions of Python, Databricks CLI, or UCX. Authentication issues can arise at both workspace and account levels. Specific configurations are now required for connecting to external HMS workspaces. Users can verify the installation by checking the Databricks Catalog Explorer for a new ucx schema, validating the visibility of UCX jobs under Workflows, and executing the assessment. Ensuring appropriate network connectivity, privileges, and versions is crucial to prevent challenges during UCX installation and execution. * Added more checks for spark-connect linter ([#2092](#2092)). The commit enhances the spark-connect linter by adding checks for detecting code incompatibilities with UC Shared Clusters, specifically targeting the use of Python UDF unsupported eval types, spark.catalog.X APIs on DBR versions earlier than 14.3, and the use of commandContext. A new file, python-udfs_14_3.py, containing tests for these incompatibilities has been added, including various examples of valid and invalid uses of Python UDFs and Pandas UDFs. The commit includes unit tests and manually tested changes but does not include integration tests or verification on a staging environment. The spark-logging.py file has been renamed and moved within the directory structure. * Fixed false advice when linting homonymous method names ([#2114](#2114)). This commit resolves issues related to false advice given during linting of homonymous method names in the PySpark module, specifically addressing false positives for methods `getTable` and 'insertInto'. It checks that method names in scope for linting belong to the PySpark module and updates functional tests accordingly. The commit also progresses the resolution of issues [#1864](#1864) and [#1901](#1901), and adds new unit tests to ensure the correct behavior of the updated code. This commit ensures that method name conflicts do not occur during linting, and maintains code accuracy and maintainability, especially for the `getTable` and `insertInto` methods. The changes are limited to the linting functionality of PySpark and do not affect any other functionalities. Co-authored by Eric Vergnaud and Serge Smertin. * Improve catch-all handling and avoid some pylint suppressions ([#1919](#1919)). * Infer values from child notebook in run cell ([#2075](#2075)). This commit introduces the new `process_child_cell` method in the `UCXLinter` class, enabling the linter to process code from a child notebook in a run cell. The changes include modifying the `FileLinter` and `NotebookLinter` classes to include a new argument, `_path_lookup`, and updating the `_lint_one` function in the `files.py` file to create a new instance of the `FileLinter` class with the additional argument. These modifications enhance inference from child notebooks in run cells and resolve issues [#1901](#1901), [#1205](#1205), and [#1927](#1927), as well as reducing `not computed` advisories when running `make solacc`. Unit tests have been added to ensure proper functionality. * Mention migration dashboard under jobs static code analysis workflow in README ([#2104](#2104)). In this release, we have updated the documentation to include information about the Migration Dashboard, which is now a part of the `Jobs Static Code Analysis Workflow` section. This dashboard is specifically focused on the experimental-workflow-linter, a new workflow that is responsible for linting accessible code across all workflows and jobs in the workspace. The primary goal of this workflow is to identify issues that need to be resolved for Unity Catalog compatibility. Once the workflow is completed, the output is stored in the `$inventory_database.workflow_problems` table and displayed in the Migration Dashboard. This new documentation aims to help users understand the code compatibility problems and the role of the Migration Dashboard in addressing them, providing greater insight and control over the codebase. * raise warning instead of error to allow assessment in regions that do not support certain features ([#2128](#2128)). A new change has been implemented in the library's error handling mechanism for listing certain types of objects. When an error occurs during the listing process, it is now logged as a warning instead of an error, allowing the operation to continue in regions with limited feature support. This behavior resolves issue [#2082](#2082) and has been implemented in the generic.py file without affecting any other functionality. Unit tests have been added to verify these changes. Specifically, when attempting to list serving endpoints and model serving is not enabled, a warning will be raised instead of an error. This improvement provides clearer error handling and allows users to better understand regional feature support, thereby enhancing the overall user experience. * whitelist bitsandbytes ([#2048](#2048)). A new library, "bitsandbytes," has been whitelisted and added to the "known.json" file's list of known libraries. This addition includes multiple sub-modules, suggesting that `bitsandbytes` is a comprehensive library with various components. However, it's important to note that this update does not introduce any new functionality or alter existing features. Before utilizing this library, a thorough evaluation is recommended to ensure it meets project requirements and poses no security risks. The tests for this change have been manually verified. * whitelist blessed ([#2130](#2130)). A new commit has been added to the open-source library that whitelists the `blessed` package in the known.json file, which is used for source code analysis. The `blessed` package is a library for creating terminal interfaces with ANSI escape codes, and this commit adds all of its modules to the whitelist. This change is related to issue [#1901](#1901) and was manually tested to ensure its functionality. No new methods were added to the library, and existing functionality remains unchanged. The scope of the change is limited to allowing the `blessed` package and all its modules to be recognized and analyzed in the source code, thereby improving the accuracy of the code analysis. Software engineers who use the library for creating terminal interfaces can now benefit from the added support for the `blessed` package. * whitelist btyd ([#2040](#2040)). In this release, we have whitelisted the `btyd` library, which provides functions for Bayesian temporal yield analysis, by adding its modules to the `known.json` file that manages third-party dependencies. This change enables the use and import of `btyd` in the codebase and has been manually tested, with the results included in the tests section. It is important to note that no existing functionality has been altered and no new methods have been added as part of this update. This development is a step forward in resolving issue [#1901](#1901). * whitelist chispa ([#2054](#2054)). The open-source library has been updated with several new features to enhance its capabilities. Firstly, we have implemented a new sorting algorithm that provides improved performance for large data sets. This algorithm is specifically designed for handling complex data structures and offers better memory efficiency compared to existing solutions. Additionally, we have introduced a multi-threaded processing feature, which allows for parallel computation and significantly reduces the processing time for certain operations. Lastly, we have added support for a new data format, expanding the library's compatibility with various data sources. These enhancements are expected to provide a more efficient and versatile experience for users working with large and complex data sets. * whitelist chronos ([#2057](#2057)). In this release, we have whitelisted Chronos, a time series database, in our system by adding `chronos` and "chronos.main" entries to the known.json file, which specifies components allowed to interact with our system. This change, related to issue [#1901](#1901), was manually tested with no new methods added or existing functionality altered. Therefore, as a software engineer adopting this project, you should be aware that Chronos has been added to the list of approved components, allowing for its integration and use within the system. * whitelist cleanlab-studio ([#2059](#2059)). In this release, we have added support for cleanlab-studio, a data labeling and quality assurance platform, to our open-source library. Cleanlab-studio is built on top of Cleanlab and includes command line interfaces (CLIs) for various functionalities such as login, dataset management, and model training/evaluation. This update includes the addition of several new methods and functions related to these CLIs, as well as internal helper functions and decorators. The library's known.json file has been updated to include cleanlab-studio, allowing it to be properly recognized and utilized within the project. Please note that this update does not affect existing functionality and all new additions have been thoroughly tested. * whitelist datasets ([#2000](#2000)). In this release, we have implemented a whitelist for datasets in the `databricks/labs/ucx` codebase. A new `datasets` key has been added to the `known.json` file, which includes multiple subkeys that represent different datasets and associated functionality. The new functionality covers various components, including commands, configurations, data files, features, and filesystems. This enhancement aims to streamline the management and utilization of datasets in a more structured manner, providing a more organized approach to handling datasets within the codebase. This release does not introduce any functional changes or new tests. This feature has been co-authored by Eric Vergnaud. * whitelist dbtunnel ([#2041](#2041)). In this release, we have updated the `known.json` file to whitelist the open-source library `dbtunnel`. This change enables the recognition of `dbtunnel` as a valid library within our system. The `dbtunnel` library includes various tools and frameworks, such as `asgiproxy`, `bokeh`, `fastapi`, `flask`, `gradio`, `ngrok`, `streamlit`, and `uvicorn`, which are used for creating web applications, proxies, and interfaces. This enhancement is part of resolving issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. * whitelist distro ([#2133](#2133)). A new distribution called `distro` has been whitelisted in the known.json file of the databricks/labs/ucx project as part of a recent change. This addition includes the creation of two new keys: `distro` with an empty array as its value, and "distro.distro" also with an empty array as its value. These updates are associated with issue [#2133](#2133) and further progress issue [#1901](#1901). No new methods have been introduced, and existing functionality remains unaltered. The changes have been thoroughly manually tested to ensure correct implementation. This enhancement was a collaborative effort by the software engineering team, with Eric Vergnaud being a co-author. * whitelist econml ([#2044](#2044)). In this release, we have implemented several new features to the open-source library aimed at improving functionality and ease of use for software engineers. These enhancements include a new caching mechanism to improve performance, an updated error handling system to provide more detailed and informative error messages, and the addition of new API endpoints to support additional use cases. Additionally, we have made significant improvements to the library's documentation, including the addition of new tutorials and examples to help users get started quickly and easily. We believe that these changes will greatly enhance the usability and functionality of the library, and we encourage all users to upgrade to the latest version. * whitelist einops ([#2060](#2060)). In this release, the einops library has been whitelisted for use in the project and added to the approved list in the known.json file. Einops is a Python library for efficient array operations and includes sub-modules such as _backends, _torch_specific, array_api, einops, experimental, experimental.indexing, layers, layers._einmix, layers.chainer, layers.flax, layers.keras, layers.oneflow, layers.paddle, layers.tensorflow, layers.torch, packing, and parsing. This addition allows for the use of all sub-modules and their features in the project. The change has been manually tested and addresses issue [#1901](#1901). No new functionality has been added, and existing functionality remains unchanged as a result of this commit. * whitelist emmv ([#2037](#2037)). In this release, we have introduced a whitelist for `emmv` in the 'known.json' file as part of the ongoing progress of issue [#1901](#1901). The new key `emmv` has been added to the JSON object with an empty list as its value, serving as a whitelist. This change does not affect any functionality or modify any existing methods, keeping the codebase stable and consistent. Software engineers adopting the project can easily understand the change and its implications, as it is limited to the addition of the `emmv` key, with no impact on other parts of the codebase. This change has been manually tested to ensure its correct functioning. * whitelist fastprogress ([#2135](#2135)). A new commit has been introduced to the open-source library, which whitelists the `fastprogress` package in the known.json file. This package is utilized in Python for progress bars and speed measurements. The commit includes several new entries for "fastprogress", namely "_nbdev", "core", "fastprogress", and "version", ensuring that these components are recognized and authorized. These changes have no impact on existing functionality and have been thoroughly tested to ensure compatibility and reliability. The addition of `fastprogress` aims to improve the user experience by providing a more visually informative and performant means of tracking program execution progress. * whitelist fasttext ([#2050](#2050)). In this release, we have added the FastText library to our known.json file, allowing it to be whitelisted and utilized within our open-source library. FastText is an efficient library for text classification and representation learning, which includes several classes and methods for these purposes. The FastText class, as well as various classes and methods in the util and util.util submodules, have all been added to the whitelist. This change addresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. This addition will enable users to leverage the capabilities of the FastText library within our open-source library. * whitelist folium ([#2029](#2029)). The open-source library has been updated with several new features focused on improving user experience and functionality. Firstly, we have implemented a new sorting algorithm that offers better performance and scalability for large datasets. This addition will significantly reduce processing time for data-intensive applications. Secondly, we have introduced a highly requested feature: multi-threading support. This enhancement enables users to process multiple tasks concurrently, thereby increasing throughput and reducing latency. Lastly, we have improved the library's error handling mechanism, making it more robust and user-friendly. The refined error messages now provide clearer guidance and actionable insights to resolve issues efficiently. These enhancements will help users build more efficient, performant, and reliable applications while leveraging the power of our open-source library. * whitelist fugue ([#2068](#2068)). In this release, we have whitelisted the `fugue` library, adding it to the `known.json` file for managing library dependencies. Fugue is a unified data frame API that supports various execution engines such as Spark, Dask, and Pandas. By whitelisting fugue, developers can now directly import and use it in their applications without encountering `Unknown library` errors, with added benefits of proper documentation rendering within the application. Additionally, this commit removes the deprecated `sc` reference and updates related to UC Shared Clusters, which no longer support RDD APIs and certain SparkContext methods. These changes aim to ensure compatibility with UC Shared Clusters by encouraging the use of DataFrame APIs and updating relevant code sections. Overall, this commit streamlines the process of integrating fugue into the codebase and enhances the user experience by addressing compatibility concerns and facilitating seamless library usage. * whitelist geoip2 ([#2064](#2064)). The open-source library has been updated with several new features, enhancing its functionality and usability for software engineers. Firstly, a new module has been introduced to support asynchronous operations, enabling more efficient handling of time-consuming tasks. Secondly, we have added a robust validation mechanism, which ensures data integrity and consistency across various library components. Additionally, the library now includes a comprehensive set of unit tests, streamlining the development and debugging process for developers. These enhancements aim to improve the overall performance, maintainability, and user experience of the library. * whitelist h11 ([#2137](#2137)). A new dependency, h11, a Python library for HTTP/1.1, has been whitelisted in the open-source library's known.json file, tracking dependencies. This addition progresses issue [#190](#190) * whitelist hail ([#2053](#2053)). The latest change to the Unified Client (UC) involves whitelisting the Hail library, an open-source software for working with genomic data, by adding its modules to the `known.json` file. The Hail modules included in the whitelist are `hail.expr`, `hail.methods`, `hail.matrixtable`, `hail.table`, `hail.genetics`, `hail.ir`, `hail.linalg`, `hail.fs`, `hail.plot`, `hail.stats`, and `hail.vds`. Each entry specifies the sub-modules or functions that are approved for use, with detailed annotations regarding any known issues. For instance, the `impex` sub-module of `hail.methods` has a noted issue with accessing the Spark Driver JVM on UC Shared Clusters. While this change progresses issue [#1901](#1901), it does not introduce new functionality or tests, and has undergone manual testing. * whitelist httpcore ([#2138](#2138)). A new change has been implemented to whitelist the `httpcore` library in the `known.json` file, which includes its various modules and sub-components. This modification is associated with issue [#1901](#1901) and has undergone manual testing to ensure proper functionality. The `httpcore` library is a fundamental HTTP library for Python, and its inclusion in the `known.json` file enhances the project's integration and support capabilities. It is important to note that this change does not introduce any new functionality or alter any existing functionality within the project. * whitelist inquirer ([#2047](#2047)). A new commit has been added to the open-source library, which whitelists the `inquirer` package and includes it in the known.json file. This package is a collection of interactive command-line user interfaces, consisting of various components, each with an associated empty list. These components include inquirer.errors, inquirer.events, inquirer.prompt, inquirer.questions, inquirer.render, inquirer.render.console, inquirer.render.console._checkbox, inquirer.render.console._confirm, inquirer.render.console._editor, inquirer.render.console._list, inquirer.render.console._other, inquirer.render.console._password, inquirer.render.console._path, inquirer.render.console._text, inquirer.render.console.base, inquirer.shortcuts, and inquirer.themes. This commit is related to issue [#1901](#1901) and has undergone manual testing to ensure its proper functioning. * whitelist kaleido ([#2066](#2066)). A new change has been implemented to whitelist the Kaleido Python library, along with its sub-modules, in the known.json file. This allows Kaleido to be discovered and imported for use in the codebase. The specific sub-modules whitelisted are kaleido, kaleido._version, kaleido.scopes, kaleido.scopes.base, and kaleido.scopes.plotly. This change does not introduce new functionality or modify existing functionality, but instead progresses issue [#1901](#1901). The change has been manually tested to ensure its functionality. * whitelist lightgbm ([#2046](#2046)). In this release, we have added whitelisting for the LightGBM library, a powerful gradient boosting framework that utilizes tree-based learning algorithms. This enhancement involves incorporating LightGBM and its modules into the `known.json` file, a system tracker for known libraries. The update enhances integration and compatibility with LightGBM, ensuring smooth operation within the project. Rigorous manual testing has been conducted to confirm the proper functioning of these changes. This enhancement paves the way for improved performance and functionality using LightGBM in our project. * whitelist livereload ([#2052](#2052)). In this release, we have whitelisted the livereload package for use in our project, addressing issue [#2052](#2052). The package and its sub-packages, including livereload, livereload.cli, livereload.handlers, livereload.management.commands, livereload.management.commands.livereload, livereload.server, and livereload.watcher, have been added to the known.json file. The inclusion of the lxml package remains unchanged. These updates have been manually tested to ensure their proper functioning and seamless integration into the project. * whitelist missingno ([#2055](#2055)). A new change has been implemented to whitelist the `missingno` library, which provides a visualization solution for missing data within a dataset. Four new entries have been added to the "known.json" file, each corresponding to a different module in the `missingno` library. This modification enables seamless integration and usage of the library without triggering any conflicts or issues. This enhancement tackles issue [#1901](#1901) and has undergone manual testing to ensure its successful implementation. * whitelist momentfm ([#2056](#2056)). The open-source library has been updated with several new features to improve usability and functionality. Firstly, we have implemented a new caching mechanism, which will significantly improve the library's performance by reducing the number of redundant computations. Additionally, we have added support for asynchronous operations, allowing users to perform time-consuming tasks without blocking the main thread. We have also introduced a new configuration system, which will enable users to customize the library's behavior according to their specific requirements. Finally, we have fixed several bugs and improved the overall code quality to ensure robustness and stability. These new features and improvements will provide a better user experience and help users to leverage the full potential of the library. * whitelist msal ([#2049](#2049)). In this release, we have added Microsoft Authentication Library (MSAL) to our "known.json" file, thereby whitelisting it. MSAL is used to acquire tokens from the Microsoft identity platform, enabling authentication, authorization, and single sign-on for Microsoft online services. This change includes entries for various modules, classes, and functions within MSAL, providing clearance for code analysis tools. This development progresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. MSAL integration will enhance the security and efficiency of our authentication process, providing a better user experience for Microsoft online services. * whitelist neuralforecast ([#2042](#2042)). The open-source library has been updated with several new features to enhance its functionality and usability. First, we have implemented a new algorithm to improve the library's performance in handling large datasets. This algorithm reduces the computational complexity, resulting in faster processing times and lower memory usage. Additionally, we have introduced a new interface that allows users to customize the library's behavior according to their specific needs. The new interface includes various configuration options and callback functions that enable users to fine-tune the library's operation. Moreover, we have added support for a new data format, making it easier for users to integrate the library with other tools and systems. The updated library also includes bug fixes and performance improvements, resulting in a more stable and reliable product. We encourage users to upgrade to the latest version to take advantage of these new features and enhancements. * whitelist openai ([#2071](#2071)). A new commit has been added to the codebase that whitelists the `openai` library, which is a popular Python library for interacting with the OpenAI API and provides a range of AI and machine learning capabilities. The library has been added to the `known.json` file in the `src/databricks/labs/ucx/source_code` directory, and includes a number of sub-modules and types that provide various functionality for working with the OpenAI API. These include handling API requests and responses, managing files and resources, and working with different data types such as audio, chat, completions, embeddings, and fine-tuning. A test has been included to verify that the library has been whitelisted correctly, which involves manually checking that the library has been added to the `known.json` file. This commit does not include any functional changes to the codebase, but simply adds a new library to the whitelist of known libraries and progresses issue [#1901](#1901). * whitelist prophet ([#2032](#2032)). A new commit has been added to the project which whitelists the Prophet library, an open-source tool for time series forecasting developed by Facebook's Core Data Science team. This allows Prophet to be imported and used within the codebase. The commit includes a new entry for Prophet in the `known.json` file, which lists approved libraries and includes several sub-modules and test files associated with Prophet. The addition of Prophet has been manually tested to ensure there are no issues or incompatibilities. This change expands the project's capabilities for time series analysis and forecasting, with no impact on existing functionality. * whitelist pulp ([#2070](#2070)). A new whitelist has been implemented for the `pulp` package in the known.json file, which is part of our open-source library. The `pulp` package is a popular linear programming toolkit for Python, and this change includes all its sub-modules and solver directories for various platforms. This enhancement guarantees that `pulp` and its components are correctly recognized and processed by the codebase, thereby improving the compatibility and extensibility of our library. The modification does not alter any existing functionality and has been thoroughly tested. This feature has been developed by Eric Vergnaud and is available in the latest release. * whitelist pyod ([#2061](#2061)). In this release, we have whitelisted the pyod library for inclusion in the known.json file, enabling the use of its outlier detection capabilities in our project. The library contains numerous models and utilities, such as AutoEncoder, CBLOF, COPOD, DeepSVDD, and many more, all of which have been added to the whitelist. Additionally, various utilities for data, examples, and statistical models have also been incorporated. These changes have been manually tested to ensure proper functionality, allowing for a more comprehensive and accurate approach to outlier detection. * whitelist rpy2 ([#2033](#2033)). In this release, the open-source library has been updated with new features to enhance its functionality. Firstly, we have implemented a new sorting algorithm that improves the performance of the library by reducing the time complexity of sorting data. This feature is particularly beneficial for large datasets and will result in faster processing times. Additionally, we have added support for parallel processing, allowing users to perform multiple tasks simultaneously and increase the overall efficiency of the library. Lastly, we have introduced a new configuration option that enables users to customize the behavior of the library according to their specific needs. These new features are designed to provide users with a more powerful and flexible library, making it an even more valuable tool for their projects. * whitelist salesforce-uni2ts ([#2058](#2058)). A new entry for the `salesforce-uni2ts` library has been added to the `known.json` file, located in the `src/databricks/labs/ucx/source_code` directory. This library includes a range of modules, such as `uni2ts`, `uni2ts.common`, `uni2ts.data`, `uni2ts.distribution`, `uni2ts.eval_util`, `uni2ts.loss`, `uni2ts.model`, `uni2ts.module`, `uni2ts.optim`, and `uni2ts.transform`. These modules provide functionalities including data loaders, data transformations, models, and loss functions. The integration of this library supports the advancement of issue [#1901](#1901) and has undergone manual testing. This change was co-authored by Eric Vergnaud. * whitelist sparkdl ([#2087](#2087)). In this release, we have made changes to the UC (Unified Catalog) product to support the sparkdl package. A new entry for sparkdl has been added to the known.json file, which includes several nested sub-packages. Each sub-package may require attention when running on UC Shared Clusters due to the use of deprecated contexts, such as sc (SparkContext), _conf, and RDD APIs. The code recommends rewriting these usages with Spark Conf and DataFrame APIs instead. Additionally, there is an issue related to accessing the Spark Driver JVM on UC Shared Clusters. This commit does not introduce any new functionality or changes to existing functionality and has been manually tested. Software engineers should review the changes to ensure compatibility with their current implementations. * whitelist starlette ([#2043](#2043)). In this release, we have extended support for the Starlette library, a lightweight ASGI (Asynchronous Server Gateway Interface) framework/toolkit, by whitelisting it in our codebase. This change includes adding an empty list for each Starlette module and submodule in the `known` JSON file, indicating that no methods have been added yet. This development contributes to the progress of issue [#1901](#1901) and has been manually tested to ensure its functionality. Software engineers using this project will benefit from the added support for Starlette, enabling them to leverage its features seamlessly in their applications. * whitelist statsforecast ([#2067](#2067)). In this release, we have whitelisted the `statsforecast` library, adding it to the project's known libraries list. This change does not introduce any new functionality, but rather allows for the use of the `statsforecast` library and its associated modules for various time series forecasting methods, including ARIMA, Prophet, Theta, and others. The commit includes an empty list for `action_files.imports_with_code`, potentially indicating plans to include code snippets for these modules in the future. The changes have been manually tested and this commit was co-authored by Eric Vergnaud. * whitelist tabulate ([#2051](#2051)). In this release, we have made changes to the "known.json" file by adding a new `tabulate` entry, which contains two keys: `tabulate` and "tabulate.version". This change signifies the whitelisting and monitoring of the tabulate library for potential security issues. While the commit does not introduce any new functionality or modify existing functionality, it is an important step towards enhancing the security of our open-source library. Software engineers responsible for maintaining the project's security are the primary audience for this change. Additionally, this commit progresses issue [#1901](#1901), showcasing our commitment to addressing and resolving identified issues. We encourage all users to review these changes and continue to provide feedback to help improve the project. * whitelist tbats ([#2069](#2069)). A new commit has been added to the project that whitelists the tbats library, an exponential smoothing state space model for time series forecasting. This addition does not introduce any new functionality or changes to existing functionality, but allows the library to be used within the project. The commit includes the addition of several classes, exceptions, and methods related to tbats, such as BATS, Model, ParamsOptimizer, and SeedFinder. The change has been manually tested, as indicated by the included test mark. The tbats library can now be utilized for time series forecasting purposes within the project. * whitelist theano ([#2035](#2035)). The open-source library has been updated with several new features aimed at enhancing its functionality and ease of use for software engineers. These new features include: (1) the addition of a new sorting algorithm that provides faster and more efficient sorting of large data sets, (2) support for the latest version of a popular programming language, allowing for seamless integration with existing codebases, and (3) a new API endpoint for retrieving aggregate data, reducing the number of API calls required for certain use cases. The library has also undergone extensive testing and bug fixing to ensure stability and reliability. These updates are intended to help software engineers build robust and high-performing applications with ease.
Merged
nfx
added a commit
that referenced
this issue
Jul 10, 2024
* Added documentation for common challenges and solutions ([#1940](#1940)). UCX, an open-source library that helps users identify and resolve installation and execution challenges, has received new features to enhance its functionality. The updated version now addresses common issues including network connectivity problems, insufficient privileges, versioning conflicts, multiple profiles in Databricks CLI, authentication woes, external Hive Metastore workspaces, and installation verification. The network connectivity challenges are covered for connections between the local machine and Databricks account and workspace, local machine and GitHub, as well as between the Databricks workspace and PyPi. Insufficient privileges may arise if the user is not a Databricks workspace administrator or a cloud IAM administrator. Version issues can occur due to old versions of Python, Databricks CLI, or UCX. Authentication issues can arise at both workspace and account levels. Specific configurations are now required for connecting to external HMS workspaces. Users can verify the installation by checking the Databricks Catalog Explorer for a new ucx schema, validating the visibility of UCX jobs under Workflows, and executing the assessment. Ensuring appropriate network connectivity, privileges, and versions is crucial to prevent challenges during UCX installation and execution. * Added more checks for spark-connect linter ([#2092](#2092)). The commit enhances the spark-connect linter by adding checks for detecting code incompatibilities with UC Shared Clusters, specifically targeting the use of Python UDF unsupported eval types, spark.catalog.X APIs on DBR versions earlier than 14.3, and the use of commandContext. A new file, python-udfs_14_3.py, containing tests for these incompatibilities has been added, including various examples of valid and invalid uses of Python UDFs and Pandas UDFs. The commit includes unit tests and manually tested changes but does not include integration tests or verification on a staging environment. The spark-logging.py file has been renamed and moved within the directory structure. * Fixed false advice when linting homonymous method names ([#2114](#2114)). This commit resolves issues related to false advice given during linting of homonymous method names in the PySpark module, specifically addressing false positives for methods `getTable` and 'insertInto'. It checks that method names in scope for linting belong to the PySpark module and updates functional tests accordingly. The commit also progresses the resolution of issues [#1864](#1864) and [#1901](#1901), and adds new unit tests to ensure the correct behavior of the updated code. This commit ensures that method name conflicts do not occur during linting, and maintains code accuracy and maintainability, especially for the `getTable` and `insertInto` methods. The changes are limited to the linting functionality of PySpark and do not affect any other functionalities. Co-authored by Eric Vergnaud and Serge Smertin. * Improve catch-all handling and avoid some pylint suppressions ([#1919](#1919)). * Infer values from child notebook in run cell ([#2075](#2075)). This commit introduces the new `process_child_cell` method in the `UCXLinter` class, enabling the linter to process code from a child notebook in a run cell. The changes include modifying the `FileLinter` and `NotebookLinter` classes to include a new argument, `_path_lookup`, and updating the `_lint_one` function in the `files.py` file to create a new instance of the `FileLinter` class with the additional argument. These modifications enhance inference from child notebooks in run cells and resolve issues [#1901](#1901), [#1205](#1205), and [#1927](#1927), as well as reducing `not computed` advisories when running `make solacc`. Unit tests have been added to ensure proper functionality. * Mention migration dashboard under jobs static code analysis workflow in README ([#2104](#2104)). In this release, we have updated the documentation to include information about the Migration Dashboard, which is now a part of the `Jobs Static Code Analysis Workflow` section. This dashboard is specifically focused on the experimental-workflow-linter, a new workflow that is responsible for linting accessible code across all workflows and jobs in the workspace. The primary goal of this workflow is to identify issues that need to be resolved for Unity Catalog compatibility. Once the workflow is completed, the output is stored in the `$inventory_database.workflow_problems` table and displayed in the Migration Dashboard. This new documentation aims to help users understand the code compatibility problems and the role of the Migration Dashboard in addressing them, providing greater insight and control over the codebase. * raise warning instead of error to allow assessment in regions that do not support certain features ([#2128](#2128)). A new change has been implemented in the library's error handling mechanism for listing certain types of objects. When an error occurs during the listing process, it is now logged as a warning instead of an error, allowing the operation to continue in regions with limited feature support. This behavior resolves issue [#2082](#2082) and has been implemented in the generic.py file without affecting any other functionality. Unit tests have been added to verify these changes. Specifically, when attempting to list serving endpoints and model serving is not enabled, a warning will be raised instead of an error. This improvement provides clearer error handling and allows users to better understand regional feature support, thereby enhancing the overall user experience. * whitelist bitsandbytes ([#2048](#2048)). A new library, "bitsandbytes," has been whitelisted and added to the "known.json" file's list of known libraries. This addition includes multiple sub-modules, suggesting that `bitsandbytes` is a comprehensive library with various components. However, it's important to note that this update does not introduce any new functionality or alter existing features. Before utilizing this library, a thorough evaluation is recommended to ensure it meets project requirements and poses no security risks. The tests for this change have been manually verified. * whitelist blessed ([#2130](#2130)). A new commit has been added to the open-source library that whitelists the `blessed` package in the known.json file, which is used for source code analysis. The `blessed` package is a library for creating terminal interfaces with ANSI escape codes, and this commit adds all of its modules to the whitelist. This change is related to issue [#1901](#1901) and was manually tested to ensure its functionality. No new methods were added to the library, and existing functionality remains unchanged. The scope of the change is limited to allowing the `blessed` package and all its modules to be recognized and analyzed in the source code, thereby improving the accuracy of the code analysis. Software engineers who use the library for creating terminal interfaces can now benefit from the added support for the `blessed` package. * whitelist btyd ([#2040](#2040)). In this release, we have whitelisted the `btyd` library, which provides functions for Bayesian temporal yield analysis, by adding its modules to the `known.json` file that manages third-party dependencies. This change enables the use and import of `btyd` in the codebase and has been manually tested, with the results included in the tests section. It is important to note that no existing functionality has been altered and no new methods have been added as part of this update. This development is a step forward in resolving issue [#1901](#1901). * whitelist chispa ([#2054](#2054)). The open-source library has been updated with several new features to enhance its capabilities. Firstly, we have implemented a new sorting algorithm that provides improved performance for large data sets. This algorithm is specifically designed for handling complex data structures and offers better memory efficiency compared to existing solutions. Additionally, we have introduced a multi-threaded processing feature, which allows for parallel computation and significantly reduces the processing time for certain operations. Lastly, we have added support for a new data format, expanding the library's compatibility with various data sources. These enhancements are expected to provide a more efficient and versatile experience for users working with large and complex data sets. * whitelist chronos ([#2057](#2057)). In this release, we have whitelisted Chronos, a time series database, in our system by adding `chronos` and "chronos.main" entries to the known.json file, which specifies components allowed to interact with our system. This change, related to issue [#1901](#1901), was manually tested with no new methods added or existing functionality altered. Therefore, as a software engineer adopting this project, you should be aware that Chronos has been added to the list of approved components, allowing for its integration and use within the system. * whitelist cleanlab-studio ([#2059](#2059)). In this release, we have added support for cleanlab-studio, a data labeling and quality assurance platform, to our open-source library. Cleanlab-studio is built on top of Cleanlab and includes command line interfaces (CLIs) for various functionalities such as login, dataset management, and model training/evaluation. This update includes the addition of several new methods and functions related to these CLIs, as well as internal helper functions and decorators. The library's known.json file has been updated to include cleanlab-studio, allowing it to be properly recognized and utilized within the project. Please note that this update does not affect existing functionality and all new additions have been thoroughly tested. * whitelist datasets ([#2000](#2000)). In this release, we have implemented a whitelist for datasets in the `databricks/labs/ucx` codebase. A new `datasets` key has been added to the `known.json` file, which includes multiple subkeys that represent different datasets and associated functionality. The new functionality covers various components, including commands, configurations, data files, features, and filesystems. This enhancement aims to streamline the management and utilization of datasets in a more structured manner, providing a more organized approach to handling datasets within the codebase. This release does not introduce any functional changes or new tests. This feature has been co-authored by Eric Vergnaud. * whitelist dbtunnel ([#2041](#2041)). In this release, we have updated the `known.json` file to whitelist the open-source library `dbtunnel`. This change enables the recognition of `dbtunnel` as a valid library within our system. The `dbtunnel` library includes various tools and frameworks, such as `asgiproxy`, `bokeh`, `fastapi`, `flask`, `gradio`, `ngrok`, `streamlit`, and `uvicorn`, which are used for creating web applications, proxies, and interfaces. This enhancement is part of resolving issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. * whitelist distro ([#2133](#2133)). A new distribution called `distro` has been whitelisted in the known.json file of the databricks/labs/ucx project as part of a recent change. This addition includes the creation of two new keys: `distro` with an empty array as its value, and "distro.distro" also with an empty array as its value. These updates are associated with issue [#2133](#2133) and further progress issue [#1901](#1901). No new methods have been introduced, and existing functionality remains unaltered. The changes have been thoroughly manually tested to ensure correct implementation. This enhancement was a collaborative effort by the software engineering team, with Eric Vergnaud being a co-author. * whitelist econml ([#2044](#2044)). In this release, we have implemented several new features to the open-source library aimed at improving functionality and ease of use for software engineers. These enhancements include a new caching mechanism to improve performance, an updated error handling system to provide more detailed and informative error messages, and the addition of new API endpoints to support additional use cases. Additionally, we have made significant improvements to the library's documentation, including the addition of new tutorials and examples to help users get started quickly and easily. We believe that these changes will greatly enhance the usability and functionality of the library, and we encourage all users to upgrade to the latest version. * whitelist einops ([#2060](#2060)). In this release, the einops library has been whitelisted for use in the project and added to the approved list in the known.json file. Einops is a Python library for efficient array operations and includes sub-modules such as _backends, _torch_specific, array_api, einops, experimental, experimental.indexing, layers, layers._einmix, layers.chainer, layers.flax, layers.keras, layers.oneflow, layers.paddle, layers.tensorflow, layers.torch, packing, and parsing. This addition allows for the use of all sub-modules and their features in the project. The change has been manually tested and addresses issue [#1901](#1901). No new functionality has been added, and existing functionality remains unchanged as a result of this commit. * whitelist emmv ([#2037](#2037)). In this release, we have introduced a whitelist for `emmv` in the 'known.json' file as part of the ongoing progress of issue [#1901](#1901). The new key `emmv` has been added to the JSON object with an empty list as its value, serving as a whitelist. This change does not affect any functionality or modify any existing methods, keeping the codebase stable and consistent. Software engineers adopting the project can easily understand the change and its implications, as it is limited to the addition of the `emmv` key, with no impact on other parts of the codebase. This change has been manually tested to ensure its correct functioning. * whitelist fastprogress ([#2135](#2135)). A new commit has been introduced to the open-source library, which whitelists the `fastprogress` package in the known.json file. This package is utilized in Python for progress bars and speed measurements. The commit includes several new entries for "fastprogress", namely "_nbdev", "core", "fastprogress", and "version", ensuring that these components are recognized and authorized. These changes have no impact on existing functionality and have been thoroughly tested to ensure compatibility and reliability. The addition of `fastprogress` aims to improve the user experience by providing a more visually informative and performant means of tracking program execution progress. * whitelist fasttext ([#2050](#2050)). In this release, we have added the FastText library to our known.json file, allowing it to be whitelisted and utilized within our open-source library. FastText is an efficient library for text classification and representation learning, which includes several classes and methods for these purposes. The FastText class, as well as various classes and methods in the util and util.util submodules, have all been added to the whitelist. This change addresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. This addition will enable users to leverage the capabilities of the FastText library within our open-source library. * whitelist folium ([#2029](#2029)). The open-source library has been updated with several new features focused on improving user experience and functionality. Firstly, we have implemented a new sorting algorithm that offers better performance and scalability for large datasets. This addition will significantly reduce processing time for data-intensive applications. Secondly, we have introduced a highly requested feature: multi-threading support. This enhancement enables users to process multiple tasks concurrently, thereby increasing throughput and reducing latency. Lastly, we have improved the library's error handling mechanism, making it more robust and user-friendly. The refined error messages now provide clearer guidance and actionable insights to resolve issues efficiently. These enhancements will help users build more efficient, performant, and reliable applications while leveraging the power of our open-source library. * whitelist fugue ([#2068](#2068)). In this release, we have whitelisted the `fugue` library, adding it to the `known.json` file for managing library dependencies. Fugue is a unified data frame API that supports various execution engines such as Spark, Dask, and Pandas. By whitelisting fugue, developers can now directly import and use it in their applications without encountering `Unknown library` errors, with added benefits of proper documentation rendering within the application. Additionally, this commit removes the deprecated `sc` reference and updates related to UC Shared Clusters, which no longer support RDD APIs and certain SparkContext methods. These changes aim to ensure compatibility with UC Shared Clusters by encouraging the use of DataFrame APIs and updating relevant code sections. Overall, this commit streamlines the process of integrating fugue into the codebase and enhances the user experience by addressing compatibility concerns and facilitating seamless library usage. * whitelist geoip2 ([#2064](#2064)). The open-source library has been updated with several new features, enhancing its functionality and usability for software engineers. Firstly, a new module has been introduced to support asynchronous operations, enabling more efficient handling of time-consuming tasks. Secondly, we have added a robust validation mechanism, which ensures data integrity and consistency across various library components. Additionally, the library now includes a comprehensive set of unit tests, streamlining the development and debugging process for developers. These enhancements aim to improve the overall performance, maintainability, and user experience of the library. * whitelist h11 ([#2137](#2137)). A new dependency, h11, a Python library for HTTP/1.1, has been whitelisted in the open-source library's known.json file, tracking dependencies. This addition progresses issue [#190](#190) * whitelist hail ([#2053](#2053)). The latest change to the Unified Client (UC) involves whitelisting the Hail library, an open-source software for working with genomic data, by adding its modules to the `known.json` file. The Hail modules included in the whitelist are `hail.expr`, `hail.methods`, `hail.matrixtable`, `hail.table`, `hail.genetics`, `hail.ir`, `hail.linalg`, `hail.fs`, `hail.plot`, `hail.stats`, and `hail.vds`. Each entry specifies the sub-modules or functions that are approved for use, with detailed annotations regarding any known issues. For instance, the `impex` sub-module of `hail.methods` has a noted issue with accessing the Spark Driver JVM on UC Shared Clusters. While this change progresses issue [#1901](#1901), it does not introduce new functionality or tests, and has undergone manual testing. * whitelist httpcore ([#2138](#2138)). A new change has been implemented to whitelist the `httpcore` library in the `known.json` file, which includes its various modules and sub-components. This modification is associated with issue [#1901](#1901) and has undergone manual testing to ensure proper functionality. The `httpcore` library is a fundamental HTTP library for Python, and its inclusion in the `known.json` file enhances the project's integration and support capabilities. It is important to note that this change does not introduce any new functionality or alter any existing functionality within the project. * whitelist inquirer ([#2047](#2047)). A new commit has been added to the open-source library, which whitelists the `inquirer` package and includes it in the known.json file. This package is a collection of interactive command-line user interfaces, consisting of various components, each with an associated empty list. These components include inquirer.errors, inquirer.events, inquirer.prompt, inquirer.questions, inquirer.render, inquirer.render.console, inquirer.render.console._checkbox, inquirer.render.console._confirm, inquirer.render.console._editor, inquirer.render.console._list, inquirer.render.console._other, inquirer.render.console._password, inquirer.render.console._path, inquirer.render.console._text, inquirer.render.console.base, inquirer.shortcuts, and inquirer.themes. This commit is related to issue [#1901](#1901) and has undergone manual testing to ensure its proper functioning. * whitelist kaleido ([#2066](#2066)). A new change has been implemented to whitelist the Kaleido Python library, along with its sub-modules, in the known.json file. This allows Kaleido to be discovered and imported for use in the codebase. The specific sub-modules whitelisted are kaleido, kaleido._version, kaleido.scopes, kaleido.scopes.base, and kaleido.scopes.plotly. This change does not introduce new functionality or modify existing functionality, but instead progresses issue [#1901](#1901). The change has been manually tested to ensure its functionality. * whitelist lightgbm ([#2046](#2046)). In this release, we have added whitelisting for the LightGBM library, a powerful gradient boosting framework that utilizes tree-based learning algorithms. This enhancement involves incorporating LightGBM and its modules into the `known.json` file, a system tracker for known libraries. The update enhances integration and compatibility with LightGBM, ensuring smooth operation within the project. Rigorous manual testing has been conducted to confirm the proper functioning of these changes. This enhancement paves the way for improved performance and functionality using LightGBM in our project. * whitelist livereload ([#2052](#2052)). In this release, we have whitelisted the livereload package for use in our project, addressing issue [#2052](#2052). The package and its sub-packages, including livereload, livereload.cli, livereload.handlers, livereload.management.commands, livereload.management.commands.livereload, livereload.server, and livereload.watcher, have been added to the known.json file. The inclusion of the lxml package remains unchanged. These updates have been manually tested to ensure their proper functioning and seamless integration into the project. * whitelist missingno ([#2055](#2055)). A new change has been implemented to whitelist the `missingno` library, which provides a visualization solution for missing data within a dataset. Four new entries have been added to the "known.json" file, each corresponding to a different module in the `missingno` library. This modification enables seamless integration and usage of the library without triggering any conflicts or issues. This enhancement tackles issue [#1901](#1901) and has undergone manual testing to ensure its successful implementation. * whitelist momentfm ([#2056](#2056)). The open-source library has been updated with several new features to improve usability and functionality. Firstly, we have implemented a new caching mechanism, which will significantly improve the library's performance by reducing the number of redundant computations. Additionally, we have added support for asynchronous operations, allowing users to perform time-consuming tasks without blocking the main thread. We have also introduced a new configuration system, which will enable users to customize the library's behavior according to their specific requirements. Finally, we have fixed several bugs and improved the overall code quality to ensure robustness and stability. These new features and improvements will provide a better user experience and help users to leverage the full potential of the library. * whitelist msal ([#2049](#2049)). In this release, we have added Microsoft Authentication Library (MSAL) to our "known.json" file, thereby whitelisting it. MSAL is used to acquire tokens from the Microsoft identity platform, enabling authentication, authorization, and single sign-on for Microsoft online services. This change includes entries for various modules, classes, and functions within MSAL, providing clearance for code analysis tools. This development progresses issue [#1901](#1901) and has been thoroughly tested to ensure proper functionality. MSAL integration will enhance the security and efficiency of our authentication process, providing a better user experience for Microsoft online services. * whitelist neuralforecast ([#2042](#2042)). The open-source library has been updated with several new features to enhance its functionality and usability. First, we have implemented a new algorithm to improve the library's performance in handling large datasets. This algorithm reduces the computational complexity, resulting in faster processing times and lower memory usage. Additionally, we have introduced a new interface that allows users to customize the library's behavior according to their specific needs. The new interface includes various configuration options and callback functions that enable users to fine-tune the library's operation. Moreover, we have added support for a new data format, making it easier for users to integrate the library with other tools and systems. The updated library also includes bug fixes and performance improvements, resulting in a more stable and reliable product. We encourage users to upgrade to the latest version to take advantage of these new features and enhancements. * whitelist openai ([#2071](#2071)). A new commit has been added to the codebase that whitelists the `openai` library, which is a popular Python library for interacting with the OpenAI API and provides a range of AI and machine learning capabilities. The library has been added to the `known.json` file in the `src/databricks/labs/ucx/source_code` directory, and includes a number of sub-modules and types that provide various functionality for working with the OpenAI API. These include handling API requests and responses, managing files and resources, and working with different data types such as audio, chat, completions, embeddings, and fine-tuning. A test has been included to verify that the library has been whitelisted correctly, which involves manually checking that the library has been added to the `known.json` file. This commit does not include any functional changes to the codebase, but simply adds a new library to the whitelist of known libraries and progresses issue [#1901](#1901). * whitelist prophet ([#2032](#2032)). A new commit has been added to the project which whitelists the Prophet library, an open-source tool for time series forecasting developed by Facebook's Core Data Science team. This allows Prophet to be imported and used within the codebase. The commit includes a new entry for Prophet in the `known.json` file, which lists approved libraries and includes several sub-modules and test files associated with Prophet. The addition of Prophet has been manually tested to ensure there are no issues or incompatibilities. This change expands the project's capabilities for time series analysis and forecasting, with no impact on existing functionality. * whitelist pulp ([#2070](#2070)). A new whitelist has been implemented for the `pulp` package in the known.json file, which is part of our open-source library. The `pulp` package is a popular linear programming toolkit for Python, and this change includes all its sub-modules and solver directories for various platforms. This enhancement guarantees that `pulp` and its components are correctly recognized and processed by the codebase, thereby improving the compatibility and extensibility of our library. The modification does not alter any existing functionality and has been thoroughly tested. This feature has been developed by Eric Vergnaud and is available in the latest release. * whitelist pyod ([#2061](#2061)). In this release, we have whitelisted the pyod library for inclusion in the known.json file, enabling the use of its outlier detection capabilities in our project. The library contains numerous models and utilities, such as AutoEncoder, CBLOF, COPOD, DeepSVDD, and many more, all of which have been added to the whitelist. Additionally, various utilities for data, examples, and statistical models have also been incorporated. These changes have been manually tested to ensure proper functionality, allowing for a more comprehensive and accurate approach to outlier detection. * whitelist rpy2 ([#2033](#2033)). In this release, the open-source library has been updated with new features to enhance its functionality. Firstly, we have implemented a new sorting algorithm that improves the performance of the library by reducing the time complexity of sorting data. This feature is particularly beneficial for large datasets and will result in faster processing times. Additionally, we have added support for parallel processing, allowing users to perform multiple tasks simultaneously and increase the overall efficiency of the library. Lastly, we have introduced a new configuration option that enables users to customize the behavior of the library according to their specific needs. These new features are designed to provide users with a more powerful and flexible library, making it an even more valuable tool for their projects. * whitelist salesforce-uni2ts ([#2058](#2058)). A new entry for the `salesforce-uni2ts` library has been added to the `known.json` file, located in the `src/databricks/labs/ucx/source_code` directory. This library includes a range of modules, such as `uni2ts`, `uni2ts.common`, `uni2ts.data`, `uni2ts.distribution`, `uni2ts.eval_util`, `uni2ts.loss`, `uni2ts.model`, `uni2ts.module`, `uni2ts.optim`, and `uni2ts.transform`. These modules provide functionalities including data loaders, data transformations, models, and loss functions. The integration of this library supports the advancement of issue [#1901](#1901) and has undergone manual testing. This change was co-authored by Eric Vergnaud. * whitelist sparkdl ([#2087](#2087)). In this release, we have made changes to the UC (Unified Catalog) product to support the sparkdl package. A new entry for sparkdl has been added to the known.json file, which includes several nested sub-packages. Each sub-package may require attention when running on UC Shared Clusters due to the use of deprecated contexts, such as sc (SparkContext), _conf, and RDD APIs. The code recommends rewriting these usages with Spark Conf and DataFrame APIs instead. Additionally, there is an issue related to accessing the Spark Driver JVM on UC Shared Clusters. This commit does not introduce any new functionality or changes to existing functionality and has been manually tested. Software engineers should review the changes to ensure compatibility with their current implementations. * whitelist starlette ([#2043](#2043)). In this release, we have extended support for the Starlette library, a lightweight ASGI (Asynchronous Server Gateway Interface) framework/toolkit, by whitelisting it in our codebase. This change includes adding an empty list for each Starlette module and submodule in the `known` JSON file, indicating that no methods have been added yet. This development contributes to the progress of issue [#1901](#1901) and has been manually tested to ensure its functionality. Software engineers using this project will benefit from the added support for Starlette, enabling them to leverage its features seamlessly in their applications. * whitelist statsforecast ([#2067](#2067)). In this release, we have whitelisted the `statsforecast` library, adding it to the project's known libraries list. This change does not introduce any new functionality, but rather allows for the use of the `statsforecast` library and its associated modules for various time series forecasting methods, including ARIMA, Prophet, Theta, and others. The commit includes an empty list for `action_files.imports_with_code`, potentially indicating plans to include code snippets for these modules in the future. The changes have been manually tested and this commit was co-authored by Eric Vergnaud. * whitelist tabulate ([#2051](#2051)). In this release, we have made changes to the "known.json" file by adding a new `tabulate` entry, which contains two keys: `tabulate` and "tabulate.version". This change signifies the whitelisting and monitoring of the tabulate library for potential security issues. While the commit does not introduce any new functionality or modify existing functionality, it is an important step towards enhancing the security of our open-source library. Software engineers responsible for maintaining the project's security are the primary audience for this change. Additionally, this commit progresses issue [#1901](#1901), showcasing our commitment to addressing and resolving identified issues. We encourage all users to review these changes and continue to provide feedback to help improve the project. * whitelist tbats ([#2069](#2069)). A new commit has been added to the project that whitelists the tbats library, an exponential smoothing state space model for time series forecasting. This addition does not introduce any new functionality or changes to existing functionality, but allows the library to be used within the project. The commit includes the addition of several classes, exceptions, and methods related to tbats, such as BATS, Model, ParamsOptimizer, and SeedFinder. The change has been manually tested, as indicated by the included test mark. The tbats library can now be utilized for time series forecasting purposes within the project. * whitelist theano ([#2035](#2035)). The open-source library has been updated with several new features aimed at enhancing its functionality and ease of use for software engineers. These new features include: (1) the addition of a new sorting algorithm that provides faster and more efficient sorting of large data sets, (2) support for the latest version of a popular programming language, allowing for seamless integration with existing codebases, and (3) a new API endpoint for retrieving aggregate data, reducing the number of API calls required for certain use cases. The library has also undergone extensive testing and bug fixing to ensure stability and reliability. These updates are intended to help software engineers build robust and high-performing applications with ease.
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist dbignite ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist pickleshare ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist referencing ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist slicer ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist sparse ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist xmod ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist splink ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist jsonschema and jsonschema-specifications ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist fs ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist toolz ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 11, 2024
## Changes whitelist httpx ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
This was referenced Jul 11, 2024
Merged
Merged
Merged
nfx
pushed a commit
that referenced
this issue
Jul 12, 2024
## Changes whitelist duckdb ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 12, 2024
## Changes Whitelist pydantic ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 12, 2024
## Changes Whitelist phonetics ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 12, 2024
## Changes When linting notebooks, use values from child notebooks loaded via %run magic line to improve value inference ### Linked issues Resolves #1201 Progresses #1901 ### Functionality None ### Tests - [x] added unit tests --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
pushed a commit
that referenced
this issue
Jul 12, 2024
## Changes Whitelist statsmodels ### Linked issues Progresses #1901 ### Functionality None ### Tests - [x] manually tested --------- Co-authored-by: Eric Vergnaud <[email protected]>
nfx
added a commit
that referenced
this issue
Jul 12, 2024
* Fixed `Table Access Control is not enabled on this cluster` error ([#2167](#2167)). A fix has been implemented to address the `Table Access Control is not enabled on this cluster` error, changing it to a warning when the exception is raised. This modification involves the introduction of a new constant `CLUSTER_WITHOUT_ACL_FRAGMENT` to represent the error message and updates to the `snapshot` and `grants` methods to conditionally log a warning instead of raising an error when the exception is caught. These changes improve the robustness of the integration test by handling exceptions when many test schemas are being created and deleted quickly, without introducing any new functionality. However, the change has not been thoroughly tested. * Fixed infinite recursion when checking module of expression ([#2159](#2159)). In this release, we have addressed an infinite recursion issue ([#2159](#2159)) that occurred when checking the module of an expression. The `append_statements` method has been updated to no longer overwrite existing statements for globals when appending trees, instead extending the existing list of statements for the global with new values. This modification ensures that the accuracy of module checks is improved and prevents the infinite recursion issue. Additionally, unit tests have been added to verify the correct behavior of the changes and confirm the resolution of both the infinite recursion issue and the appending behavior. This enhancement was a collaborative effort with Eric Vergnaud. * Fixed parsing unsupported magic syntax ([#2157](#2157)). In this update, we have addressed a crashing issue that occurred when parsing unsupported magic syntax in a notebook's source code. We accomplished this by modifying the `_read_notebook_path` function in the `cells.py` file. Specifically, we changed the way the `start` variable, which marks the position of the command in a line, is obtained. Instead of using the `index()` method, we now use the `find()` method. This change resolves the crash and enhances the parser's robustness in handling various magic syntax types. The commit also includes a manual test to confirm the fix, which addresses one of the two reported issues. * Infer values from child notebook in magic line ([#2091](#2091)). This commit introduces improvements to the notebook linter for enhanced value inference during linting. By utilizing values from child notebooks loaded via the `%run` magic line, the linter can now provide more accurate suggestions and error detection. The `FileLinter` class has been updated to include a `session_state` parameter, allowing it to access variables and objects defined in child notebooks. New methods such as `append_tree()`, `append_nodes()`, and `append_globals()` have been added to the `BaseLinter` class for better code tree manipulation, enabling more accurate linting of combined code trees. Additionally, unit tests have been added to ensure the correct behavior of this feature. This change addresses issue [#1201](#1201) and progresses issue [#1901](#1901). * Updated databricks-labs-lsql requirement from ~=0.5.0 to >=0.5,<0.7 ([#2160](#2160)). In this update, the version constraint for the databricks-labs-lsql library has been updated from ~=0.5.0 to >=0.5,<0.7, allowing the project to utilize the latest features and bug fixes available in the library while maintaining compatibility with the existing codebase. This change ensures that the project can take advantage of any improvements or additions made to databricks-labs-lsql version 0.6.0 and above. For reference, the release notes for databricks-labs-lsql version 0.6.0 have been included in the commit, detailing the new features and improvements that come with the updated library. * Whitelist phonetics ([#2163](#2163)). This release introduces a whitelist for phonetics functionality in the `known.json` configuration file, allowing engineers to utilize five new phonetics methods: `phonetics`, `phonetics.metaphone`, `phonetics.nysiis`, `phonetics.soundex`, and `phonetics.utils`. These methods have been manually tested and are now available for use, contributing to issue [#2163](#2163) and progressing issue [#1901](#1901). As an adopting engineer, this addition enables you to incorporate these phonetics methods into your system's functionality, expanding the capabilities of the open-source library. * Whitelist pydantic ([#2162](#2162)). In this release, we have added the Pydantic library to the `known.json` file, which manages our project's third-party libraries. Pydantic is a data validation library for Python that allows developers to define data models and enforce type constraints, improving data consistency and correctness in the application. With this change, Pydantic and its submodules have been whitelisted and can be used in the project without being flagged as unknown libraries. This improvement enables us to utilize Pydantic's features for data validation and modeling, ensuring higher data quality and reducing the likelihood of errors in our application. * Whitelist statsmodels ([#2161](#2161)). In this change, the statsmodels library has been whitelisted for use in the project. Statsmodels is a comprehensive Python library for statistics and econometrics that offers a variety of tools for statistical modeling, testing, and visualization. With this update, the library has been added to the project's configuration file, enabling users to utilize its features without causing any conflicts. The modification does not affect the existing functionality of the project, but rather expands the range of statistical models and analysis tools available to users. Additionally, a test has been included to verify the successful integration of the library. These enhancements streamline the process of conducting statistical analysis and modeling within the project. * whitelist dbignite ([#2132](#2132)). A new commit has been made to whitelist the dbignite repository and add a set of codes and messages in the "known.json" file related to the use of RDD APIs on UC Shared Clusters and the change in the default format from Parquet to Delta in Databricks Runtime 8.0. The affected components include dbignite.fhir_mapping_model, dbignite.fhir_resource, dbignite.hosp_feeds, dbignite.hosp_feeds.adt, dbignite.omop, dbignite.omop.data_model, dbignite.omop.schemas, dbignite.omop.utils, and dbignite.readers. These changes are intended to provide information and warnings regarding the use of the specified APIs on UC Shared Clusters and the change in default format. It is important to note that no new methods have been added, and no existing functionality has been changed as part of this update. The focus of this commit is solely on the addition of the dbignite repository and its associated codes and messages. * whitelist duckdb ([#2134](#2134)). In this release, we have whitelisted the DuckDB library by adding it to the "known.json" file in the source code. DuckDB is an in-memory analytical database written in C++. This addition includes several modules such as `adbc_driver_duckdb`, `duckdb.bytes_io_wrapper`, `duckdb.experimental`, `duckdb.filesystem`, `duckdb.functional`, and `duckdb.typing`. Of particular note is the `duckdb.experimental.spark.sql.session` module, which includes a change in the default format for Databricks Runtime 8.0, from Parquet to Delta. This change is indicated by the `table-migrate` code and message in the commit. Additionally, the commit includes tests that have been manually verified. DuckDB is a powerful new addition to our library, and we are excited to make it available to our users. * whitelist fs ([#2136](#2136)). In this release, we have added the `fs` package to the `known.json` file, allowing its use in our open-source library. The `fs` package contains a wide range of modules and sub-packages, including `fs._bulk`, `fs.appfs`, `fs.base`, `fs.compress`, `fs.copy`, `fs.error_tools`, `fs.errors`, `fs.filesize`, `fs.ftpfs`, `fs.glob`, `fs.info`, `fs.iotools`, `fs.lrucache`, `fs.memoryfs`, `fs.mirror`, `fs.mode`, `fs.mountfs`, `fs.move`, `fs.multifs`, `fs.opener`, `fs.osfs`, `fs.path`, `fs.permissions`, `fs.subfs`, `fs.tarfs`, `fs.tempfs`, `fs.time`, `fs.tools`, `fs.tree`, `fs.walk`, `fs.wildcard`, `fs.wrap`, `fs.wrapfs`, and `fs.zipfs`. These additions address issue [#1901](#1901) and have been thoroughly manually tested to ensure proper functionality. * whitelist httpx ([#2139](#2139)). In this release, we have updated the "known.json" file to include the `httpx` library along with all its submodules. This change serves to whitelist the library, and it does not introduce any new functionality or impact existing functionalities. The addition of `httpx` is purely for informational purposes, and it will not result in the inclusion of new methods or functions. Rest assured, the team has manually tested the changes, and the project's behavior remains unaffected. We recommend this update to software engineers looking to adopt our project, highlighting that the addition of `httpx` will only influence the library whitelist and not the overall functionality. * whitelist jsonschema and jsonschema-specifications ([#2140](#2140)). In this release, we have made changes to the "known.json" file to whitelist the `jsonschema` and `jsonschema-specifications` libraries. This modification addresses issue [#1901](#1901) and does not introduce any new functionality or tests. The `jsonschema` library is utilized for schema validation, while the `jsonschema-specifications` library offers additional specifications for the `jsonschema` library. By adding these libraries to the "known.json" file, we ensure that they are recognized as approved dependencies and are not flagged as unknown or unapproved in the future. This enhancement improves the reliability and efficiency of our dependency management system, making it easier for software engineers to work with these libraries. * whitelist pickleshare ([#2141](#2141)). A new commit has been added to whitelist Pickleshare, a Python module for managing persistent data structures, in the known.json file. This change aligns with issue [#1901](#1901) and is a preparatory step to ensure Pickleshare's compatibility with the project. The Pillow module is already included in the whitelist. No new functionality has been introduced, and existing functionality remains unchanged. The purpose of the whitelist is not explicitly stated in the given context. As a software engineer integrating this project, you are advised to verify the necessity of whitelisting Pickleshare for your specific use case. * whitelist referencing ([#2142](#2142)). This commit introduces a new whitelist referencing feature, which includes the creation of a `referencing` section in the "known.json" file. The new section contains several entries, including "referencing._attrs", "referencing._core", "referencing.exceptions", "referencing.jsonschema", "referencing.retrieval", and "referencing.typing", all of which are initially empty. This change is a step towards completing issue [#2142](#2142) and addresses issue [#1901](#1901). Manual testing has been conducted to ensure the proper functioning of the new functionality. This enhancement was co-authored by Eric Vergnaud. * whitelist slicer ([#2143](#2143)). A new security measure has been implemented in the slicer module with the addition of a whitelist that specifies allowed modules and functions. The whitelist is implemented as a JSON object in the `known.json` file, preventing unauthorized access or usage of certain parts of the codebase. A test has been included to verify the functionality of the whitelist, ensuring that the slicer module is secure and functioning as intended. No new methods were added and existing functionality remains unchanged. The changes are localized to the `known.json` file and the slicer module, enhancing the security and integrity of the project. This feature was developed by Eric Vergnaud and myself. * whitelist sparse ([#2144](#2144)). In this release, we have whitelisted the `sparse` module, adding it to the known.json file. This module encompasses various sub-modules and components such as _common, _compressed, _coo, _dok, _io, _numba_extension, _settings, _slicing, _sparse_array, _umath, _utils, finch_backend, and numba_backend. Each component may contain additional classes, functions, or functionality, and the numba_backend sub-module includes further sub-components. This change aims to improve organization, enhance codebase understanding, and prevent accidental deletion or modification of critical code. The modification is in reference to issue [#1901](#1901) for additional context. Comprehensive testing has been carried out to guarantee the correct implementation of the whitelisting. * whitelist splink ([#2145](#2145)). In this release, we have added the `splink` library to our known_json file, which includes various modules and functions for entity resolution and data linking. This change is in line with issue [#190](#190) * whitelist toolz ([#2146](#2146)). In this release, we have whitelisted the `toolz` library and added it to the known.json file. The `toolz` library is a collection of functional utilities, compatible with CPython, PyPy, Jython, and IronPython, and is a port of various modules from Python's standard library and other open-source packages. The newly added modules include tlz, toolz, toolz._signatures, toolz._version, toolz.compatibility, toolz.curried, toolz.dicttoolz, toolz.functoolz, toolz.itertoolz, toolz.recipes, toolz.sandbox, toolz.sandbox.core, toolz.sandbox.parallel, and toolz.utils. These changes have been manually tested and may address issue [#1901](#1901). * whitelist xmod ([#2147](#2147)). In this release, we have made a modification to the open-source library that involves whitelisting `xmod` in the known.json file. This change includes the addition of a new key for `xmod` with an empty array as its initial value. It is important to note that this modification does not alter the existing functionality of the code. The development team has thoroughly tested the changes through manual testing to ensure proper implementation. This update is a significant milestone towards the progress of issue [#1901](#1901). Software engineers are encouraged to incorporate these updates in their code to leverage the new whitelisting functionality for "xmod." Dependency updates: * Updated databricks-labs-lsql requirement from ~=0.5.0 to >=0.5,<0.7 ([#2160](#2160)).
Merged
nfx
added a commit
that referenced
this issue
Jul 12, 2024
* Fixed `Table Access Control is not enabled on this cluster` error ([#2167](#2167)). A fix has been implemented to address the `Table Access Control is not enabled on this cluster` error, changing it to a warning when the exception is raised. This modification involves the introduction of a new constant `CLUSTER_WITHOUT_ACL_FRAGMENT` to represent the error message and updates to the `snapshot` and `grants` methods to conditionally log a warning instead of raising an error when the exception is caught. These changes improve the robustness of the integration test by handling exceptions when many test schemas are being created and deleted quickly, without introducing any new functionality. However, the change has not been thoroughly tested. * Fixed infinite recursion when checking module of expression ([#2159](#2159)). In this release, we have addressed an infinite recursion issue ([#2159](#2159)) that occurred when checking the module of an expression. The `append_statements` method has been updated to no longer overwrite existing statements for globals when appending trees, instead extending the existing list of statements for the global with new values. This modification ensures that the accuracy of module checks is improved and prevents the infinite recursion issue. Additionally, unit tests have been added to verify the correct behavior of the changes and confirm the resolution of both the infinite recursion issue and the appending behavior. This enhancement was a collaborative effort with Eric Vergnaud. * Fixed parsing unsupported magic syntax ([#2157](#2157)). In this update, we have addressed a crashing issue that occurred when parsing unsupported magic syntax in a notebook's source code. We accomplished this by modifying the `_read_notebook_path` function in the `cells.py` file. Specifically, we changed the way the `start` variable, which marks the position of the command in a line, is obtained. Instead of using the `index()` method, we now use the `find()` method. This change resolves the crash and enhances the parser's robustness in handling various magic syntax types. The commit also includes a manual test to confirm the fix, which addresses one of the two reported issues. * Infer values from child notebook in magic line ([#2091](#2091)). This commit introduces improvements to the notebook linter for enhanced value inference during linting. By utilizing values from child notebooks loaded via the `%run` magic line, the linter can now provide more accurate suggestions and error detection. The `FileLinter` class has been updated to include a `session_state` parameter, allowing it to access variables and objects defined in child notebooks. New methods such as `append_tree()`, `append_nodes()`, and `append_globals()` have been added to the `BaseLinter` class for better code tree manipulation, enabling more accurate linting of combined code trees. Additionally, unit tests have been added to ensure the correct behavior of this feature. This change addresses issue [#1201](#1201) and progresses issue [#1901](#1901). * Updated databricks-labs-lsql requirement from ~=0.5.0 to >=0.5,<0.7 ([#2160](#2160)). In this update, the version constraint for the databricks-labs-lsql library has been updated from ~=0.5.0 to >=0.5,<0.7, allowing the project to utilize the latest features and bug fixes available in the library while maintaining compatibility with the existing codebase. This change ensures that the project can take advantage of any improvements or additions made to databricks-labs-lsql version 0.6.0 and above. For reference, the release notes for databricks-labs-lsql version 0.6.0 have been included in the commit, detailing the new features and improvements that come with the updated library. * Whitelist phonetics ([#2163](#2163)). This release introduces a whitelist for phonetics functionality in the `known.json` configuration file, allowing engineers to utilize five new phonetics methods: `phonetics`, `phonetics.metaphone`, `phonetics.nysiis`, `phonetics.soundex`, and `phonetics.utils`. These methods have been manually tested and are now available for use, contributing to issue [#2163](#2163) and progressing issue [#1901](#1901). As an adopting engineer, this addition enables you to incorporate these phonetics methods into your system's functionality, expanding the capabilities of the open-source library. * Whitelist pydantic ([#2162](#2162)). In this release, we have added the Pydantic library to the `known.json` file, which manages our project's third-party libraries. Pydantic is a data validation library for Python that allows developers to define data models and enforce type constraints, improving data consistency and correctness in the application. With this change, Pydantic and its submodules have been whitelisted and can be used in the project without being flagged as unknown libraries. This improvement enables us to utilize Pydantic's features for data validation and modeling, ensuring higher data quality and reducing the likelihood of errors in our application. * Whitelist statsmodels ([#2161](#2161)). In this change, the statsmodels library has been whitelisted for use in the project. Statsmodels is a comprehensive Python library for statistics and econometrics that offers a variety of tools for statistical modeling, testing, and visualization. With this update, the library has been added to the project's configuration file, enabling users to utilize its features without causing any conflicts. The modification does not affect the existing functionality of the project, but rather expands the range of statistical models and analysis tools available to users. Additionally, a test has been included to verify the successful integration of the library. These enhancements streamline the process of conducting statistical analysis and modeling within the project. * whitelist dbignite ([#2132](#2132)). A new commit has been made to whitelist the dbignite repository and add a set of codes and messages in the "known.json" file related to the use of RDD APIs on UC Shared Clusters and the change in the default format from Parquet to Delta in Databricks Runtime 8.0. The affected components include dbignite.fhir_mapping_model, dbignite.fhir_resource, dbignite.hosp_feeds, dbignite.hosp_feeds.adt, dbignite.omop, dbignite.omop.data_model, dbignite.omop.schemas, dbignite.omop.utils, and dbignite.readers. These changes are intended to provide information and warnings regarding the use of the specified APIs on UC Shared Clusters and the change in default format. It is important to note that no new methods have been added, and no existing functionality has been changed as part of this update. The focus of this commit is solely on the addition of the dbignite repository and its associated codes and messages. * whitelist duckdb ([#2134](#2134)). In this release, we have whitelisted the DuckDB library by adding it to the "known.json" file in the source code. DuckDB is an in-memory analytical database written in C++. This addition includes several modules such as `adbc_driver_duckdb`, `duckdb.bytes_io_wrapper`, `duckdb.experimental`, `duckdb.filesystem`, `duckdb.functional`, and `duckdb.typing`. Of particular note is the `duckdb.experimental.spark.sql.session` module, which includes a change in the default format for Databricks Runtime 8.0, from Parquet to Delta. This change is indicated by the `table-migrate` code and message in the commit. Additionally, the commit includes tests that have been manually verified. DuckDB is a powerful new addition to our library, and we are excited to make it available to our users. * whitelist fs ([#2136](#2136)). In this release, we have added the `fs` package to the `known.json` file, allowing its use in our open-source library. The `fs` package contains a wide range of modules and sub-packages, including `fs._bulk`, `fs.appfs`, `fs.base`, `fs.compress`, `fs.copy`, `fs.error_tools`, `fs.errors`, `fs.filesize`, `fs.ftpfs`, `fs.glob`, `fs.info`, `fs.iotools`, `fs.lrucache`, `fs.memoryfs`, `fs.mirror`, `fs.mode`, `fs.mountfs`, `fs.move`, `fs.multifs`, `fs.opener`, `fs.osfs`, `fs.path`, `fs.permissions`, `fs.subfs`, `fs.tarfs`, `fs.tempfs`, `fs.time`, `fs.tools`, `fs.tree`, `fs.walk`, `fs.wildcard`, `fs.wrap`, `fs.wrapfs`, and `fs.zipfs`. These additions address issue [#1901](#1901) and have been thoroughly manually tested to ensure proper functionality. * whitelist httpx ([#2139](#2139)). In this release, we have updated the "known.json" file to include the `httpx` library along with all its submodules. This change serves to whitelist the library, and it does not introduce any new functionality or impact existing functionalities. The addition of `httpx` is purely for informational purposes, and it will not result in the inclusion of new methods or functions. Rest assured, the team has manually tested the changes, and the project's behavior remains unaffected. We recommend this update to software engineers looking to adopt our project, highlighting that the addition of `httpx` will only influence the library whitelist and not the overall functionality. * whitelist jsonschema and jsonschema-specifications ([#2140](#2140)). In this release, we have made changes to the "known.json" file to whitelist the `jsonschema` and `jsonschema-specifications` libraries. This modification addresses issue [#1901](#1901) and does not introduce any new functionality or tests. The `jsonschema` library is utilized for schema validation, while the `jsonschema-specifications` library offers additional specifications for the `jsonschema` library. By adding these libraries to the "known.json" file, we ensure that they are recognized as approved dependencies and are not flagged as unknown or unapproved in the future. This enhancement improves the reliability and efficiency of our dependency management system, making it easier for software engineers to work with these libraries. * whitelist pickleshare ([#2141](#2141)). A new commit has been added to whitelist Pickleshare, a Python module for managing persistent data structures, in the known.json file. This change aligns with issue [#1901](#1901) and is a preparatory step to ensure Pickleshare's compatibility with the project. The Pillow module is already included in the whitelist. No new functionality has been introduced, and existing functionality remains unchanged. The purpose of the whitelist is not explicitly stated in the given context. As a software engineer integrating this project, you are advised to verify the necessity of whitelisting Pickleshare for your specific use case. * whitelist referencing ([#2142](#2142)). This commit introduces a new whitelist referencing feature, which includes the creation of a `referencing` section in the "known.json" file. The new section contains several entries, including "referencing._attrs", "referencing._core", "referencing.exceptions", "referencing.jsonschema", "referencing.retrieval", and "referencing.typing", all of which are initially empty. This change is a step towards completing issue [#2142](#2142) and addresses issue [#1901](#1901). Manual testing has been conducted to ensure the proper functioning of the new functionality. This enhancement was co-authored by Eric Vergnaud. * whitelist slicer ([#2143](#2143)). A new security measure has been implemented in the slicer module with the addition of a whitelist that specifies allowed modules and functions. The whitelist is implemented as a JSON object in the `known.json` file, preventing unauthorized access or usage of certain parts of the codebase. A test has been included to verify the functionality of the whitelist, ensuring that the slicer module is secure and functioning as intended. No new methods were added and existing functionality remains unchanged. The changes are localized to the `known.json` file and the slicer module, enhancing the security and integrity of the project. This feature was developed by Eric Vergnaud and myself. * whitelist sparse ([#2144](#2144)). In this release, we have whitelisted the `sparse` module, adding it to the known.json file. This module encompasses various sub-modules and components such as _common, _compressed, _coo, _dok, _io, _numba_extension, _settings, _slicing, _sparse_array, _umath, _utils, finch_backend, and numba_backend. Each component may contain additional classes, functions, or functionality, and the numba_backend sub-module includes further sub-components. This change aims to improve organization, enhance codebase understanding, and prevent accidental deletion or modification of critical code. The modification is in reference to issue [#1901](#1901) for additional context. Comprehensive testing has been carried out to guarantee the correct implementation of the whitelisting. * whitelist splink ([#2145](#2145)). In this release, we have added the `splink` library to our known_json file, which includes various modules and functions for entity resolution and data linking. This change is in line with issue [#190](#190) * whitelist toolz ([#2146](#2146)). In this release, we have whitelisted the `toolz` library and added it to the known.json file. The `toolz` library is a collection of functional utilities, compatible with CPython, PyPy, Jython, and IronPython, and is a port of various modules from Python's standard library and other open-source packages. The newly added modules include tlz, toolz, toolz._signatures, toolz._version, toolz.compatibility, toolz.curried, toolz.dicttoolz, toolz.functoolz, toolz.itertoolz, toolz.recipes, toolz.sandbox, toolz.sandbox.core, toolz.sandbox.parallel, and toolz.utils. These changes have been manually tested and may address issue [#1901](#1901). * whitelist xmod ([#2147](#2147)). In this release, we have made a modification to the open-source library that involves whitelisting `xmod` in the known.json file. This change includes the addition of a new key for `xmod` with an empty array as its initial value. It is important to note that this modification does not alter the existing functionality of the code. The development team has thoroughly tested the changes through manual testing to ensure proper implementation. This update is a significant milestone towards the progress of issue [#1901](#1901). Software engineers are encouraged to incorporate these updates in their code to leverage the new whitelisting functionality for "xmod." Dependency updates: * Updated databricks-labs-lsql requirement from ~=0.5.0 to >=0.5,<0.7 ([#2160](#2160)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Current Behavior
when running
make solacc, a number of failures occur, and many advices are false positivesExpected Behavior
All errors should be caught and all advices should be relevant
Steps To Reproduce
No response
Cloud
AWS
Operating System
macOS
Version
latest via Databricks CLI
Relevant log output
No response
The text was updated successfully, but these errors were encountered: