feat: a manager for SQLAlchemy engines

UPDATING.md

@@ -24,6 +24,41 @@ assists people when migrating to a new version.
 
 ## Next
 
+### Engine Manager for Connection Pooling
+
+A new `EngineManager` class has been introduced to centralize SQLAlchemy engine creation and management. This enables connection pooling for analytics databases and provides a more flexible architecture for engine configuration.
+
+#### Breaking Changes
+
+1. **Removed `SSH_TUNNEL_MANAGER_CLASS` config**: SSH tunnel handling is now integrated into the EngineManager. If you have custom SSH tunnel managers, you'll need to migrate to the new architecture.
+
+2. **Removed `nullpool` parameter**: The `get_sqla_engine()` and `get_raw_connection()` methods on the `Database` model no longer accept a `nullpool` parameter. Pool configuration is now controlled through the engine manager.
+
+3. **Removed `_get_sqla_engine()` method**: The private `_get_sqla_engine()` method has been removed from the `Database` model. All engine creation now goes through the `EngineManager`.
+
+#### New Configuration Options
+
+```python
+# Engine manager mode:
+# - EngineModes.NEW: Creates a new engine for every connection (default, original behavior)
+# - EngineModes.SINGLETON: Reuses engines with connection pooling
+from superset.engines.manager import EngineModes
+ENGINE_MANAGER_MODE = EngineModes.NEW
+
+# Cleanup interval for abandoned locks (default: 5 minutes)
+from datetime import timedelta
+ENGINE_MANAGER_CLEANUP_INTERVAL = timedelta(minutes=5)
+
+# Automatically start cleanup thread for SINGLETON mode (default: True)
+ENGINE_MANAGER_AUTO_START_CLEANUP = True
+```
+
+#### Migration Guide
+
+- If you were using the `nullpool` parameter, remove it from your calls
+- If you had a custom `SSH_TUNNEL_MANAGER_CLASS`, refactor to use the new EngineManager architecture
+- If you need connection pooling, set `ENGINE_MANAGER_MODE = EngineModes.SINGLETON` and configure the pool in your database's `extra` JSON field
+
 ### WebSocket config for GAQ with Docker
 
 [35896](https://github.com/apache/superset/pull/35896) and [37624](https://github.com/apache/superset/pull/37624) updated documentation on how to run and configure Superset with Docker. Specifically for the WebSocket configuration, a new `docker/superset-websocket/config.example.json` was added to the repo, so that users could copy it to create a `docker/superset-websocket/config.json` file. The existing `docker/superset-websocket/config.json` was removed and git-ignored, so if you're using GAQ / WebSocket make sure to:

superset/config.py

@@ -52,10 +52,15 @@
 from superset.advanced_data_type.plugins.internet_port import internet_port
 from superset.advanced_data_type.types import AdvancedDataType
 from superset.constants import CHANGE_ME_SECRET_KEY
+from superset.engines.manager import EngineModes
 from superset.jinja_context import BaseTemplateProcessor
 from superset.key_value.types import JsonKeyValueCodec
 from superset.stats_logger import DummyStatsLogger
-from superset.superset_typing import CacheConfig
+from superset.superset_typing import (
+    CacheConfig,
+    DBConnectionMutator,
+    EngineContextManager,
+)
 from superset.tasks.types import ExecutorType
 from superset.themes.types import Theme
 from superset.utils import core as utils
@@ -260,6 +265,22 @@ def _try_json_readsha(filepath: str, length: int) -> str | None:
 # SQLALCHEMY_CUSTOM_PASSWORD_STORE = lookup_password
 SQLALCHEMY_CUSTOM_PASSWORD_STORE = None
 
+# ---------------------------------------------------------
+# Engine Manager Configuration
+# ---------------------------------------------------------
+
+# Engine manager mode: "NEW" creates a new engine for every connection (default),
+# "SINGLETON" reuses engines with connection pooling
+ENGINE_MANAGER_MODE = EngineModes.NEW
+
+# Cleanup interval for abandoned locks in seconds (default: 5 minutes)
+ENGINE_MANAGER_CLEANUP_INTERVAL = timedelta(minutes=5)
+
+# Automatically start cleanup thread for SINGLETON mode (default: True)
+ENGINE_MANAGER_AUTO_START_CLEANUP = True
+
+# ---------------------------------------------------------
+
 #
 # The EncryptedFieldTypeAdapter is used whenever we're building SqlAlchemy models
 # which include sensitive fields that should be app-encrypted BEFORE sending
@@ -809,7 +830,6 @@ class D3TimeFormat(TypedDict, total=False):
 #                          FIREWALL (only port 22 is open)
 
 # ----------------------------------------------------------------------
-SSH_TUNNEL_MANAGER_CLASS = "superset.extensions.ssh.SSHManager"
 SSH_TUNNEL_LOCAL_BIND_ADDRESS = "127.0.0.1"
 #: Timeout (seconds) for tunnel connection (open_channel timeout)
 SSH_TUNNEL_TIMEOUT_SEC = 10.0
@@ -1684,7 +1704,7 @@ def engine_context_manager(  # pylint: disable=unused-argument
     yield None
 
 
-ENGINE_CONTEXT_MANAGER = engine_context_manager
+ENGINE_CONTEXT_MANAGER: EngineContextManager = engine_context_manager
 
 # A callable that allows altering the database connection URL and params
 # on the fly, at runtime. This allows for things like impersonation or
@@ -1701,7 +1721,7 @@ def engine_context_manager(  # pylint: disable=unused-argument
 #
 # Note that the returned uri and params are passed directly to sqlalchemy's
 # as such `create_engine(url, **params)`
-DB_CONNECTION_MUTATOR = None
+DB_CONNECTION_MUTATOR: DBConnectionMutator | None = None
 
 
 # A callable that is invoked for every invocation of DB Engine Specs

tests/unit_tests/extensions/ssh_test.py → superset/engines/__init__.py

@@ -14,23 +14,3 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-from unittest.mock import Mock
-
-import sshtunnel
-
-from superset.extensions.ssh import SSHManagerFactory
-
-
-def test_ssh_tunnel_timeout_setting() -> None:
-    app = Mock()
-    app.config = {
-        "SSH_TUNNEL_MAX_RETRIES": 2,
-        "SSH_TUNNEL_LOCAL_BIND_ADDRESS": "test",
-        "SSH_TUNNEL_TIMEOUT_SEC": 123.0,
-        "SSH_TUNNEL_PACKET_TIMEOUT_SEC": 321.0,
-        "SSH_TUNNEL_MANAGER_CLASS": "superset.extensions.ssh.SSHManager",
-    }
-    factory = SSHManagerFactory()
-    factory.init_app(app)
-    assert sshtunnel.TUNNEL_TIMEOUT == 123.0
-    assert sshtunnel.SSH_TIMEOUT == 321.0