diff --git a/docs/src/main/sphinx/develop/connectors.rst b/docs/src/main/sphinx/develop/connectors.rst index ae8d428ad7c5..49f2b1042b30 100644 --- a/docs/src/main/sphinx/develop/connectors.rst +++ b/docs/src/main/sphinx/develop/connectors.rst @@ -12,30 +12,74 @@ ConnectorFactory Instances of your connector are created by a ``ConnectorFactory`` instance which is created when Trino calls ``getConnectorFactory()`` on the -plugin. The connector factory is a simple interface responsible for creating an -instance of a ``Connector`` object that returns instances of the -following services: +plugin. The connector factory is a simple interface responsible for providing +the connector name and creating an instance of a ``Connector`` object. +A basic connector implementation that only supports reading, but +not writing data, should return instances of the following services: -* ``ConnectorMetadata`` -* ``ConnectorSplitManager`` -* ``ConnectorRecordSetProvider`` +* :ref:`connector-metadata` +* :ref:`connector-split-manager` +* :ref:`connector-record-set-provider` or :ref:`connector-page-source-provider` + +.. _connector-metadata: ConnectorMetadata ^^^^^^^^^^^^^^^^^ -The connector metadata interface has a large number of important -methods that are responsible for allowing Trino to look at lists of -schemas, lists of tables, lists of columns, and other metadata about a -particular data source. +The connector metadata interface allows Trino to get a lists of schemas, +tables, columns, and other metadata about a particular data source. + +A basic read-only connector should implement the following methods: + +* ``listSchemaNames`` +* ``listTables`` +* ``streamTableColumns`` +* ``getTableHandle`` +* ``getTableMetadata`` +* ``getColumnHandles`` +* ``getColumnMetadata`` + +If you are interested in seeing strategies for implementing more methods, +look at the :doc:`example-http` and the Cassandra connector. If your underlying +data source supports schemas, tables and columns, this interface should be +straightforward to implement. If you are attempting to adapt something that +is not a relational database (as the Example HTTP connector does), you may +need to get creative about how you map your data source to Trino's schema, +table, and column concepts. + +The connector metadata interface allows to also implement other connector +features, like: + +* Schema management, that is creating, altering and dropping schemas, tables, + table columns, views, and materialized views. +* Support for table and column comments, and properties. +* Schema, table and view authorization. +* Executing :doc:`table-functions`. +* Providing table statistics used by the CBO and collecting statistics + during writes and when analyzing selected tables. +* Data modification, that is: + + * inserting, updating, and deleting rows in tables, + * refreshing materialized views, + * truncating whole tables, + * and creating tables from query results. -This interface is too big to list in this documentation, but if you -are interested in seeing strategies for implementing these methods, -look at the :doc:`example-http` and the Cassandra connector. If -your underlying data source supports schemas, tables and columns, this -interface should be straightforward to implement. If you are attempting -to adapt something that is not a relational database (as the Example HTTP -connector does), you may need to get creative about how you map your -data source to Trino's schema, table, and column concepts. +* Role and grant management. +* Pushing down: + + * Limit + * Predicates + * Projections + * Sampling + * Aggregations + * Joins + * Top N - limit with sort items + * Table function invocation + +Note that data modification also requires implementing +a :ref:`connector-page-sink-provider`. + +.. _connector-split-manager: ConnectorSplitManager ^^^^^^^^^^^^^^^^^^^^^ @@ -48,6 +92,8 @@ For data sources that don't have partitioned data, a good strategy here is to simply return a single split for the entire table. This is the strategy employed by the Example HTTP connector. +.. _connector-record-set-provider: + ConnectorRecordSetProvider ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -55,3 +101,60 @@ Given a split and a list of columns, the record set provider is responsible for delivering data to the Trino execution engine. It creates a ``RecordSet``, which in turn creates a ``RecordCursor`` that is used by Trino to read the column values for each row. + +.. _connector-page-source-provider: + +ConnectorPageSourceProvider +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given a split and a list of columns, the page source provider is +responsible for delivering data to the Trino execution engine. +It creates a ``ConnectorPageSource``, which in turn creates ``Page`` objects +that are used by Trino to read the column values. + +If not implemented, a default ``RecordPageSourceProvider`` will be used. +Given a record set provider, it returns an instance of ``RecordPageSource`` +that builds ``Page`` objects from records in a record set. + +A connector should implement a page source provider instead of a record set +provider when it's possible to create pages directly. The conversion of individual +records from a record set provider into pages adds overheads during query execution. + +To add support for updating and/or deleting rows in a connector, it needs +to implement a ``ConnectorPageSourceProvider`` that returns +an ``UpdatablePageSource``. See :doc:`delete-and-update` for more. + +.. _connector-page-sink-provider: + +ConnectorPageSinkProvider +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given an insert table handle, the page sink provider is responsible for +consuming data from the Trino execution engine. +It creates a ``ConnectorPageSink``, which in turn accepts ``Page`` objects +that contains the column values. + +Example that shows how to iterate over the page to access single values: + +.. code-block:: java + + @Override + public CompletableFuture appendPage(Page page) + { + for (int channel = 0; channel < page.getChannelCount(); channel++) { + Block block = page.getBlock(channel); + for (int position = 0; position < page.getPositionCount(); position++) { + if (block.isNull(position)) { + // or handle this differently + continue; + } + + // channel should match the column number in the table + // use it to determine the expected column type + String value = VARCHAR.getSlice(block, position).toStringUtf8(); + // TODO do something with the value + } + } + return NOT_BLOCKED; + } + diff --git a/docs/src/main/sphinx/develop/spi-overview.rst b/docs/src/main/sphinx/develop/spi-overview.rst index df4dff084e0c..d2646e260be1 100644 --- a/docs/src/main/sphinx/develop/spi-overview.rst +++ b/docs/src/main/sphinx/develop/spi-overview.rst @@ -3,10 +3,24 @@ SPI overview ============ When you implement a new Trino plugin, you implement interfaces and -override methods defined by the SPI. +override methods defined by the Service Provider Interface (SPI). + +Plugins can provide additional: + +* :doc:`connectors`, +* block encodings, +* :doc:`types`, +* :doc:`functions`, +* :doc:`system-access-control`, +* :doc:`group-provider`, +* :doc:`password-authenticator`, +* :doc:`header-authenticator`, +* :doc:`certificate-authenticator`, +* :doc:`event-listener`, +* resource group configuration managers, +* session property configuration managers, +* and exchange managers. -Plugins can provide additional :doc:`connectors`, :doc:`types`, -:doc:`functions`, and :doc:`system-access-control`. In particular, connectors are the source of all data for queries in Trino: they back each catalog available to Trino. @@ -82,11 +96,21 @@ For an example ``pom.xml`` file, see the example HTTP connector in the Deploying a custom plugin ------------------------- -In order to add a custom plugin to a Trino installation, create a directory -for that plugin in the Trino plugin directory and add all the necessary jars -for the plugin to that directory. For example, for a plugin called -``my-functions``, you would create a directory ``my-functions`` in the Trino -plugin directory and add the relevant jars to that directory. +Because Trino plugins use the ``trino-plugin`` packaging type, building +a plugin will create a ZIP file in the ``target`` directory. This file +contains the plugin JAR and all its dependencies JAR files. + +In order to add a custom plugin to a Trino installation, extract the plugin +ZIP file and move the extracted directory into the Trino plugin directory. +For example, for a plugin called ``my-functions``, with a version of 1.0, +you would extract ``my-functions-1.0.zip`` and then move ``my-functions-1.0`` +to ``my-functions`` in the Trino plugin directory. + +.. note:: + + Every Trino plugin should be in a separate directory. Do not put JAR files + directly into the ``plugin`` directory. Plugins should only contain JAR files, + so any subdirectories will not be traversed and will be ignored. By default, the plugin directory is the ``plugin`` directory relative to the directory in which Trino is installed, but it is configurable using the