feat: High performance pandas integration.

README.rst

@@ -34,6 +34,22 @@ The latest version of the library is 1.0.2.
             columns={'temperature': 20.0, 'humidity': 0.5})
         sender.flush()
 
+You can also send Pandas dataframes:
+
+.. code-block:: python
+
+    import pandas as pd
+    from questdb.ingress import Sender
+
+    df = pd.DataFrame({
+        'id': pd.Categorical(['toronto1', 'paris3']),
+        'temperature': [20.0, 21.0],
+        'humidity': [0.5, 0.6],
+        'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])'})
+
+    with Sender('localhost', 9009) as sender:
+        sender.dataframe(df, table_name='sensors')
+
 
 Docs
 ====

docs/installation.rst

@@ -5,9 +5,9 @@ Installation
 The Python QuestDB client does not have any additional run-time dependencies and
 will run on any version of Python >= 3.7 on most platforms and architectures.
 
-You can install it globally by running::
+You can install it (or updated it) globally by running::
 
-    python3 -m pip install questdb
+    python3 -m pip install -U questdb
 
 
 Or, from within a virtual environment::
@@ -20,6 +20,15 @@ If you're using poetry, you can add ``questdb`` as a dependency::
     poetry add questdb
 
 
+Note that the :func:`questdb.ingress.Buffer.dataframe` and the
+:func:`questdb.ingress.Sender.dataframe` methods also require the following
+dependencies to be installed:
+
+* ``pandas``
+* ``pyarrow``
+* ``numpy``
+
+
 Verifying the Installation
 ==========================
 
@@ -34,3 +43,16 @@ following statements from a ``python3`` interactive shell:
     <questdb.ingress.Buffer object at 0x104b68240>
     >>> str(buf)
     'test,a=b\n'
+
+If you also want to check you can serialize from Pandas
+(which requires additional dependencies):
+
+.. code-block:: python
+
+    >>> import questdb.ingress
+    >>> import pandas as pd
+    >>> df = pd.DataFrame({'a': [1, 2]})
+    >>> buf = questdb.ingress.Buffer()
+    >>> buf.dataframe(df, table_name='test')
+    >>> str(buf)
+    'test a=1i\ntest a=2i\n'

src/questdb/ingress.pyx

@@ -1200,6 +1200,31 @@ cdef class Buffer:
               If a datetime value is specified as ``None`` (``NaT``), it is
               interpreted as the current QuestDB server time set on receipt of
               message.
+
+        **Error Handling and Recovery**
+
+        In case an exception is raised during dataframe serialization, the
+        buffer is left in its previous state.
+        The buffer remains in a valid state and can be used for further calls
+        even after an error.
+
+        For clarification, as an example, if an invalid ``None``
+        value appears at the 3rd row for a ``bool`` column, neither the 3rd nor
+        the preceding rows are added to the buffer.
+
+        **Note**: This differs from the :func:`Sender.dataframe` method, which
+        modifies this guarantee due to its ``auto_flush`` logic.
+
+        **Performance Considerations**
+
+        The Python GIL is released during serialization if it is not needed.
+        If any column requires the GIL, the entire serialization is done whilst
+        holding the GIL.
+
+        Column types that require the GIL are:
+
+        * Columns of ``str``, ``float`` or ``int`` or ``float`` Python objects.
+        * The ``'string[python]'`` dtype.
         """
         _dataframe(
             auto_flush_blank(),
@@ -1599,6 +1624,11 @@ cdef class Sender:
 
         Additionally, this method also supports auto-flushing the buffer
         as specified in the ``Sender``'s ``auto_flush`` constructor argument.
+        Auto-flushing is implemented incrementally, meanting that when
+        calling ``sender.dataframe(df)`` with a large ``df``, the sender may
+        have sent some of the rows to the server already whist the rest of the
+        rows are going to be sent at the next auto-flush or next explicit call
+        to :func:`Sender.flush`.
 
         In case of data errors with auto-flushing enabled, some of the rows
         may have been transmitted to the server already.
@@ -1636,6 +1666,8 @@ cdef class Sender:
             If ``False``, the flushed buffer is left in the internal buffer.
             Note that ``clear=False`` is only supported if ``buffer`` is also
             specified.
+
+        The Python GIL is released during the network IO operation.
         """
         cdef line_sender* sender = self._impl
         cdef line_sender_error* err = NULL