LaTeX: allow more cases of table nesting, fix #13646

Tables using longtable can now contain nested tables inclusive of those
rendered by tabulary, up to the suppression of the latter horizontal
lines due to an upstream LaTeX bug.  A longtable can never itself be
nested, and will fall-back to tabular.

Formerly longtable would raise (in principle) an error if it contained
any sort of nested table, but the detection of being a longtable was
faulty if not specified as class option.

Relates #6838.

Author: jfbu

doc/usage/restructuredtext/directives.rst

@@ -1472,6 +1472,15 @@ Check the :confval:`latex_table_style`.
    complex contents such as multiple paragraphs, blockquotes, lists, literal
    blocks, will render correctly to LaTeX output.
 
+.. versionchanged:: 8.3.0
+
+   The partial support for nesting a table in another has been extended.
+   Formerly Sphinx would raise an error if ``longtable`` class was specified
+   for a table containing a nested table, and some cases would not raise an
+   error at Sphinx level but fail at LaTeX level during PDF build.  This is a
+   complex topic in LaTeX rendering and the output can sometimes be improved
+   via the :rst:dir:`tabularcolumns` directive.
+
 .. rst:directive:: .. tabularcolumns:: column spec
 
    This directive influences only the LaTeX output for the next table in
@@ -1489,40 +1498,38 @@ Check the :confval:`latex_table_style`.
       :rst:dir:`tabularcolumns` conflicts with ``:widths:`` option of table
       directives.  If both are specified, ``:widths:`` option will be ignored.
 
-   Sphinx will render tables with more than 30 rows with ``longtable``.
-   Besides the ``l``, ``r``, ``c`` and ``p{width}`` column specifiers, one can
-   also use ``\X{a}{b}`` (new in version 1.5) which configures the column
-   width to be a fraction ``a/b`` of the total line width and ``\Y{f}`` (new
-   in version 1.6) where ``f`` is a decimal: for example ``\Y{0.2}`` means that
-   the column will occupy ``0.2`` times the line width.
+   Sphinx renders tables with at most 30 rows using ``tabulary``, and those
+   with more rows with ``longtable``.
 
-   When this directive is used for a table with at most 30 rows, Sphinx will
-   render it with ``tabulary``.  One can then use specific column types ``L``
-   (left), ``R`` (right), ``C`` (centered) and ``J`` (justified).  They have
-   the effect of a ``p{width}`` (i.e. each cell is a LaTeX ``\parbox``) with
-   the specified internal text alignment and an automatically computed
-   ``width``.
-
-   .. warning::
+   ``tabulary`` tries to compute automatically (internally to LaTeX) suitable
+   column widths.  However, cells are then not allowed to contain
+   "problematic" elements such as lists, object descriptions,
+   blockquotes... Sphinx will fall back to using ``tabular`` if such a cell is
+   encountered (or a nested ``tabulary``).  In such a case the table will have
+   a tendency to try to fill the whole available line width.
 
-      - Cells that contain list-like elements such as object descriptions,
-        blockquotes or any kind of lists are not compatible with the ``LRCJ``
-        column types.  The column type must then be some ``p{width}`` with an
-        explicit ``width`` (or ``\X{a}{b}`` or ``\Y{f}``).
+   :rst:dir:`tabularcolumns` can help in coercing the usage of ``tabulary`` if
+   one is careful to not employ the ``tabulary`` column types (``L``, ``R``,
+   ``C`` or ``J``) for those columns with at least one "problematic" cell, but
+   only LaTeX's ``p{<width>}`` or Sphinx ``\X`` and ``\Y`` (described next).
 
-      - Literal blocks do not work with ``tabulary`` at all.  Sphinx will
-        fall back to ``tabular`` or ``longtable`` environments and generate a
-        suitable column specification.
+   Literal blocks do not work at all with ``tabulary``.  Sphinx will fall back
+   to ``tabular`` or ``longtable`` environments depending on the number of
+   rows.  It will employ the :rst:dir:`tabularcolumns` specification only if it
+   contains no usage of the ``tabulary`` specific types.
 
-In absence of the :rst:dir:`tabularcolumns` directive, and for a table with at
-most 30 rows and no problematic cells as described in the above warning,
-Sphinx uses ``tabulary`` and the ``J`` column-type for every column.
+   Besides the LaTeX ``l``, ``r``, ``c`` and ``p{width}`` column specifiers,
+   one can also use ``\X{a}{b}`` which configures the column width to be a
+   fraction ``a/b`` of the total line width and ``\Y{f}`` where ``f`` is a
+   decimal: for example ``\Y{0.2}`` means that the column will occupy ``0.2``
+   times the line width.
 
 .. versionchanged:: 1.6
 
-   Formerly, the ``L`` column-type was used (text is flushed-left).  To revert
-   to this, include ``\newcolumntype{T}{L}`` in the LaTeX preamble, as in fact
-   Sphinx uses ``T`` and sets it by default to be an alias of ``J``.
+   Use ``J`` (justified) by default with ``tabulary``, not ``L``
+   (flushed-left).  To revert, include ``\newcolumntype{T}{L}`` in the LaTeX
+   preamble, as in fact Sphinx uses ``T`` and sets it by default to be an
+   alias of ``J``.
 
 .. hint::
 

sphinx/templates/latex/tabulary.tex.jinja

@@ -21,6 +21,9 @@
 <% if 'nocolorrows' in table.styles -%>
 \sphinxthistablewithnocolorrowsstyle
 <% endif -%>
+<% if table.is_nested -%>
+\sphinxthistabularywithnohlinesifinlongtable
+<% endif -%>
 <% if table.align -%>
   <%- if table.align in ('center', 'default') -%>
   \centering

sphinx/texinputs/sphinxlatextables.sty

@@ -47,6 +47,8 @@
 % The method here is with no changes to neither writer nor templates.
 \newif\ifspx@intable
 \newif\ifspx@thistableisnested
+% Try to allow nested tables in a longtable.  But tabulary causes problems.
+\newif\ifspx@longtable
 % 
 % Also provides user command (see docs)
 % - \sphixncolorblend
@@ -115,6 +117,7 @@
     \edef\sphinxbaselineskip{\dimexpr\the\dimexpr\baselineskip\relax\relax}%
     \spx@inframedtrue % message to sphinxheavybox
     \spx@table@setnestedflags
+    \spx@longtabletrue
 }
 % Compatibility with caption package
 \def\sphinxthelongtablecaptionisattop{%
@@ -128,7 +131,10 @@
 \def\sphinxatlongtableend{\@nobreakfalse % latex3/latex2e#173
     \prevdepth\z@\vskip\sphinxtablepost\relax}%
 % B. Table with tabular or tabulary
-\def\sphinxattablestart{\par\vskip\dimexpr\sphinxtablepre\relax
+\def\sphinxattablestart{\par
+    \ifvmode % guard agains being nested in a table cell
+      \vskip\dimexpr\sphinxtablepre\relax
+    \fi
                         \spx@inframedtrue % message to sphinxheavybox
                         \spx@table@setnestedflags
                         }%
@@ -142,7 +148,12 @@
         \spx@intabletrue
     \fi
    }%
-\let\sphinxattableend\sphinxatlongtableend
+\def\sphinxattableend{%
+    \@nobreakfalse % <- probably unneeded as this is not a longtable
+    \ifvmode       % guard against being nested in a table cell
+      \prevdepth\z@\vskip\sphinxtablepost\relax
+    \fi
+}%
 % This is used by tabular and tabulary templates
 \newcommand*\sphinxcapstartof[1]{%
    \vskip\parskip
@@ -1083,14 +1094,20 @@ local use of booktabs table style}%
 
 % borderless style
 \def\sphinxthistablewithborderlessstyle{%
+    \sphinxthistablewithnohlines
+    \def\spx@arrayrulewidth{\z@}%
+}%
+\def\sphinxthistablewithnohlines{%
     \let\sphinxhline      \@empty
     \let\sphinxcline      \@gobble
     \let\sphinxvlinecrossing\@gobble
     \let\sphinxfixclines    \@gobble
     \let\spx@toprule      \@empty
     \let\sphinxmidrule    \@empty
     \let\sphinxbottomrule \@empty
-    \def\spx@arrayrulewidth{\z@}%
+}%
+\def\sphinxthistabularywithnohlinesifinlongtable{%
+  \ifspx@longtable\sphinxthistablewithnohlines\fi
 }%
 
 % colorrows style

sphinx/writers/latex.py

@@ -134,6 +134,8 @@ def __init__(self, node: Element) -> None:
         self.has_problematic = False
         self.has_oldproblematic = False
         self.has_verbatim = False
+        # cf https://github.com/sphinx-doc/sphinx/issues/13646#issuecomment-2958309632
+        self.is_nested = False
         self.entry_needs_linetrimming = 0
         self.caption: list[str] = []
         self.stubs: list[int] = []
@@ -147,29 +149,47 @@ def __init__(self, node: Element) -> None:
         self.cell_id = 0  # last assigned cell_id
 
     def is_longtable(self) -> bool:
-        """True if and only if table uses longtable environment."""
+        """True if and only if table uses longtable environment.
+
+        In absence of longtable class can only be used trustfully on departing
+        the table, as the number of rows is not known until then.
+        """
         return self.row > 30 or 'longtable' in self.classes
 
     def get_table_type(self) -> str:
         """Returns the LaTeX environment name for the table.
 
+        It is used at time of ``depart_table()`` and again via ``get_colspec()``.
         The class currently supports:
 
         * longtable
         * tabular
         * tabulary
         """
-        if self.is_longtable():
+        if self.is_longtable() and not self.is_nested:
             return 'longtable'
         elif self.has_verbatim:
             return 'tabular'
         elif self.colspec:
-            return 'tabulary'
+            if any(c in 'LRCJT' for c in self.colspec):
+                # tabulary would complain "no suitable columns" if none of its
+                # column type were used so we ensure at least one matches.
+                # It is responsability of user to make sure not to use tabulary
+                # column types for a column containing a problematic cell.
+                return 'tabulary'
+            else:
+                return 'tabular'
         elif self.has_problematic or (
             self.colwidths and 'colwidths-given' in self.classes
         ):
             return 'tabular'
         else:
+            # A nested tabulary in a longtable can not use any \hline's,
+            # i.e. it can not use "booktabs" or "standard" styles (due to a
+            # LaTeX upstream bug we do not try to solve).  But we can't know
+            # here if it ends up in a tabular or longtable.  So it is via
+            # LaTeX macros inserted by the tabulary template that the problem
+            # will be solved.
             return 'tabulary'
 
     def get_colspec(self) -> str:
@@ -179,6 +199,7 @@ def get_colspec(self) -> str:
 
         .. note::
 
+           This is used by the template renderer at time of depart_table().
            The ``\\X`` and ``T`` column type specifiers are defined in
            ``sphinxlatextables.sty``.
         """
@@ -1146,23 +1167,17 @@ def visit_tabular_col_spec(self, node: Element) -> None:
         raise nodes.SkipNode
 
     def visit_table(self, node: Element) -> None:
-        if len(self.tables) == 1:
-            assert self.table is not None
-            if self.table.get_table_type() == 'longtable':
-                raise UnsupportedError(
-                    '%s:%s: longtable does not support nesting a table.'
-                    % (self.curfilestack[-1], node.line or '')
-                )
-            # change type of parent table to tabular
-            # see https://groups.google.com/d/msg/sphinx-users/7m3NeOBixeo/9LKP2B4WBQAJ
-            self.table.has_problematic = True
-        elif len(self.tables) > 2:
+        table = Table(node)
+        assert table is not None
+        if len(self.tables) >= 1:
+            table.is_nested = True
+        # TODO: do we want > 2, > 1, or actually nothing here?
+        if len(self.tables) > 2:
             raise UnsupportedError(
                 '%s:%s: deeply nested tables are not implemented.'
                 % (self.curfilestack[-1], node.line or '')
             )
 
-        table = Table(node)
         self.tables.append(table)
         if table.colsep is None:
             table.colsep = '|' * (
@@ -1191,6 +1206,25 @@ def depart_table(self, node: Element) -> None:
         assert self.table is not None
         labels = self.hypertarget_to(node)
         table_type = self.table.get_table_type()
+        if table_type == 'tabulary':
+            if len(self.tables) > 1:
+                # tell parents to not be tabulary
+                for _ in self.tables[:-1]:
+                    _.has_problematic = True
+        else:
+            if self.table.colspec:
+                if any(c in self.table.colspec for c in 'LRJCT'):
+                    logger.warning(
+                        __(
+                            'colspec %s was given which uses '
+                            'tabulary syntax.  But this table can not be '
+                            'rendered as a tabulary; colspec will be ignored.'
+                        ),
+                        self.table.colspec[:-1],
+                        type='latex',
+                        location=node,
+                    )
+                    self.table.colspec = ''
         table = self.render(
             table_type + '.tex.jinja', {'table': self.table, 'labels': labels}
         )

tests/roots/test-latex-table/expects/tabularcolumn.tex

@@ -4,7 +4,7 @@
 \sphinxthistablewithglobalstyle
 \sphinxthistablewithnovlinesstyle
 \centering
-\begin{tabulary}{\linewidth}[t]{cc}
+\begin{tabular}[t]{cc}
 \sphinxtoprule
 \sphinxstyletheadfamily 
 \sphinxAtStartPar
@@ -36,6 +36,6 @@
 cell3\sphinxhyphen{}2
 \\
 \sphinxbottomrule
-\end{tabulary}
+\end{tabular}
 \sphinxtableafterendhook\par
 \sphinxattableend\end{savenotes}

tests/roots/test-root/markup.txt

@@ -229,6 +229,14 @@ Tables with multirow and multicol:
    | +---+ |
    +---+---+
 
+   .. rst-class:: longtable
+
+   +---+---+
+   | +---+ |
+   | | h | |
+   | +---+ |
+   +---+---+
+
 .. list-table::
    :header-rows: 0