LaTeX: longtable class forbids nested table but implicit longtable will allow it, and crash on PDF build

[jfbu]

Describe the bug

The following source:

.. table:: Test table
   :widths: auto
   :class: longtable

   =====  =====
     A    not A
   =====  =====
   False  True
   True   +---+
          | h |
          +---+
   =====  =====

is intercepted on build by an error:

    sphinx.writers.latex.UnsupportedError: index:: longtable does not support nesting a table.

If one does not explicitly use the longtable class but trigger it via enough many rows, then no error is triggered by the latex builder. However the nesting of tabulary inside a longtable causes a crash when building the PDF:

! Extra alignment tab has been changed to \cr.
<template> \endtemplate 
                        
l.156 \end{tabulary}
                    
? X

How to Reproduce

Use above source

Environment Information

Platform:              darwin
Python version:        3.13.3 (v3.13.3:6280bb54784, Apr  8 2025, 10:47:54) [Clang 15.0.0 (clang-1500.3.9.4)])
Python implementation: CPython
Sphinx version:        8.3.0+/a5366394a
Docutils version:      0.21.2
Jinja2 version:        3.1.6
Pygments version:      2.19.1

Sphinx extensions

Additional context

It is definitely possible to nest a (simple) tabular inside longtable, and even a (simple) tabulary. For example this is legal LaTeX source:

\begin{longtable}{|c|c|}
\hline
  A&B\\
\hline
  C&D\\
\hline
  E&
    \begin{tabulary}{\linewidth}{|c|c|}
      A&B\\\hline
      C&D\\\hline
    \end{tabulary}
\\\hline
\end{longtable}

and produces (not quite correct, due to horizontal grid lines of sub-table merging with outer table) output.

In the Sphinx case of tabulary inside longtable via enough many rows to trigger it, the PDF though can not be built.

Following possibilities of treatment of this report:

1. make sure all cases of a longtable do raise an error if a cell contains a table,
2. force a nested table in a longtable to never use tabulary but only tabular which seems to work with no issue,
3. understand why the Sphinx LaTeX mark-up for tabulary inside longtable causes a PDF build crash and fix that, so that longtable will only explicitly forbid nesting another longtable.

[jfbu]

The cause of this is that the LaTeX writer checks too early the table type at a time it is not decided yet if it will be a longtable.

At https://groups.google.com/g/sphinx-users/c/7m3NeOBixeo/m/9LKP2B4WBQAJ one finds the following table of possibilities

Regarding nesting, let's use TY=tabulary, TR=tabular, LT=longtable in table below

A can be nested in B:

A\B	TR	TY	LT
TR	YES	YES	YES
TY	YES	NO	YES
LT	NO	NO	NO

As I said in previous comment, a longtable can contain tabular and tabulary. The tabulary can not contain any kind of nested table apart from tabular.

Prior to #6838 no nesting was possible at all.

Update The table above is not correct, a tabulary inside a longtable will cause a crash if it uses \hline and the tabulary is configured with less columns than the longtable.

\documentclass{article}
\usepackage{array}
\usepackage{color}
\usepackage{colortbl}
\usepackage{tabulary}
\usepackage{longtable}

\begin{document}

% Tabulary in longtable crashes if
% 1. number of columns not at least as the one of longtable
% 2. and \hline is used in the tabulary
% ---> Extra alignment tab has been changed to \cr.
% No such problem with tabular inside longtable
\begin{longtable}{ccc}% 3 columns
  A&B&\\
  C&D&\\
  E&
    \begin{tabular}{cc}% 2 columns
      A&B\\\hline % ok
      C&D
    \end{tabular}
&
    \begin{tabulary}{\linewidth}{cc}% 2 columns
      A&B\\\hline % crash
      C&D
    \end{tabulary}
\end{longtable}
\end{document}

A\B	TR	TY	LT
TR	YES	YES	YES
TY	YES	NO	NO if using hline
LT	NO	NO	NO