Generated index.html closes surrounding html tags

[kai-tub]

Environment

Python Version: 3.9.6
Nikola Version: 8.1.3
Operating System: Linux

Description:

The actual issue is:
The main index.html landing page doesn't center the listed posts:

I would assume that the second post would also be directly under the first one.

To reproduce

nikola init --demo
nikola new_post -f markdown
nikola auto --browser

Looking at the output for the index.html one can see that too many tags are closed.
Specifically, referring to the source tags.tmpl file:
There is a surrounding div class that loops over the posts <div class="postindex">
If we add an HTML comment before/after loading the text, we see that the div and everything before that is closed and the next line of the loop starts with a new HTML tag.

nikola theme --copy-template index.tmpl

Add some comments in tmpl file, note that the endfor does not contain any </html> tag or similar

            <!-- From here to next comment post.text is inserted and article closes the main html tag, even if it shouldn't -->
            %if index_teasers:
                <div class="p-summary entry-summary">
                    ${post.text(teaser_only=True)}
                </div>
            %else:
                <div class="e-content entry-content">
                    ${post.text(teaser_only=False)}
                </div>
            %endif
            </article>
            <!-- According to index.tmpl should start with new article now! -->
        % endfor

The index.html in output becomes after build:

                    <!-- From here to next comment post.text is inserted and article closes the main html tag, even if it shouldn't --> <div class="e-content entry-content">
                        <p>Write your post here.</p>
                    </div>
                </article>
            </div>
        </div>
    </div>
</body>

</html><!-- According to index.tmpl should start with new article now! -->

[Kwpolska]

This looks like a duplicate of #3560. Please rm -rf cache and make sure all Nikola dependencies (in particular lxml, docutils, mako / jinja2 are up-to-date).

[kai-tub]

Interesting.
Yes, as in #3560 I am also using conda as a package manager (actually mamba with conda-forge).
I can reproduce the error by running:

mamba create --name nikola
conda activate nikola
mamba install -c conda-forge nikola
pip install Nikola[extra]

Then the output becomes ill-formatted.
Here is the exact dependency yml file, to reproduce:

name: nikola2
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - babel=2.9.1=pyh44b312d_0
  - blinker=1.4=py_1
  - brotlipy=0.7.0=py39h3811e60_1001
  - ca-certificates=2021.7.5=h06a4308_1
  - certifi=2021.5.30=py39hf3d152e_0
  - cffi=1.14.6=py39he32792d_0
  - chardet=4.0.0=py39hf3d152e_1
  - charset-normalizer=2.0.0=pyhd8ed1ab_0
  - cloudpickle=1.6.0=py_0
  - cryptography=3.4.7=py39hbca0aa6_0
  - docutils=0.17.1=py39hf3d152e_0
  - doit=0.33.1=py39hde42818_1
  - freetype=2.10.4=h0708190_1
  - icu=68.1=h58526e2_0
  - idna=3.2=pyhd3eb1b0_0
  - importlib-metadata=4.6.3=py39hf3d152e_0
  - jbig=2.1=h7f98852_2003
  - jpeg=9d=h516909a_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=2.2.1=h9c3ff4c_0
  - libdeflate=1.7=h7f98852_5
  - libffi=3.3=h58526e2_2
  - libgcc-ng=11.1.0=hc902ee8_8
  - libgomp=11.1.0=hc902ee8_8
  - libiconv=1.16=h516909a_0
  - libpng=1.6.37=hed695b0_2
  - libstdcxx-ng=11.1.0=h56837e0_8
  - libtiff=4.3.0=hf544144_1
  - libwebp-base=1.2.0=h7f98852_2
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - lxml=4.6.3=py39h107f48f_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - mako=1.1.4=pyh44b312d_0
  - markdown=3.3.4=pyhd8ed1ab_0
  - markupsafe=2.0.1=py39h3811e60_0
  - natsort=7.1.1=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - nikola=8.1.3=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1k=h7f98852_0
  - piexif=1.1.3=py_2
  - pillow=8.3.1=py39ha612740_0
  - pip=21.2.2=pyhd8ed1ab_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.9.0=pyhd8ed1ab_0
  - pyinotify=0.9.6=py39hf3d152e_1003
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyrss2gen=1.1=pyhd3deb0d_0
  - pysocks=1.7.1=py39hf3d152e_3
  - python=3.9.6=h49503c6_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - readline=8.1=h46c0cb4_0
  - requests=2.26.0=pyhd8ed1ab_0
  - setuptools=52.0.0=py39h06a4308_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.36.0=h9cd32fc_0
  - tk=8.6.10=hed695b0_1
  - tzdata=2021a=he74cb21_1
  - unidecode=1.2.0=pyhd8ed1ab_0
  - urllib3=1.26.6=pyhd8ed1ab_0
  - wheel=0.36.2=pyhd3deb0d_0
  - xz=5.2.5=h516909a_1
  - yapsy=1.12.2=py_0
  - zipp=3.5.0=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.5.0=ha95c52a_0
  - pip:
    - aiohttp==3.7.4.post0
    - argon2-cffi==20.1.0
    - async-generator==1.10
    - async-timeout==3.0.1
    - attrs==21.2.0
    - backcall==0.2.0
    - bleach==4.0.0
    - debugpy==1.4.1
    - decorator==5.0.9
    - defusedxml==0.7.1
    - entrypoints==0.3
    - ghp-import==2.0.1
    - hsluv==5.0.2
    - html5lib==1.1
    - ipykernel==6.0.3
    - ipython==7.26.0
    - ipython-genutils==0.2.0
    - jedi==0.18.0
    - jinja2==3.0.1
    - jsonschema==3.2.0
    - jupyter-client==6.1.12
    - jupyter-core==4.7.1
    - jupyterlab-pygments==0.1.2
    - matplotlib-inline==0.1.2
    - micawber==0.5.4
    - mistune==0.8.4
    - multidict==5.1.0
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.1
    - packaging==21.0
    - pandocfilters==1.4.3
    - parso==0.8.2
    - pexpect==4.8.0
    - phpserialize==1.3
    - pickleshare==0.7.5
    - prometheus-client==0.11.0
    - prompt-toolkit==3.0.19
    - ptyprocess==0.7.0
    - pygal==2.4.0
    - pyparsing==2.4.7
    - pyphen==0.11.0
    - pyrsistent==0.18.0
    - pyzmq==22.2.1
    - ruamel-yaml==0.17.10
    - ruamel-yaml-clib==0.2.6
    - send2trash==1.7.1
    - smartypants==2.0.1
    - terminado==0.10.1
    - testpath==0.5.0
    - toml==0.10.2
    - tornado==6.1
    - traitlets==5.0.5
    - typing-extensions==3.10.0.0
    - typogrify==2.0.7
    - watchdog==2.1.3
    - wcwidth==0.2.5
    - webencodings==0.5.1
    - yarl==1.6.3

If I create a new environment and install everything via pip, I do not get the error. The output of pip freeze is:

aiohttp==3.7.4.post0
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
bleach==4.0.0
blinker==1.4
certifi==2021.5.30
cffi==1.14.6
chardet==4.0.0
charset-normalizer==2.0.4
cloudpickle==1.6.0
debugpy==1.4.1
decorator==5.0.9
defusedxml==0.7.1
docutils==0.17.1
doit==0.33.1
entrypoints==0.3
ghp-import==2.0.1
hsluv==5.0.2
html5lib==1.1
idna==3.2
ipykernel==6.0.3
ipython==7.26.0
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==3.0.1
jsonschema==3.2.0
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
lxml==4.6.3
Mako==1.1.4
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib-inline==0.1.2
micawber==0.5.4
mistune==0.8.4
multidict==5.1.0
natsort==7.1.1
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nest-asyncio==1.5.1
Nikola==8.1.3
notebook==6.4.1
packaging==21.0
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
phpserialize==1.3
pickleshare==0.7.5
piexif==1.1.3
Pillow==8.3.1
prometheus-client==0.11.0
prompt-toolkit==3.0.19
ptyprocess==0.7.0
pycparser==2.20
pygal==2.4.0
Pygments==2.9.0
pyinotify==0.9.6
pyparsing==2.4.7
pyphen==0.11.0
pyrsistent==0.18.0
PyRSS2Gen==1.1
python-dateutil==2.8.2
pytz==2021.1
pyzmq==22.2.1
requests==2.26.0
ruamel.yaml==0.17.10
ruamel.yaml.clib==0.2.6
Send2Trash==1.7.1
six==1.16.0
smartypants==2.0.1
terminado==0.10.1
testpath==0.5.0
toml==0.10.2
tornado==6.1
traitlets==5.0.5
typing-extensions==3.10.0.0
typogrify==2.0.7
Unidecode==1.2.0
urllib3==1.26.6
watchdog==2.1.3
wcwidth==0.2.5
webencodings==0.5.1
Yapsy==1.12.2
yarl==1.6.3

I will do some extra research tomorrow and maybe I am able to find the underlying dependency issue.
So please let the issue open for now. :)

[kai-tub]

As expected, re-installing in the "broken" conda environment with:
pip install -U --force-reinstall Nikola[extras] fixes the issue.
At least from the version number, there seems to be no difference between

lxml (both 4.6.3)
mako (both 1.1.4
jinja2 (both 3.0.1)
docutils (both 0.17.3)

[kai-tub]

Bug is also reproducible after only installing from conda-forge feedstock
So only installing mamba install -c conda-forge nikola

name: nikola2
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - babel=2.9.1=pyh44b312d_0
  - blinker=1.4=py_1
  - brotlipy=0.7.0=py39h3811e60_1001
  - ca-certificates=2021.7.5=h06a4308_1
  - certifi=2021.5.30=py39hf3d152e_0
  - cffi=1.14.6=py39he32792d_0
  - chardet=4.0.0=py39hf3d152e_1
  - charset-normalizer=2.0.0=pyhd8ed1ab_0
  - cloudpickle=1.6.0=py_0
  - cryptography=3.4.7=py39hbca0aa6_0
  - docutils=0.17.1=py39hf3d152e_0
  - doit=0.33.1=py39hde42818_1
  - freetype=2.10.4=h0708190_1
  - icu=68.1=h58526e2_0
  - idna=3.2=pyhd3eb1b0_0
  - importlib-metadata=4.6.3=py39hf3d152e_0
  - jbig=2.1=h7f98852_2003
  - jpeg=9d=h516909a_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=2.2.1=h9c3ff4c_0
  - libdeflate=1.7=h7f98852_5
  - libffi=3.3=h58526e2_2
  - libgcc-ng=11.1.0=hc902ee8_8
  - libgomp=11.1.0=hc902ee8_8
  - libiconv=1.16=h516909a_0
  - libpng=1.6.37=hed695b0_2
  - libstdcxx-ng=11.1.0=h56837e0_8
  - libtiff=4.3.0=hf544144_1
  - libwebp-base=1.2.0=h7f98852_2
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - lxml=4.6.3=py39h107f48f_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - mako=1.1.4=pyh44b312d_0
  - markdown=3.3.4=pyhd8ed1ab_0
  - markupsafe=2.0.1=py39h3811e60_0
  - natsort=7.1.1=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - nikola=8.1.3=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1k=h7f98852_0
  - piexif=1.1.3=py_2
  - pillow=8.3.1=py39ha612740_0
  - pip=21.2.2=pyhd8ed1ab_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.9.0=pyhd8ed1ab_0
  - pyinotify=0.9.6=py39hf3d152e_1003
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyrss2gen=1.1=pyhd3deb0d_0
  - pysocks=1.7.1=py39hf3d152e_3
  - python=3.9.6=h49503c6_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - readline=8.1=h46c0cb4_0
  - requests=2.26.0=pyhd8ed1ab_0
  - setuptools=52.0.0=py39h06a4308_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.36.0=h9cd32fc_0
  - tk=8.6.10=hed695b0_1
  - tzdata=2021a=he74cb21_1
  - unidecode=1.2.0=pyhd8ed1ab_0
  - urllib3=1.26.6=pyhd8ed1ab_0
  - wheel=0.36.2=pyhd3deb0d_0
  - xz=5.2.5=h516909a_1
  - yapsy=1.12.2=py_0
  - zipp=3.5.0=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.5.0=ha95c52a_0

[Kwpolska]

It might be some other package causing that, or it might be caused by bad builds of one of those in conda repos.

[kai-tub]

For future reference:
The issue seems to be caused by the lxml package from conda-forge.
After installing via conda and then running:
pip install -U --force-reinstall lxml the issue seems resolved

[Kwpolska]

@kai-tub Could you report this as a bug to conda-forge?

[kai-tub]

Done 👍
I've also spent some more time to find the issue: It is caused by a newer libxml2 package from conda over my local version.
The bug appears starting with libxml2==2.9.11 and is fixed when installing libxml2==2.9.10 (at least in the conda environment)
So, I think this bug might appear more often in the future when/if the Linux distros move away from 2.9.10 to 2.9.11
But I did not check if the issue really appears when installing pip and linking to the newer libxml2 version.
Still, maybe something to keep in mind for the future :)
Hopefully, we can update the conda-recipe to ensure libxml2<=2.9.10

[ickc]

If true, this is an upstream bug that should not be reported to the conda-forge package, but to libxml2. 2.9.x should be critical bug fixing that is non-breaking.

[ickc]

Possibly you should report to https://github.com/conda-forge/libxml2-feedstock too so that they can block the problematic releases. If there exist such a bug report on libxml2 already, reference that. The maintainer(s) of libxml2-feedstock should decide(s) what they want to do about it.