File as test instead of cell

[RomeshA]

Would it be possible to support treating entire files as a single test, rather than individual cells? We have a decent amount of documentation in the form of notebooks that get tested during our CI workflow, but there are on the order of 1000 cells across about 20 notebook files, vs. about 100 actual Python test functions. Since the notebooks are structured as documents, workflows are often split across cells separated by narrative text, so it doesn't make so much sense for us to treat each individual cell as a separate test. We are currently using https://github.com/pyviz-dev/nbsmoke which does treat each file as a single test, but that module is now broken for the same reason as #139 and it looks like that project is recommending to switch to nbval

[RomeshA]

Thanks @alex-treebeard I haven't seen treebeard before and it looks like a useful project! For our use case though, we already have CI setup in Azure Pipelines that handles our other pytest tests, and also documentation generation via Sphinx, and deployment both of the documentation and to PyPI. So my inclination would be to try and integrate into our existing pipeline if possible, whereas treebeard seems to be a more complete solution package if we weren't using pytest and needed to set up a new CI pipeline.

I think this is also related to some of @ceball's thoughts e.g. #141 (comment) regarding the use case 'Does the notebook execute all the way through without exception?' which is in fact exactly our use case

[vidartf]

Just so we are on the same page, what is the main motivation for wanting it as a single test? Is it purely a "looks better" case, or are there negative consequences?

[sk1p]

are there negative consequences?

One consequence of having one test per cell is that pytest --ff doesn't work properly - it tries to execute the cells out-of-order. Other than that, I like the per-cell version, too.