Version 0.9.23 - Date: 2024-09-04

Version 0.9.23 - Date: 2024-09-04

This release continued our focus on enabling fixing for Rule Md031 and uncovering any issues with the more deeply nested container cases. And our luck held out, with the majority of the issues being related to the fixing algorithms. As mentioned in the last release, our detection rules rely on accurate parsing of the Markdown documents, with the only truthful way to verify that being to reconstitute the Markdown documents from our internal parsed format.

We continue to try different combinations of containers elements and leaf elements, verifying that PyMarkdown creates the correct HTML
and the correct Markdown from our parsed format. The good news is that the largest percentage of issues deal with how we represent and
reconstitute that whitespace. And we are diligently working to detect any issues with that process and to fix them.

That is where you, the users, come in. If you are scanning any Markdown documents and the results seem off, please file an issue. If you are starting to use our fix mode on your Markdown documents and there are issues, please file an issue. We appreciate any help that we can get to improve the project for everyone!

Added

• None

Fixed

• Issue 1141
  • fixed assert issue (test_extra_044mcv0)
• Issue 1142
  • fixed assert issue (test_extra_044lc)
• Issue 1143
  • fixed indent issue (test_extra_044ldb0)
• Issue 1144
  • fixed parsing issue (test_extra_044ldb1)
• Issue 1145
  • fixed indent issue (test_extra_044mx60)
• Issue 1146
  • fixed indent issue (test_extra_044lex1)
• Issue 1147
  • fixed indent issue (test_extra_044mcx)
• Issue 1148
  • fixed parsing issue (test_extra_044ldb1)
• Issue 1149
  • fixed parsing issue (test_extra_044mcz0)
• Issue 1150
  • fixed indent issue (test_extra_044lex3)
• Issue 1151
  • fixed assert issue with untested path (test_extra_044ldg)
• Issue 1152
  • fixed indent issue (test_extra_044mcs0)
• Issue 1153
  • fixed indent issue (test_extra_044mcu0)
• Issue 1154
  • fixed indent issue (test_extra_044mx31)
• Issue 1155
  • fixed indent issue (test_extra_044lde)
• Issue 1156
  • fixed indent issue (test_extra_044ldb0)

Changed

• None

Version 0.9.22 - Date: 2024-08-05

Version 0.9.22 - Date: 2024-08-05

This release was focused on enabling fixing for Rule Md031 and uncovering any issues with the more deeply nested container cases. The good news is that, as the list in the fixed section shows, we fixed a lot of issues. The better news is that only a handful of those fixes dealt with the parser, with the bulk of the issues dealing with transitioning from Markdown to our internal token format and back to Markdown again.

Why is this important? When a user asks the PyMarkdown linter to fix any issues that it can, our team wants to have the utmost confidence that PyMarkdown is producing the correct fix. Therefore, we tokenize the Markdown and base our rules off tokens that we know are correct. The only way to validate that we have the correct tokens is to take those tokens and recreate the Markdown. If we cannot produce the exact Markdown that we started with, then we have a problem.

In most of the fixed issues below, the tokens are correct and can produce the proper HTML from the Markdown. However, in over 90% of the fixed issues below, when we recreate the Markdown, the Markdown that we produce if off by a couple of whitespace characters. For
the reasons stated above, it is important to our team to fix these issues with transparency. Therefore, while the fixed list is somewhat
long, it is an honest reflection of the issues that we found and addressed.

Added

• Issue 818
  • Adding Fix Mode for Md031.

Fixed

• Issue 1120
  • within Block-List, thematic break can sometimes not report newlines to the
    list block
• Issue 1122
  • opening a fenced code block in a Bq-List-Bq was closing the outer BQ
• Issue 1123
  • in some cases within a Bq-List-Bq, not counting the newlines properly
• Issue 1124
  • list items within a Bq-List-Bq can have incorrect starting text regarding
    the innermost block
• Issue 1125
  • parsing of blank lines within Bq-List-Bq does not always add the right
    newlines to the list
• Issue 1126
  • under some circumstances, with a Bq-List-Bq, thematic break can cause
    the block quote to close
• Issue 1127
  • rehydration can be wrong with indented blocks in Bq-List-Bq
• Issue 1130
  • check for adding extra line to list with blank line in *-List-Bq
    not flexible enough
• Issue 1132
  • false positives (negatives?) for list looseness fixed
• Issue 1135
  • fixed issue introduced with above shortcuting in Bq-List-Bq scenarios
    to avoid assert
• Issue 1137
  • fixed issue with hanging indents and some Bq-List-Bq scenarios
• Issue 1141
  • fixed assert with Bq-List-Bq with previously untested branch
• Issue 1142
  • fixed assert with list-list-bq-bq with previously untested branch
• Issue 1143
  • fixed rehydate with first leading space not being calculated properly
• Issue 1144
  • fixed parsing error with bq-list-bq-list and HTML block not being recongized
• Issue 1145
  • fixed rehydration where last leading space of just closed block not being set
    properly
• Issue 1146
  • fixed parsing issue with text after whitespace not taking indent into account
• Issue 1147
  • fixed issue with double counting of spaces to list and paragraph
• Issue 1148
  • fixed parsing error with bq-list-bq-list and ATX block not being recongized
• Issue 1149
  • fixed parsing error with bq-list-bq-list and fenced block not being recongized
• Issue 1150
  • fixed hydration with thematic break after multiple lists and bq to render previous
    leading spaces as invalid
• Issue 1151
  • fixed assert with Bq-List-Bq with previously untested branch
• Issue 1152
  • fixed rehydrate problem with indents not being calculated properly for inner
    blocks
• Issue 1153
  • fixed rehydrate issue with sequences causing leading spaces to be incorrect
• Issue 1154
  • fixed rehydrate issue with sequences causing leading spaces to be incorrect
• Issue 1155
  • fixed rehydrate issue with prior and closed block quotes not being factored
    in properly
• Issue 1156
  • fixed rehydrate issue with extra block quote character being added at end
    of document

Changed

• None

Version 0.9.21 - Date: 2024-07-01

This release focuses on enabling the fix modes for various rules, performing more testing of scenarios to prepare for the release. Not finding anything major but uncovering some "weird" combinations that are causing unpredictable behavior. That behavior is mostly in the area of producing the correct Markdown from tokens to allow the fix mode to produce reliable fixes.

Added

• Issue 826
  • Added fix mode for Rule Md044
• Issue 824
  • Added fix mode for Rule Md046

Fixed

• Issue 1099
  • Fixed longstanding issue with tabs and newlines in code spans

Changed

• Issue 1103
  • added round of coalescing any text tokens separated during inline processing
  • rewrote rule md037 to new text tokens

v0.9.20

Version 0.9.20 - Date: 2024-05-30

This release focuses on completing the work to get the documentation up to date and in the new read-the-docs format. Some user issues were addressed, but this was mainly to get the documents into good shape for release.

To view the new documentation, go to ReadTheDocs.

Added

• Issue 1075
  • Complete redo of advanced extensions documentation.
• Issue 1079
  • Complete redo of advanced rules documentation.
• Issue 1083
  • Complete redo of api documentation.
• Issue 1081
  • Added "plugins info" extension to show current configuration.

Fixed

• Issue 1015
  • Fixed issue with double tabs within fenced block
• Issue 1077
  • Fixed issue with previous cleanup

Changed

• None

Version 0.9.19

This release focuses on getting the documentation up to date and in the new read-the-docs format. Some small issues were addressed, but this is mainly to get the documents into good shape for release.

To view the new documentation, go to ReadTheDocs.

Version 0.9.18

Version 0.9.18 - Date: 2024-03-18

This release focuses on getting the feature list complete for a version 1.0 release in the first half of 2024. This release was mainly fixes addressing the new "fix" functionality for a number of the rules, as well as issues with tab characters.

PLEASE NOTE. We have an extensive test suite, but we can always use your help with scenarios that we have not thought of. If you run into a situation that causes a bad parsing or other exception, please let us know.

A lot of the work we are doing to prepare for the new releases is centered around extra cases for tab characters as well as the newer fix functionality. Please report any issues with these features to help us make it better for others.

Added

• Issue 990
  • added ability to use a TOML file in pyproject.toml format with the
    --config command line flag

Fixed

• Issue 992
  • Verified behavior of Rule Md009, fixing some small issues
• Issue 994
  • Verified behavior of Rule Md029, adding configuration for starting ordered
    lists from integers greater than 1
• Issue 1001
  • Verified behavior of rules Md019 and Md021, fixing issues with Md021
• Issue 1003
  • Verified behavior of rule Md037 with more types of inline elements
• Issue 1007
  • Verified behavior of rule md023 with more cases, especially tab characters
  • multiple cases of fixing the rule and multiple cases of fixing the parser
• Issue 1015
  • Fixed issue with split tab and a simple list indent

Changed

• Issue 944
  • verified that all existing fix tests are parameterized
  • new fix tests for rule going forward will require this
• Issue 1005
  • verified behavior of rule Md039
  • added link reference defintion's link title to the previous list
    of link title and image's link title for examination
• Issue 1007
  • changed GHA workflow to be more precise when unable to start remote job

v0.9.17

Version 0.9.17 - Date: 2024-02-05

This release focuses on getting the feature list complete
for a version 1.0 release in early 2024. This release marked the
start of moving documentation from this repository to the more
curated ReadTheDocs.

Some notable additions/changes are:

• taking a second pass at the outputs from the recent fix addition, re-verifying
  the output and fixing any issues
• cleaning up documentation to properly note what type of whitespace is used
  in the core and well as various extensions and plugins
  • at the same time, clearly followed the specification on what kind of whitespace
    to use, instead of allowing unicode whitespace by default
• for parsers like Python-Markdown, used in the MkDocs tools, added Rule Pml101
  to handle the different indentation requirements
  • note that this new rule give advice against Md007, so only one of the two
    rules should be enabled at any one time

Added

• Issue 975
  • Added a new rule Pml101 to deal with "anchored list indents"
  • This adds support for Python-Markdown and other parsers like it
    • Used by tools such as mkdocs to build documentation sites
  • Defaults to an indent of 4 that starts from the beginning of the line or
    after a block quote start
  • Updated documentation for Md007 to mention Pml101 and when to use it
• Issue 983
  • Added base foundation for new documentation, publishing on
    read-the-docs.

Fixed

• Issue 929
  • second try, specifically missing github blob reference in urls
• Issue 945
  • inconsistent splitting of whitespace caused some issues
  • went through all strip() calls and ensured that they have the
    specific type of whitespace identified and documented
• Issue 964
  • final fix states needed verification and fixing of any issues
  • uncovered and fixed issues in Md007, Md019, and Md029
• Issue 977
  • fixed issue with md019 continuing to search for text blocks once the
    heading was completed
  • verified that Md021 does not have the same issue, but added tests to be sure
• Issue 981
  • added documentation for Rule Pml100
  • cleaned up mentions of whitespace in pragma and front matter extensions
• Issue 986
  • initial setting of log level to DEBUG with --stack-trace command line
    flag not working as expected
  • fixed to properly set log level to DEBUG for early application debugging

Changed

• Issue 966
  • adjusted fix for Md009 to remove any trailing whitespace if the line is within
    an Atx Heading

v0.9.16

Version 0.9.16 - Date: 2024-01-20

This release is going to focus on getting the feature list complete
for a version 1.0 release in early 2024. To a large extent, this
involves adding the "fix" feature for some rules, and double checking
the output of many of the existing rules, looking for missing issues.

Some other notable additions/changes are:

• the --continue-on-error command line flag allows PyMarkdown to
  continue processing after any tokenization error or plugin error
  • while we hope this is not necessary long term, it is useful
• added py.typed file for any API users
  • this allows mypy to understand the typing included with the
    PyMarkdown API
• more parameterized tests
  • borrowing a pattern we have observed, transitioning scenario tests
    over to this new pattern
  • any plugin with the new Fix feature has parameterized tests

Added

• Issue 618
  • Ability to tell PyMarkdown to fix issues
  • Not every plugin supports fix, see pymarkdown plugins list for
    the current list of plugins and fix status
• Issue 802
  • Extension: Extended Autolinks
• Issue 803
  • Extension: Strikethrough
• Issue 805
  • Extension: Task List Items
• Issue 808
  • Rule MD004 - Added fix options
• Issue 809
  • Rule MD007 - Added fix options
• Issie 813
  • Rule MD019 - Added fix options
• Issue 814
  • Rule MD021 - Added fix options
• Issue 816
  • Rule MD023 - Added fix options
• Issue 817
  • Rule MD029 - Added fix options
• Issue 820
  • Rule MD035 - Added fix options
• Issue 821
  • Rule MD037 - Added fix options
• Issue 822
  • Rule MD038 - Added fix options
• Issue 823
  • Rule MD039 - Added fix options
• Issue 825
  • Rule MD048 - Added fix options
• Issue 931
  • Rule MD005 - Added fix options
• Issue 938
  • Rule MD027 - Added fix options
• Issue 940
  • Rule MD006 (disabled) - Added fix options
• Issue 941
  • Rule MD030 - Added fix options
• Issue 946
  • Added --continue-on-error command line flag to "ignore" errors
    and to keep on processing.

Changed

• Issue 806
  • Documentation updated to denote fixes.
• Issue 812
  • Rule MD014 - Changed documentation to describe why not autofix
• Issue 827
  • Finished research on which rules are fixable and sorted.
• Issue 901
  • noticed cases where len(x) was being used instead of x or not x
• Issue 913
  • making proper use of is_xxx_end function from MarkdownToken class
• Issue 934
  • fix mode scans multiple times, with each scan producing lots of logs if
    on DEBUG
  • first fix was to allow a new command line option to suppress logs on any
    scan in fix mode past the first one
  • other fix was to make a clearer message when two rules trigger on the
    same field of the same token
• Issue 936
  • change documentation for rules that will not have a fix

Fixed

• Issue 929
  • Improper links in README.md when viewed at PyPi.org.
• Issue 930
  • Fixed issue of missing py.typed file.

Completed

• Issue 827
  • researched annotated each rule
  • rules "in queue" have no annotation yet, ones that have fixes have docs
    updated, ones that are not eligible have reason why

v0.9.15

Version 0.9.15 - Date: 2023-12-05

This release is mainly to fix issues related to technical debt. The PyMarkdown project takes Markdown and generates a token stream to represent that Markdown. To verify that the tokens are correct, HTML output is generated and matched against reference implementations of the specification. If those pass, the tests then try to recreate the Markdown from the information in the tokens. So, to pass a single test, the Markdown must generate tokens without any assertions, generate the correct HTML, and be able to recreate the Markdown that it parsed.

The issues fixed include some issues that fixed assertions, caused improperly formed HTML, and caused improperly formed Markdown. The majority of these issue involved tab characters and containers.

Added

• None

Changed

• None

Fixed

• Issue 731
  • see issue for details. fixed issues with unorder-bq-ordered that need more examination in the future
• Issue 828
  • list new items within block quotes were not always rendering properly in Markdown, required changes to markdown regen
• Issue 829
  • fixed issue with BlockQuoteData instance not being passed back properly. as a result, one closing of a block quote was not noticed by another closing further down the line
• Issue 832
  • partially fixed by other work, partial issue with the HTML output showing spaces instead of tabs, could be slightly different versions of commonmark tests
• Issue 833
  • not handling the split tab properly in these cases
• Issue 834
  • issues were tabs that were split on the same line as a list start
• Issue 835
  • a double block quote followed by a fenced block in a single block quote was not properly closing
• Issue 836
  • these cases were hitting split tab cases within processing for fenced code blocks
• Issue 837
  • these cases were split tab cases where the text to compare to its detabified forms was incorrect, resulting in a failed match
• Issue 838
  • extra tab was showing up in double list scenarios where indented block start was split over the last list and the indented block
• Issue 839
  • handling of split tabs within lists was not added at all, causing an assert to fire
• Issue 840
  • was not properly looking up in tabbed map when rendering
• Issue 841
  • fixed issue with assert
  • had commented out branch because no cases were found, finally found one
  • spawned other issues to fix less serious issues
• Issue 842
  • fixed problem with HTML and lists and split tabs causing assertions
  • spawned other issues to fix less serious issues
• Issue 843
  • whitespace check not being suspended for one check caused the html block not to be closed
• Issue 848
  • indent spacing within containers causing tab to not be changed back properly
• Issue 849
  • in cases with double lists, split tab can sometimes get missed
• Issue 850
  • some of the lines we adding whitespace for both the bq indent (already taken care of) and the list, resulting in too many spaces
  • tab support added to those cases
• Issue 852
  • fixed bad tokenization. previous fix was improper, causing strings to be improperly indexed into to fix spacing issue
  • spawned other issues to fix less serious issues
• Issue 854
  • "fixed". not sure why this happened, and will probably open another issue to properly figure this out at a later date
  • in cases where a list is within a block quote, and the next line is a paragraph continuation that fails the requirements for a "normal" list continuation, this fix was required.
• Issue 857
  • fixed along with Issue 840, just filed separately
• Issue 878
  • a doubly indented list with a new paragraph continuation line starting with multiple tabs was not capturing the first tab properly in the list
• Issue 888
  • during fixing of 731, found some outside cases which were throwing asserts as they were outside of normal paths
  • mostly dealt with proper spacing with block quotes nested within lists.
• Issue 889
  • cleaned up issue with MD032 firing with nested blocks.
• Issue 891
  • fixed up Markdown issues with regenerating. Cause was improper adding of an extra newline in the leading spaces

v0.9.14

Version 0.9.14 - Date: 2023-10-31

This release is mostly to incorporate a number of small fixes and additions.
Behind the scenes, we spent a while looking at the roadmap and trying to figure
out the best path going forward. Part of that was experimenting with different
forms of testing, to see how applicable they would be to this project. That
work will be starting soon and be incremental. It also pointed out a need for
better documentation, which will also be incremental going forward.

That experimentation lead to a couple of changes:

• Front matter is now interpretted as strict YAML, and not as some mashup of rules.
  This was decided on as it easier to explain and document. This may cause some
  existing front-matter settings to not work as expected, especially validation
  of the front-matter itself and case-sensitivity on front-matter key fields.
• Added a new pragma command disable-num-lines to handle disabling rules for
  a given count of lines.

Added

• Issue 776
  • implement disable-num-lines pragma command
• Issue 786
  • added extension to disallow html, per GFM
  • added rule PML100 which detected the same thing, but does not fix

Changed

• Issue 750
  • started work on making the tests more compact and efficient

Fixed

• Issue 774
  • was following "weird" rules, changed to allow for standard YAML parser to
    determine validity
  • changed rule md001 to look for case sensitive front-matter key name instead
    of case sensitive per switch to full YAML processing
• Issue 791
  • html blocks inside of 2+ levels of block quote was asserting
  • slight change to surrounding code to deal with less than case
• Issue 793
  • list block end with no paragraphs was causing an assert with MD027
  • double checked assert, is not needed and was just preventative