In my last article, I started to add the proper support for line and column numbers for both the text tokens and the emphasis tokens. In this article, I increase my confidence in the line and column numbers for those two inline tokens by adding the consistency checks for those tokens.


I know that I am fallible. It therefore stands to reason that any code that I write will have some issues with it. Those issues may be obvious issues, or they may be issues that only occur under a bizarre set of circumstances, but they are there. Rather than fight against them, I embrace the attitude that good test automation will help me to identify those types of issues as early as possible.

For the PyMarkdown project, this test automation takes the form of scenario tests containing consistency checks. These consistency checks validate that the Markdown documents in the scenario tests are properly interpreted by the PyMarkdown project. But while these consistency checks are beneficial, the consistency checks have taken a long while to complete. After 3 calendar months have passed, it can easily be said that my decision to add consistency checks to the project removed 3 months of project development time and replaced it with 3 months of project test time. Plain and simple, those statements are facts.

My confidence about the project and its ability to work correctly is an emotional and abstract statement. However, with effort, I have been able to move it in the direction of being more of a fact than a feeling. The consistency checks are a form of test automation that apply a generalized set of rules over a group of tokens, looking for each group to behave in a predictable manner. Before this work, my confidence was expressed as a feeling: “I believe the project is stable”. With this work nearing its completion, I can now point to the scenario tests and consistency checks that run within those scenario tests. I can state that each of the scenario tests is passing a rigorous set of criteria before it is marked as passing. That confidence can now be expressed as: “Here are the tests that are passing and the checks that are being performed on each test.”

From that point of view, it made sense that before I start working on setting the line/column numbers for the remaining inline tokens that I would implement the consistency checks for the Text token and the Emphasis tokens.

What Is the Audience for This Article?

While detailed more eloquently in this article, my goal for this technical article is to focus on the reasoning behind my solutions, rather that the solutions themselves. For a full record of the solutions presented in this article, please go to this project’s GitHub repository and consult the commit of 02 Sep 2020.

Getting Started With Inline Token Validation

At the start of the week, the code used to verify the consistency of inline tokens was extremely simple:

    print(">>last_token:" + ParserHelper.make_value_visible(last_token))
    next_token_index = last_token_index + 1
    while actual_tokens[next_token_index] != current_token:
            "-token:" + ParserHelper.make_value_visible(actual_tokens[next_token_index])
        next_token_index += 1

Added in as a placeholder to allow me to see what was going on with the inline tokens, it served its purpose well. But as I started to work on the inline tokens and their line/column numbers, I needed to facilitate better consistency checking of those inline tokens.

To start the work off, I removed that placeholder code from two places in the code and replaced both with a call to a new function verify_inline. The only difference between the two invocations of the function were the fourth argument, current_token. Called for the first time from the___verify_token_height function, the current_token variable is set to the block token after a series of inline tokens. The second time it is called, it is called at the end of processing to capture any inline tokens that are within one of the valid text elements, but at the very end of the document. When it is invoked from that location, that same argument is set to None. In both cases, the inline tokens to be validated were clearly outlined for the verify_inline function.

Clearly Defining the Problem

Before doing any real processing with the inline tokens, I needed to create a simple list containing the actual inline tokens that I wanted to check. I could have done that with the main list of tokens and the previously document outlining. However, I thought about it and decided that it was clearer to have a separate list that just contained the tokens that I was concerned about. Once I had all the inline tokens between the two block tokens in that new list, there was a small amount of work to do before the list was usable. While it was not difficult, the new list had some extra end tokens at the end of the list that needed to be removed. Working around those extra end tokens would have been okay, but I just felt that it was simpler to remove them from the list before I did any further processing.

Having a simple list of the inline tokens to work with, the first iteration of the checking algorithm started with an easy outline to follow:

    if inline_tokens:
        for token_index, current_inline_token in enumerate(inline_tokens):
            if not token_index:
                __verify_first_inline(last_token, inline_tokens[0])
                    inline_tokens[token_index - 1],

        # verify_last_inline(inline_tokens[-1], current_inline_token)

From my viewpoint, the processing of the inline tokens had 3 distinct phases: the first element in that list, each element after it, and the last element in that list. Based on their locations, the first and last elements are special in that they anchor the other inline tokens to the block tokens on either side of the middle elements. Without those anchors, the middle elements lack a foundation with which they can based their positions on.

Based on those observations, I chose to implement the check for the first inline token against the previous block token, and not the check for the last inline token against the following block token. Without validating the first element, validating any of the elements on the inside of the list would be useless. So, whether I liked the idea or not, validation of the first element in the list was mandatory. The last element is a different story. While it would be nice to tie the last inline token to the following block token, I felt that it was not as important as the verification of the first element. However, I added in a placeholder to the code to make sure that I would follow up on it later.

Validating the First Element

Following the pattern that I have used for validation in the past, I created the __verify_first_inline function with my standard starting template:

def __verify_first_inline(last_non_inline_token, first_inline_token):
    if <something>:
        assert False, last_non_inline_token.token_name

As this function is comparing the starting position of the first inline token to the last valid block token, the <something> in the above code sample was quickly replaced with:

    if last_non_inline_token.token_name == MarkdownToken.token_atx_heading:
        assert False
    elif last_non_inline_token.token_name == MarkdownToken.token_setext_heading:
        assert False
    elif last_non_inline_token.token_name == MarkdownToken.token_paragraph:
        assert False
    elif last_non_inline_token.token_name == MarkdownToken.token_fenced_code_block:
        assert False
    elif last_non_inline_token.token_name == MarkdownToken.token_indented_code_block:
        assert False
    elif last_non_inline_token.token_name == MarkdownToken.token_html_block:
        assert False
        assert False, last_non_inline_token.token_name

and one by one I added the validation functions to replace the assert False statements. Following that same pattern for resolving these as I have before, I ran the scenario tests over the entire project using the command line:

pipenv run pytest -m gfm

Each time, I just picked one of the failing tests, and worked on that tests in that group until they were all passing. For each validation function, I repeated the same pattern with the first inline token that was observed. For example, the __verify_first_inline_atx function quickly evolved to look like:

def __verify_first_inline_atx(last_non_inline_token, first_inline_token):
    Handle the case where the last non-inline token is an Atx Heading token.

    col_pos = last_non_inline_token.column_number + last_non_inline_token.hash_count

    if first_inline_token.token_name == MarkdownToken.token_text:
        replaced_extracted_whitespace = ParserHelper.resolve_replacement_markers_from_text(
        col_pos += len(replaced_extracted_whitespace)
        assert first_inline_token.line_number == last_non_inline_token.line_number
        assert first_inline_token.column_number == col_pos
    elif first_inline_token.token_name == MarkdownToken.token_inline_hard_break:
        assert False

        assert (
            first_inline_token.token_name != MarkdownToken.token_inline_link
            and first_inline_token.token_name
            != EndMarkdownToken.type_name_prefix + MarkdownToken.token_inline_link
        ), first_inline_token.token_name

What Did I Discover?

Predictably, I discovered that there are 2 groups of text within block tokens: ones that support inline tokens other than the Text token, and ones that do not. The ones that do not support inline tokens are mostly easy: assert that the inline token is a Text token, and then assert on a simple calculation of the first line/column number. The validation of the HTML Block token and the Indented Code Block token both followed this pattern, with very simple validation.

def __verify_first_inline_html_block(last_non_inline_token, first_inline_token):
    assert first_inline_token.token_name == MarkdownToken.token_text
    leading_whitespace_count = len(first_inline_token.extracted_whitespace)
    assert last_non_inline_token.line_number == first_inline_token.line_number
    assert (
        last_non_inline_token.column_number + leading_whitespace_count
        == first_inline_token.column_number

The Fenced Code Block tokens required a bit more effort, but not much. As the Fenced Code Blocks can start with 0-3 space characters that then need to be managed on any subsequent line in the code block, the owning block token’s leading_spaces variable holds the information on what leading spaces were already removed. As such, when calculating the proper position of the first Text token inside of a Fenced Code Block, that removed space needs to be accounted for. To properly facilitate that, the last_token_stack argument needed to be plumbed through so the verification function could calculate the proper owning blocking token.

The second group of block tokens were the more interesting group of tokens to deal with. This group of tokens includes the Atx Heading tokens (as shown in the above example), SetExt Heading tokens, and Paragraph tokens. The __verify_first_inline_atx function and the __verify_first_inline_setext function ended up looking similar: the Text inline token case was populated, but all the other types of inline tokens were handled with assert False statements. The __verify_first_inline_paragraph function was similar, but also slightly different. The same template was used to generate the function, but each of the conditions in the if-elif-else block were met at least once. However, since only the Text token and the Emphasis token have line/column numbers, allowing this comparison to be performed for them:

        assert first_inline_token.line_number == last_non_inline_token.line_number
        assert first_inline_token.column_number == last_non_inline_token.column_number

All the other inline tokens, the ones that did not currently have a line/column assigned to them (yet), used the following comparison:

        assert first_inline_token.line_number == 0
        assert first_inline_token.column_number == 0

It was not much, but it gave me two important bits of information. The first was that there was at least one case where each available inline token was the first inline token inside of a Paragraph token. The second was that both heading tokens, the Atx Heading token and the SetExt Heading token, only contained scenario tests that started with Text tokens. I made a note of that observation in the issue’s list and moved on.

Verifying the Middle Tokens

With the validation of the first element out of the way, it was time to start working on the __verify_next_inline function. Now that the middle tokens were anchored at the beginning, each of the middle inline tokens could be validated against the inline token that preceded it. Since I knew that most of the inline tokens had not been handled yet, I started out that function with a slight change to the template:

def __verify_next_inline(
    last_token, pre_previous_inline_token, previous_inline_token, current_inline_token
    if (
        previous_inline_token.line_number == 0
        and previous_inline_token.column_number == 0
    if (
        current_inline_token.line_number == 0
        and current_inline_token.column_number == 0

    estimated_line_number = previous_inline_token.line_number
    estiated_column_number = previous_inline_token.column_number
    if previous_inline_token.token_name == MarkdownToken.token_text:
        assert False
        assert False, previous_inline_token.token_name

    assert estimated_line_number == current_inline_token.line_number, (
        + str(estimated_line_number)
        + ">act>"
        + str(current_inline_token.line_number)
    assert estiated_column_number == current_inline_token.column_number, (
        + str(estiated_column_number)
        + ">act>"
        + str(current_inline_token.column_number)

The first set of if statements made sure that if either the previous inline token or the current inline token was one that I had not worked on yet, it would return right away. While this assumed that the line/column numbers were correct to a certain extent, I was okay with that assumption in the short term. The second part computed a starting point for the new line/column numbers, and then went into the usual pattern of dealing with each of the eligible tokens by name. Finally, the third part compared the modified line/column numbers against the actual line/column numbers of the current token, asserting with meaningful information if there were any issues.

Emphasis Tokens

I thought it would be quick to get emphasis out of the way, and it was! As both the start and end Emphasis tokens contain the emphasis_length, it was a quick matter of adjusting the column number by that amount. As both tokens are confined to a single line, there was no adjusting of the line number to worry about.

Text Tokens

As mentioned in a previous section, there are two major groups of block tokens that contain Text tokens: ones that allow all inline tokens and ones that do not allow inline tokens except for the Text token. The ones that do not allow inline tokens are simple, as all the information about the Text token is contained within the token itself. It is the other group that are interesting to deal with.

The easy part of dealing with the Text token is determining the new line number. With the exception of a Text token that occurs right after a Hard Line Break token, the calculation is simple: split the text by the newline character, subtract 1, and that is the number of newlines in the Text token. If the token before the Text token was a Hard Line Break token, it already increased the line number, but the Text token that followed also started with a newline character. To remedy this, that pattern is looked for, and the current_line variable adjusted to remove the newline character at the start of the line.1

Determining the column number is a more interesting task to undertake. For any Text tokens occurring within a block that does not allow for extra inline tokens, the column number information is already in the token itself, and the calculation is as simple. The column delta is equal to the number of text characters stored within the token2. If there was a newline in the token’s text, this count is started after the last newline character.

The second group of block tokens that can contain text are the Atx Heading token, the SetExt Heading token, and the Paragraph token. Since the Atx Heading token can only contain a single line’s worth of data, no extra calculations are required to handle multiple line scenarios. In the case of the other Heading token, the SetExt Heading token, the starting whitespace is stored in the Text token’s end_whitespace field. The processing of this information is a bit tricky in that the starting and ending whitespace for the Text tokens within the SetExt Heading token is stored in that field using the \x02 character as a separator. Still, determining the proper indent and applying it to the column number is relatively simple.

Dealing with a Text token within a Paragraph token is a lot more work. Due to other design reasons, the whitespace indent for these Text tokens is stored within the owning Paragraph token. While that is not difficult by itself, keeping track of which indent goes with which line is a bit of a chore. Luckily, when I was working on the Markdown transformer, I introduced a variable rehydrate_index to the Text token. When rehydrating the Text token, I used this variable to keep track of which stripped indent needed to be added back to which line of any subsequent Text tokens. Given the prefix whitespace for any line within the Paragraph block, calculating the column number delta was easy.

Blank Line Tokens

That left the Blank Line tokens to deal with, and I hoped that the effort needed to complete them was more in line with the Emphasis tokens than the Text tokens. I was lucky, and the Blank Line tokens were easy, but with a couple of small twists. Intrinsically, a blank line increases the line number and resets the column number to 1. That was the easy part. The first twist is that if the current token is a Text token, that text token can provide leading whitespace that needs to be considered. That was easily dealt with by adding the following lines to the handler:

    if current_inline_token.token_name == MarkdownToken.token_text:
            estiated_column_number += len(current_inline_token.extracted_whitespace)

The more difficult problem occurred when 2 blank line tokens appear one after the other within a Fenced Code Block token. Because of how the numbers added up, I needed to adjust the estimated_line_number variable by one.

    if current_inline_token.token_name == MarkdownToken.token_blank_line:
        if previous_inline_token.token_name != MarkdownToken.token_blank_line:
            estimated_line_number += 1
        estiated_column_number = 1

With that tweak being done, all the tests were then passing, and it was time to wrap it up.

Was It Worth It?

The interesting part about defensive code is that sometimes you are not aware of how good that defense is. Using the analogy of a castle, is a castle better defensible if it can withstand attack or if it deters others from attacking the castle? While I did not have any information about potential attacks that were stopped ahead of time, there were 2 actual issues that the current round of consistency checks did find.

The first of those issues was an issue with the column number for example 600 as follows:


[foo]: /url "title"

Before these inline consistency checks were added, the text for the ] character was reported as (1,6). By simply counting the characters, the ! character starts at position 1 and the second o character is at position 6. As such, the ] character should be reported as (1,7).

Doing some research, I concluded that the handling of a properly initiated Image token was being handled properly. However, with a failed Image token sequence, the ! character followed by any other character than the [ character, the ! character was being emitted, but the column number’s delta wasn’t being set. Adding the line

    inline_response.delta_column_number = 1

at the end of the __handle_inline_image_link_start_character function solved that issue.

Issue 2: A Simple Adjustment

The second of those issues was more of a nitpick that an actual issue. In the tokenization for example 183:

# [Foo]
[foo]: /url
> bar

the first line was tokenized as:

        "[text(1,3):\a \a\x03\a:]",
        "[text(1,4):Foo: ]",

Having a lot of experience sight reading serializations for all the tokens, the information in the Text token leapt out at me right away. In that token, the extra data associated with the token is composed by adding the self.token_text field, the : character, and the self.extracted_whitespace. Based on the above tokenization, that meant that the text sequence \a \a\x03\a was being considered as text instead of whitespace.

To understand why I thought this is wrong requires an understanding of the existence of that character sequence. The \a sequence is used to denote that a sequence of characters in the original Markdown document was interpreted and replaced with another sequence of characters. The \x03 character within the second half of that sequence means that the {space} character in the first part of the sequence is being replaced with the empty string. Basically, to properly represent the space between the # character denoting the Atx Heading element and the [ that starts the Link element, I needed to add a space character that would not appear in any HTML transformation.

And here is where the nitpicking comes in. When I originally added that sequence when working on the Markdown transformer, it made sense to me to assign it to the token’s self.text_token field. But since then, I have grown to think of that sequence as being more extracted whitespace than token text. To resolve that, I decided to move the call to generate the replacement text from the self.token_text field to the self.extracted_whitespace field. It wasn’t a big move, but it was something that I thought was the right thing to do.

What Was My Experience So Far?

While this batch of work wasn’t as laborious as last week’s work, the effort required to make sure it was correct was equal to or exceeding last week’s work. I knew that if I made any mistakes last week, they would be caught when I implemented the consistency checks. Well, these were the consistency checks that would capture any such issues that slipped through.

I am both happy and proud that I am coming to the end of implementing the consistency checks. It has been a long 3 month voyage since I decided that consistency checks were the best way to ensure that the quality that I wanted in the PyMarkdown project was maintained. And while there were times that I questioned if I made the right decision in dedicating this large block of time to this aspect of the project, I was confident that I had made the right decision.

But looking ahead to what I needed to do after the consistency checks, I saw a fair number of items in the issues list that would need researching and possibly fixing. While I could start to release the project without them, I didn’t feel comfortable doing that. I wanted to give the project the best chance it could to make a first impression, and then move from there. And that would mean more work up front. So while I was happy that the consistency check work was coming to an end, there seemed to be a deep pool of issues that would need to be research… and I wasn’t sure how much I was looking forward to that.

I still believe that adding the consistency checks was the right move. Of that I am still certain. Instead of a feeling that I have the right code in place to do the Markdown transformations, I have hard, solid checks that verify the results of each and every scenario test. It also gave me the interesting bit of information that the scenario tests did not include any cases where the Atx Heading token and the SetExt Heading token were followed by anything other than a Text token. Something interesting to follow up on later.

To me, adding more of those checks for the inline tokens was just another solid step forward in quality.

What is Next?

Having completed the hardest inline token (Text token) and the easiest inline tokens (Emphasis tokens), it was time to buckle down and get the remaining tokens done. If I was lucky, the foundational work that I had already completed would make completing those tokens easy. If I was unlucky, there would be a whole selection of edge cases that I needed to account for. Realistically, I was expecting something square in the middle between those two scenarios. The next batch worth of work would answer that question!

  1. This has been noted in the issue’s list, and I am hoping to look at it soon. 

  2. That is, after removing any special characters and leaving the original text used to create those special characters. 

Like this post? Share on: TwitterFacebookEmail


So what do you think? Did I miss something? Is any part unclear? Leave your comments below.

Reading Time

~16 min read


Markdown Linter Core


Software Quality


Stay in Touch