Project Summarizer - Designing and Implementing the Summarizer

Summary¶

In my last article, I talked about the reasoning behind one of the tools I use: Project Summarizer. In this article, I talk more about the reasoning and designs behind the tool, and the recent improvements I have made to it.

Introduction¶

Sometimes you can use a screwdriver when you need a hammer, and sometimes you can use a hammer when you need a screwdriver. But more often than not, you really need to have the right tool for the job you are doing. If nothing else, you want to make sure that you know you did things right and did not take any short cuts.

Once I had identified a set of requirements and could not find a tool that was meeting those requirements, I knew I needed to do something. The result of meeting those requirements was the Project Summarizer tool.

What Is the Audience for This Article?¶

While detailed more eloquently in this article, my goal for this technical article is to focus on the reasoning behind my solutions, rather than the solutions themselves. For a full record of the solutions presented in this article, please consult the commits that occurred between 15 Feb 2022 and 20 Feb 2022.

Some Quick Design Work¶

As I mentioned in my last article, I had a set of requirements that were not being fulfilled by other tools. Namely, those requirements were:

• to summarize the analyses of other tools
• to give me immediate feedback on how my changes affect code metrics

While these were not very extensive requirements, they were the guiding principles that drove the initial development of the Project Summarizer tool.

From the beginning, the way I envisioned this tool as something that was fast and took information from other tools to summarize. The first use case was for my own Python projects, so meeting those requirements for my Python projects was of course my first goal. To carry out that goal, the new tool needed to fit into the scripts that I use to run tests on my projects. Added as part of the test scripts, I knew that I had two output files at my disposal: the XML coverage file from coverage.py and the XML tests file from pytest. Using those files, I knew I could provide a summary of any changes in tests or test status, and I could provide a summary of high-level coverage numbers.

Is It Enough To Help Me?¶

Would that buy me enough benefit to make it worth the cost? Yes! At the time, once I finished making changes, the four questions that I always found myself asking were:

• did I make the right fix?
• how did it affect the tests that I already had in place?
• did the fix require the addition of new tests to cover the new code in the fix?
• did the fix create code that is no longer needed, and is safe to be deleted?

The first question is one that I seriously believe can only be answered by thorough testing, experience with the code, and time. The best answer that I can usually hope for to that question is “yes, for now”. As time goes on, that answer will either lean towards or away from that “yes”. It is often hard to tell in isolation.

However, I felt that the remaining three questions could easily be answered by data. How did I know that? Because I had manual tasks that I performed to answer those questions. For the first question, I looked at the PyTest output to see if anything failed. If something failed, I look at the PyTest HTML files to see what tests failed and why. For the second and third questions, I went right to the HTML coverage output, and followed a similar procedure. And I figured that if I could do it manually, I could automate it.

If I could automate that process, it would be good enough for me.

Getting Off The Ground¶

The initial version of Project Summarizer (originally called PyScan) was quickly put together, meeting these requirements head on. Using the XML files as input, I quickly dug the required statistics out of those files. For the test output files, I only needed to know if a test was added, removed, or changed status. For the coverage output files, I only needed to know if the various coverage metrics changed.

At that point, I realized that I needed to introduce a third requirement:

• to create a “checkpoint” that could be used to compare current values against.

As I looked at the raw summary numbers from the file, the one thing that they were missing was context. To be blunt, I found it difficult to remember if I had 128 tests or 129 tests before I made the change to add a new test. I did not care most of the time. What I cared about was whether the number of tests increased by one to match the one test that I added. This pattern carried over into the coverage requirements. I only cared about the change in coverage, not the current coverage number.

But to generate a difference value, you need to have a value to compare the current value against. That was missing at that point, so I created it. Thus, the publish directory was created and the --publish flag added to the Project Summarizer. By publishing the existing summary to the publish directory, any current change was able to be measured against that checkpoint. I did have to establish a habit of running my clean test script with the -p option to publish the results when I closed out a fix, but that habit was not too bad to form.

And things were coming together nicely. I was able to update either source code or test code, and a summary of those changes was being reported to the command line. After a couple of weeks, I invested some time into learning about the Columnar package and used it to better organize the output. And it was not fancy, but it worked and worked well:

Test Results Summary
--------------------


  CLASS NAME                               TOTAL TESTS  FAILED TESTS  SKIPPED TESTS

  test.nested_three.test_markdown_nested_     133             1 (+1)         8 (-1)
  three_block_ordered
  test.test_markdown_extra                     87 (+7)        2 (+2)         1
  ---                                        ----             -            ---
  TOTALS                                     4528 (+7)        3 (+3)       171 (-1)

Taken from some work that I did during the past week, this clearly showed me what was going on with my changes. As this was a focused execution of tests, I had the coverage summary disabled. But the information provided by the test results was particularly useful. I had added seven tests and removed the @pytest.mark.skip from one test, which is reflected in the output. And as I was working on resolving issues, I had five of the tests taken care of, with three failed tests remaining. That was exactly the information that I needed to know I was going in the right direction.

Time Passes… And The Tool Matures¶

I continued to use the Project Summarizer tool for about a year before I started to think about making changes to it. And even then, it took me having a couple of issues calling it from another script to get me to realize that I needed to make a couple of changes. And those changes were not substantial changes, but simple usability changes.

I had one issue when calling Project Summarizer from a script, where it had a challenging time figuring out a consistent answer to the question “what is the width of the screen”. As Columnar bases its organization of the columns on the number of character columns on the display, it raised an interesting question of what to do if there was no actual terminal. There are defaults built into Columnar, but they did not seem to be consistent. And if I wanted predictable behavior, I needed consistency.

Therefore, the --columns argument was added. This would override any calculated values and force the Columnar package to a specific size. At the same time, I figured that there were cases where that output was not important. As such, I also added a --quiet argument to suppress the columnized output.

The second issue that I had was that the reporting and publishing directories were both hardcoded. That was easily fixed with the --report-dir and --publish-dir arguments. While I do not predict changing these directories in my own setup, it did sound like a solid addition to the tool.

Why Now?¶

I guess the question that I left unanswered was: why did I feel that I needed to work on this now?

The simple answer was that I found other things to measure. One of the things that I currently measure is the number of PyLint warnings in the current project. But that information is hard to see, and I would like to have better visibility into how those values have changed on an ongoing basis. But when I thought about how to design that tool to include summary information, I decided instead that it would be better for the Project Summarizer to do that reporting for me.

But that meant I needed to update the Project Summarizer to manage more information. As I started to do that design, it just seemed like the right thing to get those changes I had in mind out of my head and into the project. The good news is that even after I got those changes out of the way, it still felt like a good idea to upgrade the Project Summarizer tool to be extendible.

In fact, I seriously think it is a great idea!

What is Next?¶

This week, I plan to do more research and design into making the Project Summarizer tool more extensible. Hopefully, that means I will have something interesting to report next week. Stay tuned!

Like this post? Share on: Twitter ❄ Facebook ❄ Email