Summary¶
In my last article, I talked about starting to add proper plugin support to the Project Summarizer project. In this article, I take that work to its logical conclusion by adding proper plugin support to the project.
Introduction¶
It took me a while to figure out the right way to add plugins for the Project Summarizer project. But when I did my research and worked through everything, I ended up with a solid loading strategy that I knew worked. What was left? To implement the plugins properly, of course!
Implementing The Plugins¶
It is a simple phrase: “to implement the plugins properly.” But it was not going to be easy. I had experience implementing plugins, but I was not sure how much of that experience would carry over. I knew it was going to be enough to implement the base part of the plugins, that much was certain. But since it is a completely different project, I was not sure if I was going to meet something I hadn’t encountered before.
Refactoring Plugin Related Code¶
From experience with the PyMarkdown project, there were two things that I was
sure that I needed to do if I wanted to be successful with these refactorings:
move the plugins related code under a new plugins directory and create a new
plugin_manager directory to contain the management of those plugins. By taking
the step of creating these two directories, I knew that I was cleanly defining
the parts of the project that were related to plugins and to management of those
plugins.
The easy part was moving the existing code for the Cobertura plugin and the Junit plugin into those directories. A couple of quick name changes here along with moving their related measurement classes into the same directory, and those were done. The hard part was the management of those plugins.
The first thing that I created was a new class BadPluginError. Copying heavily
from the PyMarkdown project, I knew that I needed to have a singular exception that
I could throw whenever something bad happened. Instead of different exceptions
for different issues, I decided on keeping the “one exception to rule them all”
approach. While I might experiment with subclasses of this exception in the future,
having all the exception handling in one exception makes it easy to deal with.
The only part of the exception that is difficult is the formatting of the message,
but with a bit of refactoring, it will not be too bad.
I then created the new PluginManager class and started moving code that was
plugin related in the main.py module over to the new class. It was a bit
scary at first, as I am not used to seeing VSCode present me with so many code
errors at one time. But as I added the required import statements and moved
the functions over, those code errors slowly started disappearing. I knew in my head that
those errors were only temporary, but it sure was a relief to see those errors
go away.
But after everything was moved over, there still was cleanup to complete.
There were a couple of places where a sys.exit(1) was used instead of throwing
the new BadPluginError exception, so that needed to be addressed. Along with
that, I added try/except(BadPluginError) blocks around the sensitive code blocks,
to ensure that those thrown exceptions were properly handled. To centralize that
in one location, I added a __report_error function to the main.py module
specifically to ensure that there was a single location for reporting errors.
And that work continued for a bit longer. To make sure that every call into a
plugin was properly protected, I added more try/except blocks around each call
into a plugin. While it is not always a good practice, I used an except Exception
to capture any exception, creating a new BadPluginError from that exception.
After all that work was done, I knew that I could start breathing again. The hard part of the work was done.
Aside on General Exceptions Catching¶
Generally, the problem with catching a general Exception is that an overly broad
caught exception can catch more serious issues than the desired exceptions.
If developers are not careful, serious exceptions like the OutOfMemoryException
and the DivideByZeroException can be caught and ignored when the proper action
should have been taken to deal with those types of serious issues more… well,
seriously. And usually, I try and avoid catching those Exceptions. It just
does not look right to me.
But in this instance, there were two things working for me catching Exception.
One instance in which I believe it is okay to catch general exceptions is with a large enough
change of responsibility within the code being executed. Consider the following
code from the add_command_line_arguments_for_plugins function:
try:
(
plugin_argument_name,
plugin_variable_name,
) = next_plugin_instance.add_command_line_arguments(parser)
except Exception as this_exception:
raise BadPluginError(
class_name=type(next_plugin_instance).__name__,
formatted_message="Bad Plugin Error calling add_command_line_arguments.",
) from this_exception
From a puritanical point of view, this is a terrible thing to do. However, I believe
that from a realistic point of view, it is the correct thing to do. When the
function next_plugin_instance.add_command_line_arguments is executed, the
responsibility for the executing code changes from the Project Summarizer project
and its PluginManager class to that of the plugin itself. Bluntly said, there
is nothing that the Project Summarize project can do to prevent the plugin from
executing any code that it wants to. It is the responsibility of the plugin to
adhere to any provided interface as closely as possible. Even so, there is no uncomplicated
way to define which exceptions can be raised by the plugin, and therefore which
exceptions to protect against with an except block. Hence, except Exception.
That brings me to the second thing work for me in this code: the handling of the
BadPluginError itself. I did a quick check through the code as I was drafting
this article, and I could not find a single instance where the raising of the
BadPluginError was not handled by cleaning stopping the application as soon as
possible. This means that every time that exception is raised, the application
ends. With very few exceptions (no pun intended), when one of those serious
exceptions are raised, the best response is to terminate the application itself.
As the Project Summarizer takes those serious exceptions and wraps them in a BadPluginError
to supply additional context before rethrowing the caught exception, I believe I have a good argument that
those exceptions are being handled in an appropriate manner.
Little Things¶
With the challenging work of the refactoring done, I knew I needed to focus on the little things that needed to be addressed. Basically, dotting each i and crossing each t.
The first of those things was to stop initializing the self.__available_plugins
array with the two built-in summarizer plugins and load them dynamically with the
other plugins. While it would have been fine to load them the other way, I just
felt it was more consistent to load them this way. This meant that all plugins
were being loaded the same way, which just made sense.
As I was looking through the code, I noticed that there were optimizations
that could be performed on the code. In earlier iterations, I had files like
the test_results_model.py file that contained the TestTotals class and
the TestMeasurement class. Instead of artificially grouping them together,
I decided to split them up into their own files. It just made sense as they
were different concepts. One bonus to that was that it made the typing of
those classes easier, as they were in separate files now.
And then it hit me, I did not have details support.
Adding Details Support¶
One thing that I learned from the PyMarkdown project was that it was exceptionally useful to have explicit command line arguments that would detail which plugins were present and enabled. While the plugins for the Project Summarizer project are more directly visible on the command line, I thought about whether to include this kind of support for a couple of days. In the end, the cost is low and the benefit to the user is decent enough, that the benefit outweighed the cost.
With that decided on, I added a simplified version of the PluginsDetail class
from the PyMarkdown project. I was sure that I did not need to enable or disable
plugins at all. The basis for that certainty was the command line interface.
If a user does not want to use a plugin, they can simply not use the command line
argument related to the plugin. I might change my mind down the road, but that
was where I landed.
I did feel that it is right to add support, but I am still figuring out what kind
of information would look right in the details. As such, I started out with a
basic set of properties: plugin_id, plugin_name, plugin_version and
plugin_interface_version. It was just simple information that I knew that I could easily expand
on later. Given that, I made sure to include the plugin_interface_version property and set it
to the new constant: VERSION_BASIC or 1.
Lots Of Testing¶
In case anyone thinks otherwise, this entire process followed good test-driven development practices, with many tests added and performed. There is an incredibly small chance that I would write anything other than a Proof-Of-Concept without solid tests in place first. It just doesn’t feel right.
And believe me, during the refactoring, that adherence to test-driven development saved me a couple of times. While I would like to think that I have vast amounts of energy all the time, I am human. As such, there were days where I was more tired than others and tended to make more small mistakes. Mistakes that were caught by the decent set of tests that I have covering the project.
And to be clear to any readers, even on good days I make mistakes. My family and I call it “fat-finger syndrome” and I suffer from it continuously. When I am authoring articles like this one, I most often think as I type, and those two actions are coordinated with each other. But when I am writing code, I tend to have those two actions get out-of-sync with each other… with alarming frequency. But because I know I do this and have a solid process backing me up, I do not worry about it as much these days because I have confidence in the process.
Updating Badges¶
After I did chores around the house and outside in our yard, I found myself having some extra time before writing I started my writing on Sunday afternoon. As such, I wanted to have a decent sized task that I could conduct, but not so big as that it would take over my Sunday evening writing. Going to a couple of my projects, one thing that I noticed is that I was still not happy with how the badges look. I had a good amount of time to use, and I figured that was a task that was just around the correct size.
Badges are simple. Multiple sites on the internet provide images that can be used on web pages to denote various things. Most badges that I use are either static, based off the GitHub project, or based of the package information at PyPi.org.
If you look at the README.md file for the Project Summarizer project, you will see pictures that look like this:
Those images are generated by the following Markdown:
[](https://github.com/jackdewinter/pyscan)
[](https://pypi.org/project/project_summarizer)
[](https://pypi.org/project/project_summarizer)
[](https://github.com/jackdewinter/pyscan/actions/workflows/main.yml)
[](https://codecov.io/gh/jackdewinter/pyscan)
[](https://sourcery.ai)
[](https://github.com/jackdewinter/pyscan/stargazers)
[](https://pypistats.org/packages/project_summarizer)
[](https://github.com/jackdewinter/pyscan/issues)
[](https://github.com/jackdewinter/pyscan/blob/main/LICENSE.txt)
[](https://github.com/jackdewinter/pyscan/graphs/contributors)
[](https://github.com/jackdewinter/pyscan/network/members)
[](https://www.linkedin.com/in/jackdewinter/)
Yes, it is a lot of work for seventeen small images, but I feel they are worth it. Each badge related something that I feel is important about the project. That part I was okay with. It just didn’t look nice to me.
Yes, “nice”. It was not a very quantifiable word that I picked. That bugged we enough that I started playing around to figure out why I thought that. After some research, I found out that for me “nice” was the same as “organized” in this context. I liked the information, just did not like how it was being displayed.
Using that knowledge, I took another hour and kept on changing organizations of those tags until I came up with the current organization that you can see on the project’s README.md page. The first thing I decided to do was to put the information in Markdown tables to give it clear organization. After that quick fix, coming up with the categories required a bit more work and fiddling around. I played with different badges in each “category” and looked to see if they looked right together. Once I got a set that looked right, giving that category a name was simple.
The only category that I had problems with was the non-category for the third
line of the badges. I wanted to call that Dependencies, but I did not want
to add a list of every dependency. But I did want to call out the various
packages that I use to increase and maintain the quality of the project.
In the end, I decided to just leave the title for that category blank, essentially
becoming a second line for the Quality category. I am not sure if it will
stay like that, but for now I feel it is a good compromise.
What Is Next?¶
Project Summarizer plugin design. Check. Project Summarizer plugin loading code. Check. Flushing out Project Summarizer plugins. Check. With all that work done, I have been working on a project for a couple of months that will make an ideal plugin. Here is hoping I can move it along and have it ready by next week!
Comments
So what do you think? Did I miss something? Is any part unclear? Leave your comments below.