How to Debug Python Scripts with the Logging Module
Table of Contents
∘ Introduction ∘ The Logging Module ∘ The Levels of Logging ∘ Configuring Levels ∘ Configuring Levels for Debugging ∘ Creating Log Files ∘ Formatting Log Messages ∘ Key Takeaways
Introduction
Consider the following scenario: You've written a piece of code that either returns an error or yields an unexpected value.
Python">x1 = function1(x)
x2 = function2(x1)
x3 = function3(x2)To find the erroneous line of code, you write a print statement…
x1 = function1(x)
print(x1)
x2 = function2(x1)
x3 = function3(x2)Then add another print statement…
x1 = function1(x)
print(x1)
x2 = function2(x1)
print(x2)
x3 = function3(x2)Then follow it up with another print statement.
x1 = function1(x)
print(x1)
x2 = function2(x1)
print(x2)
x3 = function3(x2)
print(x3)Once you've identified and fixed the issue, these print statements are useless. So, you delete or comment out each of them one by one:
x1 = function1(x)
#print(x1)
x2 = function2(x1)
#print(x2)
x3 = function3(x2)
#print(x3)If your troubleshooting experience resembles the scenario above, you're already familiar with the frustration of using print statements to deal with erroneous lines of code.
Fortunately, there is a tool in Python that offers a much more effective strategy for debugging code: the logging module.
Here, we delve into the basic functionalities of the logging module and explore the features that make it such a powerful troubleshooting tool.
The Logging Module
The logging module is designed for programmers looking to track certain events in their programs effectively.
Fortunately, there aren't any prerequisites to learning to use the tool, especially if you're already familiar with Python's print statements.
In terms of syntax, the logging module's commands are very similar to print statements, allowing users to generate messages with simple one-liners.
import logging
logging.warning('Hello World')
That being said, logging includes features that are not provided by regular print statements, which we will now cover.
The Levels of Logging
In the logging module, all messages are not equal. Events in the logging module are broken down into different levels of importance.
The logging module's documentation lists the 5 levels of logging and explains when they should be used.
In other words, when users use logging to write messages, they also get to set the importance of said message. An INFO message is best for monitoring the status of an event, a WARNING message is best for issuing a warning, and an ERROR message is best for reporting an error.
Thus, users can assign importance to their messages by using levels depending on their purpose. For example, when loading a dataset, one can write messages with INFO and ERROR levels.
try:
mock_data = pd.read_csv('random_data_source.csv')
logging.info('random_data_source.csv has been loaded')
except:
logging.error('random_data_source.csv does not exist')With different levels of importance assigned to messages, users can define the threshold for the importance of messages that they want to or do not want to see.
By default, the logging module only considers logs that are at the level WARNING or above.
This means that logging a message without additional configuration does not return messages at the DEBUG and INFO levels.
As shown in the output, only the messages at levels WARNING, ERROR, and CRITICAL are actually generated.
Configuring Levels
Unlike print statements, logging allows users to configure the importance levels of messages that should be generaed.
For instance, suppose that we need to record messages at levels INFO and above.
We can instruct the program to generate INFO level messages by using the basicConfig method and defining the level parameter:
Now, all messages at levels INFO and above will be shown.
Since the program has been configured to accept logs at INFO and above, all logs that are not at the DEBUG level are recorded.
Configuring Levels for Debugging
Through the basicConfig method, users can shift the threshold for logging levels to include the desired levels while omitting the undesired levels.
For instance, let's suppose that we are performing transformations on a variable and wish to find the value of the variable after each transformation. Here, we perform mathematical operations on variable x:
import logging
logging.basicConfig(level=logging.DEBUG)
x = 5
logging.info('Performing mathematical operations on x')
x = x * 2
logging.debug(f'x * 2 = {x}')
x = x + 6
logging.debug(f'x + 6 = {x}')
x = x / 4
logging.debug(f'x divided by 4 = {x}')The threshold logging level has been set to DEBUG, which means that all of the logging messages will be created. Running the code will return the following:
These messages can be helpful, but once the debugging is complete, they provide no value. So, how do we stop these messages from being generated in future iterations?
If we were working with print statements, the solution would be to delete or comment out each and every unwanted print statement. Thankfully, with logging, the unwanted debug messages can be removed by simply modifying the logging level threshold using the basicConfig method.
We can now change the logging level to INFO, thereby omitting all log messages with the DEBUG level importance.
logging.basicConfig(level=logging.INFO) # Change from DEBUG to INFO
x = 5
logging.info('Performing mathematical operations on x')
x = x * 2
logging.debug(f'x * 2 = {x}')
x = x + 6
logging.debug(f'x + 6 = {x}')
x = x / 4
logging.debug(f'x divided by 4 = {x}')
This time, even though the logging commands are still in place, they are omitted as the DEBUG level is lower than the assigned INFO level threshold.
Similarly, when we wish to start the debugging process, we can use logging to do the opposite: lower the threshold importance level for log messages so that the program generates messages with lower importance levels.
Creating Log Files
When using print statements, messages are shown on a command line or console but are not stored in any location.
Using the logging module's basicConfig method, users can create a file dedicated to recording all messages created with the module.
For instance, the following snippet uses the basicConfig method to save all messages in a newly created file named "demo.log".
import logging
logging.basicConfig(filename='demo.log',
level=logging.DEBUG)
logging.debug('Debug message')
logging.info('Info message')Now, no messages will show up in the console when the script is run. Instead, the messages will be stored directly in the "demo.log" file.
Furthermore, these recorded messages are not overwritten; they remain in the "demo.log" file when additional logs are added. If we run the same script again, the new log messages will be added in new lines.
Formatting Log Messages
The messages in logs are not always that informative and readable.
To generate messages more fitting for the given use case, users can configure the format of their messages with the basicConfig method.
In the snippet below, we configure the messages to be composed of the timestamp, the logging level, and the message.
logging.basicConfig(filename='demo.log',
level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s'
)
logging.debug('Debug message')
logging.info('Info message')
Establishing the format of your logs can make the generated messages both informative and readable. Of course, the ideal format depends on the programmers' personal preference.
Key Takeaways
Overall, while print statements tend to be treated as the go-to tool for examining or troubleshooting code, the logging module is a more fitting tool for such purposes.
Logging offers a system that categorizes messages into different levels of importance. Moreover, they boast features that enable users to easily configure what messages to generate, how they should be formatted, and where they should be stored.
The logging module has the capacity to make the debugging experience quicker and stress-free, so consider adopting it if you are tired of using plain, generic print statements.
Thank you for reading!
