The 5 Best Logging Libraries for Python

When a deployed application fails unexpectedly in production, it can be difficult to pinpoint where, when, or why the problem occurred. To remedy this, you can configure logging in to your application to keep track of activities in files or send them to a monitoring tool. These logs function as an audit trail that can tell developers the state of application before the an issue arose, making it easier to debug the issue. In addition, logging also gives product teams insights into how users are using the application, which can help them make better decisions for the application's user experience.

Python ships with a logging library, and the community also offers third-party libraries that you can use for logging.

In this article, we'll explore the five best logging libraries for Python:

• Loguru

• Standard Library Logging Module

• LogBook

• Structlog

• Picologging

Why Use Logging Libraries for Python?

Often programmers gravitate to using the print() method to log and debug their applications. While this is easier to do, it has a lot of drawbacks:

• The logs are difficult to read, parse, or filter as they are not in a structured format like JSON nor have fields, such as severity level, or timestamp to add more context.

• hard to configure your application to collect all logs and send them to multiple destinations, for example, files, sockets, standard output, or emails.

• If your application is a library meant to be imported and used by other developers, the print messages can clutter the user's standard output.

Consider the following example using the print() statement:

print("This is a trace message.")
print("A debug message.")

# Output
This is a trace message.
A debug message.

It is harder to know the severity of the messages or when the message was generated. You could alter the print() statement to include the level, and timestamp but you would just be re-eventing the wheel because most logging libraries do this by default.

Now, compare the output from print() with the output from a Python logging library:

{"level": "WARNING", "message": "This is a warning message", "asctime": "2023-03-28 07:46:24,435"}

The following are some of the things that stand out from the log message:

• The log message is in a structured format(JSON), which is machine-readable.

• It includes a level that indicates the severance of the message. Most libraries support the following levels: DEBUG, INFO, ERROR, WARNING, and CRITICAL.

• It includes a timestamp(asctime), which tells you the date the log message was created.

• It contains the message that describes the event that occurred.

On top of that, you can customize the log messages to add more information, for instance, the process ID, the module name, the name of the logger, etc.

Another reason to consider using a logging library is that they can send the logs to multiple persistent storages, which include the standard output, files, emails, sockets, or monitoring tools.

Now that you have an idea of how handy a logging library is, we will go over the five best logging libraries in Python.

5 Best Python Logging Libraries

#1 Loguru

Loguru is a popular, third-party logging library developed to make logging easier in Python. It is pre-configured with a lot of useful functionality, allowing you to do common tasks without spending a lot of time messing with configurations. You can format logs, filter, or specify destinations to send logs using only a single add() function. At the time, of writing, it has 14K Github stars making it the most popular third-party logging library in Python.

Let's explore some of the Loguru features:

• Built-in support for structured logging.

• Supports pretty printing.

• Creating custom levels.

• Ships with a parser() method, which lets you extract information from the logs.

How to Use Loguru

Assuming you have a virtual environment active on your local machine, enter the following command to install loguru:

pip install loguru

Next, create a loguru_demo.py file and add the following code to log messages using loguru:

from loguru import logger

logger.trace("This is a trace message.")
logger.debug("This is a debug message")
logger.info("This is an info message.")
logger.success("This is a success message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

In the first line, you import the logger object from loguru, which is pre-configured to send logs to the standard output. You then call the methods corresponding to the logging levels the module supports.

The following are the supported levels ordered in increasing severity:

• TRACE(5): low-level details of the program's logic flow.

• DEBUG(10): Information that is helpful during debugging.

• INFO(20): Confirmation that the application is behaving as expected.

• SUCCESS(25): Indicates an operation was successful.

• WARNING(30): Indicates an issue that may disrupt the application in the future.

• ERROR(40): An issue that needs your immediate attention but won't terminate the program.

• CRITICAL(50): A severe issue that can terminate the program, like "running out of memory".

When you are finished adding the code, save the file, then run the program:

python loguru_demo.py

The output will match the following:

As you can see in the output, the log messages are colorized and include the timestamp, log level, and a message without doing any configurations.

If you examine the output closely, you will notice that a log record with the level TRACE is missing. This is because Loguru defaults to DEBUG as its minimum level; therefore, all log messages with a severity below DEBUG are ignored. To set TRACE as the minimal level, use luguru's add() method as demonstrated below:

from loguru import logger
import sys        # <!- add this line

logger.remove(0)             # <- add this line
logger.add(sys.stdout, level="TRACE")   # <- add this line

logger.trace("This is a trace message.")
...

When you set the minimum level using the add() function, you must specify the destination to send the logs. To keep things simple, you send all log messages to the standard output using sys.stdout.

In the output, you will now see the log message with the level TRACE included:

Logoru also provides the ability to customize the format of the log records using the format option. Pass the option to the add() function as done in the following simplified example:

from loguru import logger
import sys

logger.remove(0)
logger.add(sys.stderr, format="{level} : {time} : {message}: {process}")  # <-

logger.error("This is  an error message.")

# Output
# ERROR : 2023-03-28T09:27:12.674313+0200 : This is  an error message.: 25464
#  ^                  ^                            ^                     ^
# level              time                        message                process ID

Anything inside {} like {level} is a formatting directive, to learn more about them, visit [the documentation]https://loguru.readthedocs.io/en/stable/api/logger.html#record).

You don't have to use the colons :, you can use anything you see fit:

logger.add(sys.stderr, format="{level} - {time} - {message} - {process}")

# Output
# ERROR - 2023-03-28T09:33:29.514479+0200 - This is an error message. - 25734

So far, the messages have not been structured. To configure Loguru to use structured logging with JSON, you add the serialize option to the add() function:

from loguru import logger
import sys

logger.remove(0)
logger.add(sys.stderr, serialize=True) # <-

logger.error("This is  an error message.")

# output
{"text": "2023-03-28 09:36:31.458 | ERROR    | __main__:<module>:7 - This is  an error message.\n", "record": {"elapsed": {"repr": "0:00:00.023078", "seconds": 0.023078}, "exception": null, "extra": {}, "file": {"name": "loguru_demo.py", "path": "/home/<your_username/loguru_demo/loguru_demo.py"}, "function": "<module>", "level": {"icon": "❌", "name": "ERROR", "no": 40}, "line": 7, "message": "This is  an error message.", "module": "loguru_demo", "name": "__main__", "process": {"id": 25826, "name": "MainProcess"}, "thread": {"id": 140082527805440, "name": "MainThread"}, "time": {"repr": "2023-03-28 09:36:31.458896+02:00", "timestamp": 1679988991.458896}}}

That is a lot of fields! Loguru allows you cherry pick the fields as demonstrated below:

from loguru import logger
import json     # <!- add this line
import sys

# Add the following function
def serialize(record):
    subset = {
            "level": record["level"].name,
            "timestamp": record["time"].timestamp(),
            "message": record["message"]}
    return json.dumps(subset)

# Add the following function
def formatter(record):
    record["extra"]["serialized"] = serialize(record)
    return "{extra[serialized]}\n"

logger.remove(0)
logger.add(sys.stderr, format=formatter)   # <!-

logger.error("This is  an error message.")

# Output
# "level": "ERROR", "timestamp": 1679989265.887056, "message": "This is  an error message."}

First, you import the json module. Following that, you define a serialize() function that returns a JSON object that only contains the fields you want: level, timestamp, and message. From there, you define a formatter() function that takes care of the formatting. Finally, you set the format option on the add() method to the formatter function.

If you want more details about this, see serializing log messages using a custom function.

Until now, you have been sending all the logs to the standard output. Let's send them to a file instead:

from loguru import logger

logger.remove(0)
# modify the following line
logger.add("app.log", format="{level} : {time} : {message}: {process}")

logger.error("This is  an error message.")

The add() function now takes a filename as the first argument. The file will be created automatically for you.

After running the program, check the project directory. You will find the app.log file created with the following contents:

# Output
ERROR : 2023-03-28T09:45:53.483671+0200 : This is an error message.: 26021

We now looked into some of the useful features in Loguru. For more details, visit the documentation.

#2 Standard Library Logging Module

Python ships with a logging module, which contains a lot of useful features in comparison to the default logging libraries in other programming languages. It is popular among developers, well-documented, and its functionality can be extended with third-party modules. It is a bit complex to set up some tasks in comparison to Loguru, but very powerful once you get hang of it.

The following are some of the features it offers:

• Formatting log records.

• Sending logs to multiple destinations ranging from the standard output, files, emails, sockets, and HTTP.

• Sophisticated filtering.

• Defining custom levels.

• Can be extended with other modules to support structured logging, pretty-print logging

How to Use The Python Standard Library Logging Module

As mentioned, the logging module is built-in Python, which the latest version is Python 3.11 at the time of writing. There is no need for any installation.

To use it, create a stdlib_demo.py, and add the following contents:

import logging

logging.basicConfig()
logger = logging.getLogger(__name__)

logger.debug("This a debug message");
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

First, you import the logging class, and then do some basic configurations using the basicConfig() of the logging module.

Following that, you invoke the getLogger() method of the logging module with the app name obtained from __name__ or you can just pass any name of your choosing.

From there, you invoke all the methods corresponding to the levels to log the message. The logging module supports the following levels: DEBUG, INFO, WARNING, ERROR, and CRITICAL. It does not support the SUCCESS or TRACE level you saw earlier with Loguru.

When you run the file, your output will look like this:

# output
WARNING:__main__:This is a warning message
ERROR:__main__:This is an error message
CRITICAL:__main__:This is a critical message

The output shows the log level, the assigned module name, and the message.

The Python's logging module defaults to the WARNING as the minimum level going up. That is why DEBUG or INFO levels aren't shown in the output. You can modify this behavior using the setLevel() method of the logging module as follows:

import logging

logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)      #<-

logger.debug("This a debug message");
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

# Output
DEBUG:__main__:This a debug message
INFO:__main__:This is an info message
WARNING:__main__:This is a warning message
ERROR:__main__:This is an error message
CRITICAL:__main__:This is a critical message

Now all the messages have been logged.

The logging module provides you the ability to customize log records, and configure the destination to send logs with the use of the following objects:

• Handler Objects: Used to send logs to various destinations, such as the file, standard output, network socket, or email to mention a few. In this post, you will use the StreamHandler to send logs to the standard output stream(sys.stdout), and the FileHandler to send logs to files on a computer disk.

• Formatter objects: Used to change the contents or format a log message. You will use the object to add a time stamp and a process ID to the log message.

First, let's familiarize ourselves with the handler objects. We will use the StreamHandler() to send logs to the standard output with this latest example:

import sys
import logging

logger = logging.getLogger(__name__)

# call the `StreamHandler`
stdout_handler = logging.StreamHandler(stream=sys.stdout)
# add it to the logger instance
logger.addHandler(stdout_handler)


logger.error("This is an error message")
logger.critical("This is a critical message")

You call StreamHandler() with the stream option set to sys.stdout to send logs to the standard output. From there, you register it to the logger instance using the addHandler() method of the logger instance.

When run, the output will look closely to this:

#  Output
This is an error message
This is a critical message

The messages no longer have the level or name, because we no longer have the logging.basicConfig() to do some basic configurations, which includes formatting.

Now that you are familiar with how to use a Handler object, let's build upon the example to format the log records. As mentioned earlier, you will use the formatter object.

Take the following example:

import sys
import logging

logger = logging.getLogger(__name__)

stdout_handler = logging.StreamHandler(stream=sys.stdout)
format_output = logging.Formatter('%(levelname)s : %(name)s : %(message)s : %(asctime)s') # <-

# Register the formatter to the stdout handler
stdout_handler.setFormatter(format_output)      # <-
logger.addHandler(stdout_handler)

logger.error("This is an error message")
logger.critical("This is a critical message")

The Formatter() function takes a string that defines the message format using the logrecord attributes.

Running the file will yield the following output:

# Output
# ERROR : __main__ : This is an error message : 2023-03-28 13:46:11,510
# CRITICAL : __main__ : This is a critical message : 2023-03-28 13:46:11,510
#
#  ^           ^                ^                             ^
# levelname   name           message                      asctime

Now let's make the logs structured using the JSON format. The logging module currently doesn't have built-in support for creating structured logs. Thanks to the community, you can use the python-json-logger module to create the structured logs.

First, install the module as follows:

pip install python-json-logger

And modify the example as follows:

import sys
import logging
from pythonjsonlogger import jsonlogger   #<-  add this

logger = logging.getLogger(__name__)
stdout_handler = logging.StreamHandler(stream=sys.stdout)

# format with JSON
format_output = jsonlogger.JsonFormatter('%(levelname)s : %(name)s : %(message)s : %(asctime)s') # <-

stdout_handler.setFormatter(format_output)
logger.addHandler(stdout_handler)

logger.error("This is an error message")
logger.critical("This is a critical message")

First, you import the jsonlogger from the python-json-logger module. Next, you invoke the JsonFormatter() method of the jsonlogger with the string containing logrecord attributes we want to see in the JSON log message.

You will receive the following output when you run the file:

# Output
{"levelname": "ERROR", "name": "__main__", "message": "This is an error message", "asctime": "2023-03-28 14:04:01,930"}
{"levelname": "CRITICAL", "name": "__main__", "message": "This is a critical message", "asctime": "2023-03-28 14:04:01,930"}

Now, the example currently sends the structured log records to the standard output. Let's modify to forward logs in a file as well:

import sys
import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

stdout_handler = logging.StreamHandler(stream=sys.stdout)
## Create a file handler
fileHandler = logging.FileHandler("app.log")   # <-

format_output = jsonlogger.JsonFormatter('%(levelname)s : %(name)s : %(message)s : %(asctime)s')

stdout_handler.setFormatter(format_output)

fileHandler.setFormatter(format_output)     # <-

logger.addHandler(stdout_handler)
## the file handle handler
logger.addHandler(fileHandler)             # <-

logger.error("This is an error message")
logger.critical("This is a critical message")

After running the file, you will find the app.log file created in the directory with the following contents:

# Output
{"levelname": "ERROR", "name": "__main__", "message": "This is an error message", "asctime": "2023-03-28 15:57:33,362"}
{"levelname": "CRITICAL", "name": "__main__", "message": "This is a critical message", "asctime": "2023-03-28 15:57:33,362"}

We have barely scratched the surface of what the logging module is capable of doing. If you want to learn more, visit the Python documentation.

#3 LogBook

Another logging library to consider is the LogBook module, which at the time of writing has about 1.4K stars on GitHub. The developers built it as a replacement for the standard library's logging module.

It has a lot of helpful features:

• Sending log messages on several mediums, such as phone or desktop notification systems using the notifiers module.

• ships with handlers that can forward logs to streams, files, or emails.

• Supports Redis, ZeroMQ, RabbitMQ, and many more.

To explore more features, visit LogBook documentation

How to Use LogBook

First, install the library in the project directory:

pip install logbook

Create a logbook_demo.py file, then add the following:

from logbook import Logger, StreamHandler
import sys

logger = Logger(__name__)
StreamHandler(sys.stdout).push_application()

logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.notice("This is a notice message.")
logger.warning("This is a warning message.")
logger.error("This is  an error message.")
logger.critical("This is a critical message.")

In the first line, you import the Logger, and the StreamHandler. Next, you create an instance of the logger. Following that, you use the StreamHandler() method to send logs to the standard output stream.

Running the file produces the following output:

# Output
[2023-03-28 15:28:39.915738] DEBUG: __main__: This is a debug message.
[2023-03-28 15:28:39.915948] INFO: __main__: This is an info message.
[2023-03-28 15:28:39.916053] NOTICE: __main__: This is a notice message.
[2023-03-28 15:28:39.916152] WARNING: __main__: This is a warning message.
[2023-03-28 15:28:39.916258] ERROR: __main__: This is  an error message.
[2023-03-28 15:28:39.916403] CRITICAL: __main__: This is a critical message.

Take note in the output that the library supports all the levels that the standard library's logging module supports. It only adds one level NOTICE, useful for creating non-error messages.

If you want to get send logs to a file, you can use the FileHandler(). Take a look at the following example:

import logbook
import sys

logger = logbook.Logger(__name__)

log = logbook.FileHandler('app.log', level='INFO') # <-
log.push_application()

logger.error("This is  an error message.")
logger.critical("This is a critical message.")

The FileHandler() method takes the name of the log file, and the minimum supported level, which is INFO here.

Running the file creates an app.log file containing the following contents:

# Output
[2023-03-28 15:34:35.811250] ERROR: __main__: This is  an error message.
[2023-03-28 15:34:35.811626] CRITICAL: __main__: This is a critical message.

Logbook also provides a format_string property that you can use to change the format of the log messages. Consider the following example:

import logbook
import sys

logger = logbook.Logger(__name__)

log = logbook.FileHandler('app.log', level='INFO')
log.format_string = '{record.level_name} : {record.message} : {record.time} '   # <-
log.push_application()

logger.error("This is  an error message.")
logger.critical("This is a critical message.")

You set the format_string to a string, which uses the logrecord attributes time, level_name, and message of the record object.

When you run the file, the output will match the following:

# Output
ERROR : This is an error message. : 2023-03-28 15:41:08.677903
CRITICAL : This is a critical message. : 2023-03-28 15:41:08.678186

The log message now starts with the level instead of the timestamp.

Setting up structured logging is challenging with LogBook, the maintainers recommend implementing a handler to do it. For more details, visit Logbook's Github Issues page.

Now that you have the basic knowledge of how to use LogBook, visit the documentation page to learn more about other features.

#4 Structlog

Structlog is a small structured logging library in Python. It was developed in 2013 and is constantly keeping up with changes in Python, such as context variables, asyncio, and type hints. It currently has over 2.5K stars on GitHub at the time of the writing.

The following are some of the main features:

• Built-in structured logging

• Supports filtering logs

• pretty printing with colors

• Can be configured to work with the standard logging library.

• Supports both synchronous and asynchronous methods.

How to Use Structlog

Install the structlog module as follows:

pip install structlog

Create a structlog_demo.py file, then add the following contents to initialize the logger:

import structlog
logger = structlog.getLogger(__name__)

logger.debug("This is a debug message.")
logger.info("This is an info message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

structlog colorizes the output to make it easier to distinguish the messages. It supports all the five levels the standard library's logging module supports, which include DEBUG, INFO, WARNING, ERROR, and CRITICAL.

If you want the logs to be structured, you can configure structlog with a chain of processors as follows:

import logging
import structlog

structlog.configure(
        processors =  [
            structlog.processors.TimeStamper(),
            structlog.processors.add_log_level,
            structlog.processors.JSONRenderer(),
            ]
        )
logger = structlog.get_logger()
logger.error("This is an error message.")
logger.critical("This is a critical message.")

The processors modify the log messages and pass the value to the next processor. The first TimeStamper() processor adds a time stamp, then the message is passed to the add_log_level processor to add a severity level. Finally, the log message is passed to the JSONRenderer() processor to log the message in JSON format.

Upon running the file, the output will look similar to the following:

# Output
{"event": "This is an error message.", "timestamp": 1680019746.526201, "level": "error"}
{"event": "This is a critical message.", "timestamp": 1680019746.5263886, "level": "critical"}

There is so much to the structlog library, visit the documentation page to discover more features.

#5 Picologging

Picologging is a recently-made logging module, which is 9 months old and has over 500 stars on GitHub at the time of writing. Microsoft developed it to replace the standard library's logging module and claims it is 4-10x faster than the built-in solution. However, the documentation states that it is in the early-alpha stage and has some missing features. But still, it is a promising solution and something to keep an eye on.

Picologging has the following features:

• uses the same API as the built-in logging module.

• Formatting messages

• Forwarding logs to persisting storages, for instance, the standard output, files, emails, sockets, or more using handlers

Put simply, all the features provided by the built-in logging library are supported by the Picologging library with a caveat that some features are missing because it is still in the alpha stage.

How to Use Picologging

Install the module in a virtual environment:

pip install picologging

Create a pico_demo.py file with the following contents:

import picologging

pic
logger = picologging.Logger(__name__, picologging.DEBUG)

logger.debug("This a debug message");
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

# Output
This a debug message
This is an info message
This is a warning message
This is an error message
This is a critical message

As you can see, the code looks similar to the first example when we covered the standard library's logging library. Formatting messages can also be done the same as you did earlier using the standard library's logging module:

import sys
import picologging

logger = picologging.getLogger("example__app")

stdout_handler = picologging.StreamHandler(sys.stdout)
format_output = picologging.Formatter('%(levelname)s : %(name)s : %(message)s : %(asctime)s')

stdout_handler.setFormatter(format_output)
logger.addHandler(stdout_handler)

logger.error("This is an error message")
logger.critical("This is a critical message")

If you compare the example with the one using the built-in logging module, you will only notice a few differences. logging has been substituted with picologging, and StreamHandler() no longer has the stream attribute.

If you want to explore this library, refer to the section covering the standard library's logging module to experiment with logging into files, and structured logging. Be sure to check out the documentation, as well as the standard library's logging module documentation.

Conclusion

In this article, we looked at the best 5 logging libraries available for Python. If you are still undecided, we recommend starting with the loguru library as it is easy to get started. Later when you have the time, it is still worth learning the standard library's logging module as it is popular and most logging library APIs are inspired or built upon it. With the advent of Picologging, which looks promising in terms of performance, familiarity with the built-in module will help you make a smooth transition if it gets traction.