Easy Async Python Tracing with OpenTracing

Editor's note: This guest post was written by Jessie Morris, a senior architect at Midigator, an Equifax company.

Image

When using microservices, getting visibility into the lifecycle of an event or request can be difficult.

At Midigator, an Equifax company, the technology has a heavily microservice-driven architecture with many small, independent, and self-contained services. Because we are always looking for ways to improve visibility, reduce debugging time, and improve the quality of our products, we’ve sought out ways to solve common challenges.

OpenTracing makes it easier, and there are many open source tools that automate tracing libraries. Midigator has created an open source library that adds tracing to common async Python libraries.

What is Tracing?

Tracing is a method that allows you to track a transaction throughout your system. This includes across multiple services and languages, giving you insight into how much time was spent where, what decisions or inflection points were made, and if an error occurred where that happened and why.

Tracing is a more efficient alternative to the traditionally-used logging process.

Logging was oftentimes highjacked for this. Services at the edge of a system would generate a request ID and include it in their log statements. They would then send that request ID on calls to subsequent services. Those services would then propagate the request ID in their logging statements and include it in any calls to services and so on. If an issue arose, a search could be made of all logs for that requested ID, providing the ability to see the entire log history for that request.

While using logs for this purpose works, it does tend to be a bit noisy and isn’t structured. This makes it difficult to visualize and search. Enter tracing.

We decided to use OpenTracing and Jaeger. Both of these are open source projects. OpenTracing provides vendor agnostic libraries for instrumentation and tracing. It allows code to be instrumented, and then at runtime, a specific implementation can be used to actually collect and visualize the data.

Jaeger is the actual implementation of tracing collection that we decided to use. It provides searching, visualization, aggregation, dependency analysis, sampling logic, and more. Jaeger uses UDP to collect traces from the application, ensuring that any impact to performance is negligible.

Tracing Code: Python Example

We use several different languages including Go, Node, and Python. OpenTracing supports all of these and more, but I’d like to dive into some Python examples here.

Python works well, is easy to write and maintain, and encourages best practices like consistent styles, unit testing, and more. Python also has an amazing set of libraries and frameworks, including OpenTracing. Adding tracing to your Python code is extremely easy. A very basic example would be something like the following:

In this code block, we get the “global” tracer, start a span, and print our classic “Hello world!”. We use a context manager that lets us trace exactly how long our print statement takes. The context manager automatically closes the span at the end giving us a reasonably accurate time of how long the code inside of the block took to execute.

Instrumenting your code is extremely valuable, but instrumenting libraries, frameworks, and other code you don’t own can be a bit challenging. Fortunately, the open source community has created libraries that instrument those frameworks. For Python, Uber has created opentracing_instrumentation. It instruments things like boto3, SQLAlchemy, and Tornado.

Tracing Async Code

Python has recently added async I/O support enabling improved performance on I/O heavy workloads. Most of the processing we do is light on CPU and is largely I/O dependent. This makes the async paradigm a perfect fit for us. Due to how new aysncio is in the Python ecosystem, there wasn’t an existing solution for automatic instrumentation of async frameworks like aiohttp, aiomysql, and others.

OpenTracing for Python does support context propagation using asyncio, but there wasn’t a good automatic instrumentation library. We decided to create one for internal use and after some vetting felt that others would find it useful as well. This open source library is available here. It currently supports aiobotocore, aioboto3, aiohttp, and aiomysql.

Using Midigator’s python_opentracing_async_instrumentation

While the README in the library’s repo contains some helpful usage information, I’ll give a short recap here.

For clients, the library will patch (or use hooks where possible) methods to seamlessly trace those libraries. Just like opentracing_instrumentation, our async instrumentation library is intelligent and will only patch libraries you have installed. No runtime issues will occur if these libraries aren’t present. To enable these hooks use:

On servers, there is currently out-of-the-box support for aiohttp. Additional frameworks can be supported. For aiohttp, use:

If you want to use a different HTTP server, you can implement the AbstractRequestWrapper class and create a middleware that calls the before_request function. An example of this implementation can be seen for aiohttpserver.

An example trace containing aiomysql, aioboto, and generic HTTP requests. Useful tags and data are included on spans as well.

Summary

OpenTracing provides a great method to instrument code and systems. Midigator’s open source library opentracing_async_instrumentation enables a fast and easy way to instrument Python async code. We encourage others to extend and improve this library and to make a Merge Request with improvements, additional libraries, documentation, and other contributions.