• Create a local OpenTelemetry collector and Jaeger setup using Docker Compose

• Build a Python Flask Server Application and instrument it with OpenTelemetry

• Build a Python HTTP Client Application and instrument it with OpenTelemetry

• Learn how to use context propagation to traces cross-service requests

• Trace your cross-service requests in Jaeger

Tools we will use

Jaeger is an open-source distributed tracing platform. In this tutorial, we will use a version that contains all components in a single Docker image. It will enable a fast and convenient way to display distributed traces on a local machine.

Docker Compose is a tool for defining and running multi-container applications. It allows us to describe all Docker images in a single configuration file, making it handy for local testing.

We use OpenTelemetry Collector to forward traces from the Python applications to Jaeger. In this case, we could manage without it, however, the OpenTelemetry Collector is useful, for example, if you want to enrich data, or collect not only traces but also metrics, or experiment with different vendors.

Pre-requisites

We will use the Unix command interface. If you use Windows, you can use WSL.

Install the following programs if you don’t have them

1. Docker Compose

2. Python3 and pip

Set up Jaeger and OpenTelemetry Collector using Docker Compose

1. Create Collector config file

    cat > otel-collector-config.yaml << 'EOF'
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
    extensions:
      health_check: {}
    exporters:
      otlp:
        endpoint: jaeger:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
    EOF
   

2. Create a Docker compose file

    cat > docker-compose.yaml << 'EOF'
    services:
      otel-collector:
        image: otel/opentelemetry-collector-contrib:latest
        command: ["--config=/etc/otel-collector-config.yaml"]
        volumes:
          - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
        ports:
          - "4317:4317" # OTLP gRPC receiver
      jaeger:
        image: jaegertracing/all-in-one:latest
        ports:
          - "6831:6831/udp" # UDP port for Jaeger agent
          - "16686:16686" # Web UI
          - "14268:14268" # HTTP port for spans(venv)
    EOF
   

3. Start Docker Compose:

    docker compose -f docker-compose.yaml up
   

4. Verify that the local Jaeger instance works: open http://localhost:16686/ in your browser.

Create a virtual environment and install Python libraries

1. Create and activate a virtual environment

    python3 -m venv venv
    source ./venv/bin/activate
   

2. Install python packages

    pip install flask
    pip install urllib3
   

3. Install OpenTelemetry instrumentation

    pip install opentelemetry-distro
    opentelemetry-bootstrap -a install
    pip install opentelemetry-exporter-otlp-proto-grpc # send traces over OTLP
    pip install opentelemetry-instrumentation-urllib3 # instrumentation for urllib3 library
   

   Instrument a cross-server request

   1. Create a simple Flask server application. It will serve HTTP requests.

      ```bash mkdir server cat > server/app.py << 'EOF' from flask import Flask, jsonify

      app = Flask(name)

      @app.route('/example1/') def trace(arg): return jsonify({"trace": f"Trace argument is {arg}"})

if name == "main": app.run(host="0.0.0.0", port=8080, debug=True) EOF

    2. Start the Flask server application with OpenTelemetry instrumentation

        ```bash
        cd server && \
        opentelemetry-instrument \
          --service_name demo-server \
          --metrics_exporter none \
          --logs_exporter none \
          flask run -p 8080

3. Verify that your Flask application works: open http://localhost:8080/example1/test in your browser

4. Create a simple client application. We will use urllib3 to make HTTP requests and URLLib3Instrumentor for OpenTelemetry instrumentation. Create a file simple-client.py with following content

    import urllib3
    from opentelemetry.instrumentation.urllib3 import URLLib3Instrumentor
   
    def strip_query_params(url: str) -> str:
        return url.split("?")[0]
   
    URLLib3Instrumentor().instrument(
        # Remove all query params from the URL attribute on the span.
        url_filter=strip_query_params,
    )
   
    http = urllib3.PoolManager()
    response = http.request("GET", "http://localhost:8080/example1/test")
   
    if response.status == 200:
        print("Response:", response.json())
    else:
        print("Error:", response.status_code)
   

5. Run the client application

    opentelemetry-instrument \
      --service_name demo-client \
      --metrics_exporter none \
      --logs_exporter none \
    python simple-client.py
   

6. Verify the results in Jaeger: open http://localhost:16686/, choose “demo-client” in the field Service in the left panel, and click “Find Traces

7. 

   Now, you should see cross-service traces in the search results. When you open a trace, you can see how much time it took at each stage:

Congratulations! You’ve just implemented your first cross-service trace in OpenTelemetry.

Clean up

It is the optional step if you want to clean up the environment on your machine.

1. Clean up the Python virtual environment

    deactivate
    rm -rf venv/
   

2. Shut down Docker Compose components

    docker compose -f docker-compose.yaml down
   

Tip

Prefer instrumented Python libraries to generate telemetry data. For example, we used the instrumented library urllib3 to produce the traces rather than manually instrumenting with OpenTelemetry in this tutorial. You can find the full list of instrumented libraries here.

What is OpenTelemetry

OpenTelemetry is an observability framework for generating and delivering telemetry data, such as metrics, logs, and traces, to an Observability storage.

Benefits of using OpenTelemetry for metrics

When you build an OpenTelemetry infrastructure for metrics, you can enjoy the following benefits:

• Support of multiple programming languages

• Easy integration with different metric backends (storages)

• Flexibility across environments — Kubernetes, clouds, and bare metal

• Unified infrastructure for metrics, logs, and traces

What is a metric?

A metric is a time-series measure of a process or a state within software. Engineers use metrics to monitor software health, performance, and business efficiency.

What kind of metrics exist in OpenTelemetry?

Gauge

A Gauge is an instantaneous snapshot of data at each point in time.

For example, it can be the number of processes on the machine or the CPU temperature.

Sum

Sum is a numeric metric that represents the cumulative total of all reported measurements over a time interval or a cumulative value from the beginning of time. For example, it can be the total amount of requests served by a service in a minute (1), or a cumulative value of requests from the start (2).

Histogram

A histogram describes a distribution of metric values grouped into buckets, where each bucket represents a range of values and the count of measurements that fell into that range. For example, buckets can represent server request latency intervals, and values - the count of requests that fall in the bucket.

Each metric in OpenTelemetry contains the following information:

• Metric name

• Attributes - these can be a name of the application that produced a metric, an availability zone, the name of the server, the instrumentation name, and many others

• Value type of the point (integer, floating point, etc.)

• Unit of measurement

• Points at each time interval

Read more about the Metrics data model in OpenTelemetry in the official documentation.

How the OpenTelemetry metrics pipeline works

Let’s look at the metrics lifecycle within the Observability framework.

1. Metrics generation

Typically, your application needs to generate metrics. To do that, you can use a corresponding OpenTelemetry SDK – a set of tools for generating telemetry data and sending it to the OpenTelemetry Collector.

2. Metrics collection

The next step is exporting metrics to the OpenTelemetry collector. OpenTelemetry Collector can collect telemetry data from multiple applications and process it all together. This step is not necessary - the application can send metrics directly to the Metrics Backend, especially if you are just trying it out.

3. Processing in the collector

OpenTelemetry Collector can transform metrics before sending them to the Metrics backend. Transformation involves filtering, adding, or deleting attributes, as well as renaming metrics. You can write your own processor if you have a specific need.

4. Export to the Backend

Finally, OpenTelemetry Collector exports your metrics to the Metrics Backend.

How OpenTelemetry instrumentation generates Metrics

For the server application

The exact options for generating metrics depend on the chosen programming language, but the OpenTelemetry community maintains a unified approach. Opentelemetry instrumentation is a set of libraries, SDKs, and API for generating metrics and other telemetry data.

Two kinds of instrumentation are generally available:

• Code instrumentation. Check the development status and availability for your language here.

• Zero-code instrumentation. For some programming languages, you can instrument your application without touching the code. Check the availability of zero-code instrumentation for your language here. It means that once Zero-code instrumentation is configured, some common use-case metrics work out of the box, and it can also mean that you can use a metric-generation library integrated with OpenTelemetry to produce OpenTelemetry metrics. An example of such a library is Java Micrometers.

In a generic scenario, instrumentation generates metrics and other telemetry data and sends them to the OpenTelemetry collector using the OpenTelemetry (OTLP) protocol. At this stage, metrics contain information, defined in the OpenTelemetry data model.

OpenTelemetry instrumentation has flexible configuration options; in most cases, it utilizes environment variables that follow the same naming specifications across multiple programming languages. Check the OpenTelemetry Exporter Configuration and Environment variable specification to find out more.

Other platforms

OpenTelemetry also has a set of instrumentations for specific platforms, which don’t run as a server application, such as:

• AWS lambda

• Kubernetes infrastructure

• Client-side application

How OpenTelemetry collector collects Metrics

OpenTelemetry Collector collects metrics from multiple applications. The canonical approach is to collect telemetry data using otlpreceiver via the OTLP protocol, mentioned earlier; however, in principle, it can collect metrics in any format if a corresponding receiver exists.

An example of such a receiver is Prometheus-Receiver. Instead of instrumenting your application, you can implement the exposure of Prometheus metrics in it, and configure the OpenTelemetry collector to collect these Prometheus metrics via the Prometheus protocol. Several receivers exist in the opentelemetry-collector-contrib repository, and you can also implement your own receiver if needed.

How OpenTelemetry collector processes Metrics

The OpenTelemetry collector can transform metrics using processors.

Some useful processors include:

• k8sattributesprocessor: enrich telemetry data with Kubernetes information, such as cluster name, region, deployment name, etc.

• filterprocessor: filter out incorrect data

• attributeprocessor: add, rename, and delete metric attributes

You can also write a custom processor if you have specific needs.

Destinations where metrics can be exported

An OpenTelemetry collector can write metrics into any metric storage that supports OTLP format, such as. But, as always, it is not limited to that. It can export data in any format if a corresponding processor exists.

Example of metric exporters:

• prometheusremotewriteexporter: can write metrics to a storage that supports the Prometheus Remote Write protocol, such as Uptrace, VictoriaMetrics, Thanos, and others

• influxdbexporter: write metrics to InfluxDb

• clickhoseexporter: write metrics to clickhouse

Conclusion

This is a brief overview of how metrics work in OpenTelemetry, and you are now ready to try it out in real- world scenarios. Do you plan to set up OpenTelemetry Collector in production? Learn how to choose the OpenTelemetry collector distribution

Before understanding the logic of distributions, let’s revisit the architecture of the OpenTelemetry collector. Feel free to skip this section if you are already familiar with it.

Architecture of the OpenTelemetry Collector

The primary purpose of the collector is to receive, process, and export OpenTelemetry data (also referred to as signals) into the pipeline. The collector achieves these goals by combining components for:

• Receiving data

A collector can receive data from various sources, such as another Opentelemetry Collector, application Opentelemetry instrumentation, Kafka queue, and many others, by using corresponding components.

• Processing data

Between receiving and exporting data, a collector can process it using various processors. For example, it can enrich the data with attributeprocessor, sample data with tailingsampleprocessor, combine data into batches with batchprocessors for optimizing performance, and much more.

• Exporting data

Similarly to receiving data, an OpenTelemetry Collector can use various components for sending data to multiple destinations, such as another collector, Observability Backend, Kafka, and more.

• Additional purposes

Additionally, a collector can utilize connectors to connect multiple pipelines and extensions for features that do not directly relate to data processing. We will not discuss them in this manual, but the logic of using them is the same as for other components.

You can configure pipelines in OTEL Collector by combining multiple receivers, processors, and exporters, as shown in the diagram above. Within a single collector, you have the flexibility to use any number of components to create multiple pipelines, as long as these components are part of the collector build.

For example, you can create a pipeline inside the OpenTelemetry collector, which

1. Receives traces from applications over the OTLP protocol using otlpreceiver

2. Enriches them with attributes from the Kubernetes cluster using k8sattributesprocessor

3. Samples them based on probability using probalisticsampleprocessor

4. Combines in batches using batchprocessor

5. And exports results to Elastic backend using elasticsearchexporter

In the same collector, you can create another pipeline, which

1. Reads Kubernetes logs using k8slogreceiver

2. Enriches them with attributes from the Kubernetes cluster using k8sattributesprocessor

3. Sample them based on probability using probalisticsampleprocessor

4. Combine in batches using batchprocessor

5. And exports results to Elastic backend using elasticsearchexporter

For such a case, the collector build must include otlpreceiver, k8slogreceiver, k8sattributesprocessor, probalisticsampleprocessor, batchprocessor, and elasticsearchexporter, with minimal possible overhead.

The OTEL community and third-party vendors maintain a wide range of components that, when assembled, form a distribution.

What is a distribution?

The OpenTelemetry community defines a distribution as:

A distribution is a customized version of an OpenTelemetry component. A distribution is a wrapper around an upstream OpenTelemetry repository with some customizations.

Put simply, the distribution is a customized version of the OpenTelemetry collector, which may include:

• a custom set of components,

• custom default settings,

• additional tests,

• performance tunings,

• and a few other specifics.

The central part of a distribution is the set of included components. Common components are typically located in the otel-collector-contrib and opentelemetry collector repositories, while vendor-specific ones may reside in third-party repositories. The distribution pulls the required ones from any of the available repositories, as demonstrated in the diagram below.

More components in distribution means more available features to use, but, at the same time, including excessive components increases the binary size, which leads to higher resource consumption and increased security risks. Therefore, the ideal distribution should include the minimum amount of excessive components.

When deciding on the OpenTelemetry distribution for your organization, you have a few options:

• Use one of the pre-built OpenTelemetry community distributions

• Use third-party distribution maintained by Vendors

• Build a custom collector

Let’s examine each option in detail.

Opentelemetry community distributions

The OpenTelemetry community maintains a few pre-built distributions. It means that the community owns the building pipeline and version releases.

• OpenTelemetry collector contrib distro

This distribution contains all the components from both the otel-collector-contrib and opentelemetry collector repositories. It provides a convenient way to explore the full range of features with minimal effort, but it may be too resource-intensive for production. Generally, the OpenTelemetry community does not recommend the otel collector contrib distro for real-time use.

• Opentelemetry collector core distribution

This distribution is a ‘classic’ distribution. It includes the compact set of components to work with OTLP protocol, Kafka, Zipkin, Prometheus, Jaeger, and a few other features.

Example. The collector pipeline, when core distribution may be a fit:

The collector collects Metrics and traces via OTLP, enriches them with additional attributes, applies probabilistic sampling, and exports to Kafka. Additionally, it may export metrics to Prometheus-based metrics storage and traces to Jaeger.

If the collector also needs to collect events from the Kubernetes API*, the Otel collector core distro* won’t fit*.*

• Opentelemetry Collector EBPF profiling distribution

The goal of this distribution is to collect data about processes running on the system. It contains a limited set of components for collecting K8s metadata, EBPF data, and exporting them to a file or via OTLP.

Example. The collector pipeline, when EBPF profiling distribution may be a fit:

The collector runs on a Kubernetes node, collects EBPF data, enriches it with k8s attributes, and sends it via OTLP to another OTLP collector. But you can’t include components for collecting metrics or traces via OTLP.

• Opentelemetry Collector Kubernetes distro

This is the distribution optimized for Kubernetes. It contains components to collect information from journald, K8s events, fluentd, OTLP, and others. It also includes a few basic exporters, such as OTLP exporter, fileexporter, load balancer exporter. The full list of components is available in the manifest.

Example. The collector pipeline, when the Kubernetes distro may be a fit:

The collector collects Metrics, traces, and logs via OTLP, enriches them with Kubernetes attributes, applies probabilistic sampling, and additionally collects Kubernetes events and logs from Fluentd. The collector exports processed data over OTLP to another Opentelemetry collector.

The distribution won’t fit if the collector exports data to Kafka instead of OTLP or exports to the Zipkin Backend.

• Opentelemetry Collector OTLP distro

It is the most minimalistic distribution. It contains modules for receiving and exporting data over the OTLP protocol. It is a good option to serve as a proxy, protocol translation, and a few other scenarios.

Example. The collector pipeline, when OTLP distro may be a fit:

The collector runs as a sidecar for a Kubernetes application, collects data from this application over OTLP, and sends it to another OTEL collector.

Vendors distributions

Some organizations maintain and distribute their own versions of OpenTelemetry collectors. Vendor distributions often offer additional features, such as adapting the collector for easier integration with vendor software, easier configuration, optimized performance, and others.

For example, Elastic EDOT Distribution of OpenTelemetry Collector includes components from OpenTelemetry Collector Core Distro and Elastic Collector components. It offers additional features that can be useful for Elastic users.

Other examples include DataDog distribution, AWS Distro distribution, and others.

Custom Build distribution

For the highest level of control and flexibility, consider building a custom collector. By building your own collector, you gain full control over which components to include, which default configuration to use, when to upgrade the version, whether to include custom components, and so on. As a downside, you have to handle the version upgrades on your own if you choose this option.

Summary

We can summarize all the benefits and drawbacks of each option in the table below:

Conclusion

The community and third-party vendors maintain different OpenTelemetry collector distributions for various use cases. However, you don’t have to limit yourself to choosing one distribution for solving all observability pipeline tasks. Your infrastructure may utilize multiple distributions simultaneously and form a solid pipeline together. For example, you can use the k8s distribution for collecting k8s-specific information, the EBPF distribution for EBPF data, the Elastic EDOT distribution to process data for elastic backends, and so on. If none of the pre-built distributions fit your needs, or for the maximal flexibility and control, you can build a custom OpenTelemetry collector distribution.

By understanding these pros and cons, you can select the distribution that best aligns with your infrastructure's needs.