
---------------------------------------------------------------------

ECS Logging Python Reference

---------------------------------------------------------------------
---------------------------------------------------------------------

Table of Contents

1. Introduction
2. Installation


---------------------------------------------------------------------

Chapter 1. Introduction

---------------------------------------------------------------------

ECS loggers are formatter/encoder plugins for your favorite logging
libraries. They make it easy to format your logs into ECS-compatible
JSON.

Tip

Want to learn more about ECS, ECS logging, and other available
language plugins?

Ready to jump into ecs-logging-python? Get started.


---------------------------------------------------------------------

Chapter 2. Installation

---------------------------------------------------------------------

$ python -m pip install ecs-logging

Getting Started

ecs-logging-python has formatters for the standard library logging
module and the structlog package.

Standard Library logging Module

import logging
import ecs_logging

# Get the Logger
logger = logging.getLogger("app")
logger.setLevel(logging.DEBUG)

# Add an ECS formatter to the Handler
handler = logging.StreamHandler()
handler.setFormatter(ecs_logging.StdlibFormatter())
logger.addHandler(handler)

# Emit a log!
logger.debug("Example message!", extra={"http.request.method": "get"})

{
    "@timestamp": "2020-03-20T18:11:37.895Z",
    "log.level": "debug",
    "message": "Example message!",
    "ecs": {
        "version": "1.6.0"
    },
    "http": {
      "request": {
        "method": "get"
      }
    },
    "log": {
        "logger": "app",
        "origin": {
            "file": {
                "line": 14,
                "name": "test.py"
            },
            "function": "func"
        },
        "original": "Example message!"
    }
}

Excluding Fields

You can exclude fields from being collected by using the
exclude_fields option in the StdlibFormatter constructor:

from ecs_logging import StdlibFormatter

formatter = StdlibFormatter(
    exclude_fields=[
        # You can specify individual fields to ignore:
        "log.original",
        # or you can also use prefixes to ignore
        # whole categories of fields:
        "process",
        "log.origin",
    ]
)

Limiting Stack Traces

The StdlibLogger automatically gathers exc_info into ECS error.*
fields. If you’d like to control the number of stack frames that are
included in error.stack_trace you can use the stack_trace_limit
parameter (by default all frames are collected):

from ecs_logging import StdlibFormatter

formatter = StdlibFormatter(
    # Only collects 3 stack frames
    stack_trace_limit=3,
)
formatter = StdlibFormatter(
    # Disable stack trace collection
    stack_trace_limit=0,
)

Structlog Example

Note that the structlog processor should be the last processor in the
list, as it handles the conversion to JSON as well as the ECS field
enrichment.

import structlog
import ecs_logging

# Configure Structlog
structlog.configure(
    processors=[ecs_logging.StructlogFormatter()],
    wrapper_class=structlog.BoundLogger,
    context_class=dict,
    logger_factory=structlog.PrintLoggerFactory(),
)

# Get the Logger
logger = structlog.get_logger("app")

# Add additional context
logger = logger.bind(**{
    "http": {
        "version": "2",
        "request": {
            "method": "get",
            "bytes": 1337,
        },
    },
    "url": {
        "domain": "example.com",
        "path": "/",
        "port": 443,
        "scheme": "https",
        "registered_domain": "example.com",
        "top_level_domain": "com",
        "original": "https://example.com",
    }
})

# Emit a log!
logger.debug("Example message!")

{
  "@timestamp": "2020-03-26T13:08:11.728Z",
  "ecs": {
    "version": "1.6.0"
  },
  "http": {
    "request": {
      "bytes": 1337,
      "method": "get"
    },
    "version": "2"
  },
  "log": {
    "level": "debug"
  },
  "message": "Example message!",
  "url": {
    "domain": "example.com",
    "original": "https://example.com",
    "path": "/",
    "port": 443,
    "registered_domain": "example.com",
    "scheme": "https",
    "top_level_domain": "com"
  }
}

Elastic APM Log Correlation

from the Elastic APM Python agent in order to

You can also quickly turn on ECS-formatted logs in your python app by
setting in the Elastic APM Python agent.

Install Filebeat

The best way to collect the logs once they are ECS-formatted is with
Filebeat:

