

Advanced topics [advanced-topics]

- Instrumenting custom code
- Sanitizing data
- How the Agent works
- Run Tests Locally

Aiohttp Server support [aiohttp-server-support]

Getting Elastic APM set up for your Aiohttp Server project is easy, and
there are various ways you can tweak it to fit to your needs.

Installation [aiohttp-server-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add elastic-apm to your project’s requirements.txt file.

Setup [aiohttp-server-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, the
application’s settings, or as initialization arguments.

You can find a list of all available settings in the Configuration page.

To initialize the agent for your application using environment
variables:

    from aiohttp import web

    from elasticapm.contrib.aiohttp import ElasticAPM

    app = web.Application()

    apm = ElasticAPM(app)

To configure the agent using ELASTIC_APM in your application’s settings:

    from aiohttp import web

    from elasticapm.contrib.aiohttp import ElasticAPM

    app = web.Application()

    app['ELASTIC_APM'] = {
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
    }
    apm = ElasticAPM(app)

Usage [aiohttp-server-usage]

Once you have configured the agent, it will automatically track
transactions and capture uncaught exceptions within aiohttp.

Capture an arbitrary exception by calling capture_exception:

    try:
        1 / 0
    except ZeroDivisionError:
        apm.client.capture_exception()

Log a generic message with capture_message:

    apm.client.capture_message('hello, world!')

Performance metrics [aiohttp-server-performance-metrics]

If you’ve followed the instructions above, the agent has already
installed our middleware. This will measure response times, as well as
detailed performance data for all supported technologies.

due to the fact that asyncio drivers are usually separate from their
synchronous counterparts, specific instrumentation is needed for all
drivers. The support for asynchronous drivers is currently quite
limited.

Ignoring specific routes [aiohttp-server-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific routes. The list given should be a list of regular
expressions which are matched against the transaction name:

    app['ELASTIC_APM'] = {
        # ...
        'TRANSACTIONS_IGNORE_PATTERNS': ['^OPTIONS ', '/api/']
        # ...
    }

This would ignore any requests using the OPTIONS method and any requests
containing /api/.

Supported aiohttp and Python versions [supported-aiohttp-and-python-versions]

A list of supported aiohttp and Python versions can be found on our
Supported Technologies page.

Elastic APM only supports asyncio when using Python 3.7+

API reference [api]

The Elastic APM Python agent has several public APIs. Most of the public
API functionality is not needed when using one of our supported
frameworks, but they allow customized usage.

Client API [client-api]

The public Client API consists of several methods on the Client class.
This API can be used to track exceptions and log messages, as well as to
mark the beginning and end of transactions.

Instantiation [client-api-init]

{applies_to} apm_agent_python: ga 1.0.0

To create a Client instance, import it and call its constructor:

    from elasticapm import Client

    client = Client({'SERVICE_NAME': 'example'}, **defaults)

- config: A dictionary, with key/value configuration. For the possible
  configuration keys, see Configuration.
- **defaults: default values for configuration. These can be omitted in
  most cases, and take the least precedence.

framework integrations like Django and Flask instantiate the client
automatically.

elasticapm.get_client() [api-get-client]

{applies_to} apm_agent_python: ga 6.1.0

Retrieves the Client singleton. This is useful for many framework
integrations, where the client is instantiated automatically.

    client = elasticapm.get_client()
    client.capture_message('foo')

Errors [error-api]

Client.capture_exception() [client-api-capture-exception]

{applies_to} apm_agent_python: ga 1.0.0

handled added in v2.0.0.

Captures an exception object:

    try:
        x = int("five")
    except ValueError:
        client.capture_exception()

- exc_info: A (type, value, traceback) tuple as returned by
  sys.exc_info(). If not provided, it will be captured automatically.
- date: A datetime.datetime object representing the occurrence time of
  the error. If left empty, it defaults to datetime.datetime.utcnow().
- context: A dictionary with contextual information. This dictionary
  must follow the Context schema definition.
- custom: A dictionary of custom data you want to attach to the event.
- handled: A boolean to indicate if this exception was handled or not.

Returns the id of the error as a string.

Client.capture_message() [client-api-capture-message]

{applies_to} apm_agent_python: ga 1.0.0

Captures a message with optional added contextual data. Example:

    client.capture_message('Billing process succeeded.')

- message: The message as a string.

- param_message: Alternatively, a parameterized message as a dictionary.
  The dictionary contains two values: message, and params. This allows
  the APM Server to group messages together that share the same
  parameterized message. Example:

      client.capture_message(param_message={
          'message': 'Billing process for %s succeeded. Amount: %s',
          'params': (customer.id, order.total_amount),
      })

- stack: If set to True (the default), a stacktrace from the call site
  will be captured.

- exc_info: A (type, value, traceback) tuple as returned by
  sys.exc_info(). If not provided, it will be captured automatically, if
  capture_message() was called in an except block.

- date: A datetime.datetime object representing the occurrence time of
  the error. If left empty, it defaults to datetime.datetime.utcnow().

- context: A dictionary with contextual information. This dictionary
  must follow the Context schema definition.

- custom: A dictionary of custom data you want to attach to the event.

Returns the id of the message as a string.

Either the message or the param_message argument is required.

Transactions [transaction-api]

Client.begin_transaction() [client-api-begin-transaction]

{applies_to} apm_agent_python: ga 1.0.0

trace_parent support added in v5.6.0.

Begin tracking a transaction. Should be called e.g. at the beginning of
a request or when starting a background task. Example:

    client.begin_transaction('processors')

- transaction_type: (required) A string describing the type of the
  transaction, e.g. 'request' or 'celery'.
- trace_parent: (optional) A TraceParent object. See TraceParent
  generation.
- links: (optional) A list of TraceParent objects to which this
  transaction is causally linked.

Client.end_transaction() [client-api-end-transaction]

{applies_to} apm_agent_python: ga 1.0.0

End tracking the transaction. Should be called e.g. at the end of a
request or when ending a background task. Example:

    client.end_transaction('myapp.billing_process', processor.status)

- name: (optional) A string describing the name of the transaction,
  e.g. process_order. This is typically the name of the view/controller
  that handles the request, or the route name.
- result: (optional) A string describing the result of the transaction.
  This is typically the HTTP status code, or e.g. 'success' for a
  background task.

if name and result are not set in the end_transaction() call, they have
to be set beforehand by calling elasticapm.set_transaction_name() and
elasticapm.set_transaction_result() during the transaction.

TraceParent [traceparent-api]

Transactions can be started with a TraceParent object. This creates a
transaction that is a child of the TraceParent, which is essential for
distributed tracing.

elasticapm.trace_parent_from_string() [api-traceparent-from-string]

{applies_to} apm_agent_python: ga 5.6.0

Create a TraceParent object from the string representation generated by
TraceParent.to_string():

    parent = elasticapm.trace_parent_from_string('00-03d67dcdd62b7c0f7a675424347eee3a-5f0e87be26015733-01')
    client.begin_transaction('processors', trace_parent=parent)

- traceparent_string: (required) A string representation of a
  TraceParent object.

elasticapm.trace_parent_from_headers() [api-traceparent-from-headers]

{applies_to} apm_agent_python: ga 5.6.0

Create a TraceParent object from HTTP headers (usually generated by
another Elastic APM agent):

    parent = elasticapm.trace_parent_from_headers(headers_dict)
    client.begin_transaction('processors', trace_parent=parent)

- headers: (required) HTTP headers formed as a dictionary.

elasticapm.get_trace_parent_header() [api-traceparent-get-header]

{applies_to} apm_agent_python: ga 5.10.0

Return the string representation of the current transaction TraceParent
object:

    elasticapm.get_trace_parent_header()

Other APIs [api-other]

elasticapm.instrument() [api-elasticapm-instrument]

{applies_to} apm_agent_python: ga 1.0.0

Instruments libraries automatically. This includes a wide range of
standard library and 3rd party modules. A list of instrumented modules
can be found in elasticapm.instrumentation.register. This function
should be called as early as possible in the startup of your
application. For supported frameworks, this is called automatically.
Example:

    import elasticapm

    elasticapm.instrument()

elasticapm.set_transaction_name() [api-set-transaction-name]

{applies_to} apm_agent_python: ga 1.0.0

Set the name of the current transaction. For supported frameworks, the
transaction name is determined automatically, and can be overridden
using this function. Example:

    import elasticapm

    elasticapm.set_transaction_name('myapp.billing_process')

- name: (required) A string describing name of the transaction
- override: if True (the default), overrides any previously set
  transaction name. If False, only sets the name if the transaction name
  hasn’t already been set.

elasticapm.set_transaction_result() [api-set-transaction-result]

{applies_to} apm_agent_python: ga 2.2.0

Set the result of the current transaction. For supported frameworks, the
transaction result is determined automatically, and can be overridden
using this function. Example:

    import elasticapm

    elasticapm.set_transaction_result('SUCCESS')

- result: (required) A string describing the result of the transaction,
  e.g. HTTP 2xx or SUCCESS
- override: if True (the default), overrides any previously set result.
  If False, only sets the result if the result hasn’t already been set.

elasticapm.set_transaction_outcome() [api-set-transaction-outcome]

{applies_to} apm_agent_python: ga 5.9.0

Sets the outcome of the transaction. The value can either be "success",
"failure" or "unknown". This should only be called at the end of a
transaction after the outcome is determined.

The outcome is used for error rate calculations. success denotes that a
transaction has concluded successful, while failure indicates that the
transaction failed to finish successfully. If the outcome is set to
unknown, the transaction will not be included in error rate
calculations.

For supported web frameworks, the transaction outcome is set
automatically if it has not been set yet, based on the HTTP status code.
A status code below 500 is considered a success, while any value of 500
or higher is counted as a failure.

If your transaction results in an HTTP response, you can alternatively
provide the HTTP status code.

While the outcome and result field look very similar, they serve
different purposes. Other than the result field, which canhold an
arbitrary string value, outcome is limited to three different values,
"success", "failure" and "unknown". This allows the APM app to perform
error rate calculations on these values.

Example:

    import elasticapm

    elasticapm.set_transaction_outcome("success")

    # Using an HTTP status code
    elasticapm.set_transaction_outcome(http_status_code=200)

    # Using predefined constants:

    from elasticapm.conf.constants import OUTCOME

    elasticapm.set_transaction_outcome(OUTCOME.SUCCESS)
    elasticapm.set_transaction_outcome(OUTCOME.FAILURE)
    elasticapm.set_transaction_outcome(OUTCOME.UNKNOWN)

- outcome: One of "success", "failure" or "unknown". Can be omitted if
  http_status_code is provided.
- http_status_code: if the transaction represents an HTTP response, its
  status code can be provided to determine the outcome automatically.
- override: if True (the default), any previously set outcome will be
  overridden. If False, the outcome will only be set if it was not set
  before.

elasticapm.get_transaction_id() [api-get-transaction-id]

{applies_to} apm_agent_python: ga 5.2.0

Get the id of the current transaction. Example:

    import elasticapm

    transaction_id = elasticapm.get_transaction_id()

elasticapm.get_trace_id() [api-get-trace-id]

{applies_to} apm_agent_python: ga 5.2.0

Get the trace_id of the current transaction’s trace. Example:

    import elasticapm

    trace_id = elasticapm.get_trace_id()

elasticapm.get_span_id() [api-get-span-id]

{applies_to} apm_agent_python: ga 5.2.0

Get the id of the current span. Example:

    import elasticapm

    span_id = elasticapm.get_span_id()

elasticapm.set_custom_context() [api-set-custom-context]

{applies_to} apm_agent_python: ga 2.0.0

Attach custom contextual data to the current transaction and errors.
Supported frameworks will automatically attach information about the
HTTP request and the logged in user. You can attach further data using
this function.

Before using custom context, ensure you understand the different types
of metadata that are available.

Example:

    import elasticapm

    elasticapm.set_custom_context({'billing_amount': product.price * item_count})

- data: (required) A dictionary with the data to be attached. This
  should be a flat key/value dict object.

., *, and " are invalid characters for key names and will be replaced
with _.

Errors that happen after this call will also have the custom context
attached to them. You can call this function multiple times, new context
data will be merged with existing data, following the update() semantics
of Python dictionaries.

elasticapm.set_user_context() [api-set-user-context]

{applies_to} apm_agent_python: ga 2.0.0

Attach information about the currently logged in user to the current
transaction and errors. Example:

    import elasticapm

    elasticapm.set_user_context(username=user.username, email=user.email, user_id=user.id)

- username: The username of the logged in user
- email: The email of the logged in user
- user_id: The unique identifier of the logged in user, e.g. the primary
  key value

Errors that happen after this call will also have the user context
attached to them. You can call this function multiple times, new user
data will be merged with existing data, following the update() semantics
of Python dictionaries.

elasticapm.capture_span [api-capture-span]

{applies_to} apm_agent_python: ga 4.1.0

Capture a custom span. This can be used either as a function decorator
or as a context manager (in a with statement). When used as a decorator,
the name of the span will be set to the name of the function. When used
as a context manager, a name has to be provided.

    import elasticapm

    @elasticapm.capture_span()
    def coffee_maker(strength):
        fetch_water()

        with elasticapm.capture_span('near-to-machine', labels={"type": "arabica"}):
            insert_filter()
            for i in range(strength):
                pour_coffee()

            start_drip()

        fresh_pots()

- name: The name of the span. Defaults to the function name if used as a
  decorator.
- span_type: (optional) The type of the span, usually in a dot-separated
  hierarchy of type, subtype, and action, e.g. db.mysql.query.
  Alternatively, type, subtype and action can be provided as three
  separate arguments, see span_subtype and span_action.
- skip_frames: (optional) The number of stack frames to skip when
  collecting stack traces. Defaults to 0.
- leaf: (optional) if True, all spans nested below this span will be
  ignored. Defaults to False.
- labels: (optional) a dictionary of labels. Keys must be strings,
  values can be strings, booleans, or numerical (int, float,
  decimal.Decimal). Defaults to None.
- span_subtype: (optional) subtype of the span, e.g. name of the
  database. Defaults to None.
- span_action: (optional) action of the span, e.g. query. Defaults to
  None.
- links: (optional) A list of TraceParent objects to which this span is
  causally linked.

elasticapm.async_capture_span [api-async-capture-span]

{applies_to} apm_agent_python: ga 5.4.0

Capture a custom async-aware span. This can be used either as a function
decorator or as a context manager (in an async with statement). When
used as a decorator, the name of the span will be set to the name of the
function. When used as a context manager, a name has to be provided.

    import elasticapm

    @elasticapm.async_capture_span()
    async def coffee_maker(strength):
        await fetch_water()

        async with elasticapm.async_capture_span('near-to-machine', labels={"type": "arabica"}):
            await insert_filter()
            async for i in range(strength):
                await pour_coffee()

            start_drip()

        fresh_pots()

- name: The name of the span. Defaults to the function name if used as a
  decorator.
- span_type: (optional) The type of the span, usually in a dot-separated
  hierarchy of type, subtype, and action, e.g. db.mysql.query.
  Alternatively, type, subtype and action can be provided as three
  separate arguments, see span_subtype and span_action.
- skip_frames: (optional) The number of stack frames to skip when
  collecting stack traces. Defaults to 0.
- leaf: (optional) if True, all spans nested below this span will be
  ignored. Defaults to False.
- labels: (optional) a dictionary of labels. Keys must be strings,
  values can be strings, booleans, or numerical (int, float,
  decimal.Decimal). Defaults to None.
- span_subtype: (optional) subtype of the span, e.g. name of the
  database. Defaults to None.
- span_action: (optional) action of the span, e.g. query. Defaults to
  None.
- links: (optional) A list of TraceParent objects to which this span is
  causally linked.

asyncio is only supported for Python 3.7+.

elasticapm.label() [api-label]

{applies_to} apm_agent_python: ga 5.0.0

Attach labels to the the current transaction and errors.

Before using custom labels, ensure you understand the different types of
metadata that are available.

Example:

    import elasticapm

    elasticapm.label(ecommerce=True, dollar_value=47.12)

Errors that happen after this call will also have the labels attached to
them. You can call this function multiple times, new labels will be
merged with existing labels, following the update() semantics of Python
dictionaries.

Keys must be strings, values can be strings, booleans, or numerical
(int, float, decimal.Decimal) ., *, and " are invalid characters for
label names and will be replaced with _.

Avoid defining too many user-specified labels. Defining too many unique
fields in an index is a condition that can lead to a mapping explosion.

ASGI Middleware [asgi-middleware]

This functionality is in technical preview and may be changed or removed
in a future release. Elastic will work to fix any issues, but features
in technical preview are not subject to the support SLA of official GA
features.

Incorporating Elastic APM into your ASGI-based project only requires a
few easy steps.

Several ASGI frameworks are supported natively. Please check Supported
Technologies for more information

Installation [asgi-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add elastic-apm to your project’s requirements.txt file.

Setup [asgi-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, or as
initialization arguments.

You can find a list of all available settings in the Configuration page.

To set up the APM agent, wrap your ASGI app with the
ASGITracingMiddleware:

    from elasticapm.contrib.asgi import ASGITracingMiddleware

    app = MyGenericASGIApp()  # depending on framework

    app = ASGITracingMiddleware(app)

Make sure to call elasticapm.set_transaction_name() with an appropriate
transaction name in all your routes.

Currently, the agent doesn’t support automatic capturing of exceptions.
You can follow progress on this issue on Github.

Supported Python versions [supported-python-versions]

A list of supported Python versions can be found on our Supported
Technologies page.

Elastic APM only supports asyncio when using Python 3.7+

Monitoring Azure Functions [azure-functions-support]

Prerequisites [_prerequisites_2]

You need an APM Server to which you can send APM data. Follow the APM
Quick start if you have not set one up yet. For the best-possible
performance, we recommend setting up APM on {{ecloud}} in the same Azure
region as your Azure Functions app.

Currently, only HTTP and timer triggers are supported. Other trigger
types may be captured as well, but the amount of captured contextual
data may differ.

Step 1: Enable Worker Extensions [_step_1_enable_worker_extensions]

Elastic APM uses Worker Extensions to instrument Azure Functions. This
feature is not enabled by default, and must be enabled in your Azure
Functions App. Please follow the instructions in the Azure docs.

Once you have enabled Worker Extensions, these two lines of code will
enable Elastic APM’s extension:

    from elasticapm.contrib.serverless.azure import ElasticAPMExtension

    ElasticAPMExtension.configure()

Put them somewhere at the top of your Python file, before the function
definitions.

Step 2: Install the APM Python Agent [_step_2_install_the_apm_python_agent]

You need to add elastic-apm as a dependency for your Functions app.
Simply add elastic-apm to your requirements.txt file. We recommend
pinning the version to the current newest version of the agent, and
periodically updating the version.

Step 3: Configure APM on Azure Functions [_step_3_configure_apm_on_azure_functions]

The APM Python agent is configured through App Settings. These are then
picked up by the agent as environment variables.

For the minimal configuration, you will need the ELASTIC_APM_SERVER_URL
to set the destination for APM data and a ELASTIC_APM_SECRET_TOKEN. If
you prefer to use an APM API key instead of the APM secret token, use
the ELASTIC_APM_API_KEY environment variable instead of
ELASTIC_APM_SECRET_TOKEN in the following example configuration.

    $ az functionapp config appsettings set --settings ELASTIC_APM_SERVER_URL=https://example.apm.northeurope.azure.elastic-cloud.com:443
    $ az functionapp config appsettings set --settings ELASTIC_APM_SECRET_TOKEN=verysecurerandomstring

You can optionally fine-tune the Python agent.

That’s it; Once the agent is installed and working, spans will be
captured for supported technologies. You can also use capture_span to
capture custom spans, and you can retrieve the Client object for
capturing exceptions/messages using get_client.

Configuration [configuration]

To adapt the Elastic APM agent to your needs, configure it using
environment variables or framework-specific configuration.

You can either configure the agent by setting environment variables:

    ELASTIC_APM_SERVICE_NAME=foo python manage.py runserver

or with inline configuration:

    apm_client = Client(service_name="foo")

or by using framework specific configuration e.g. in your Django
settings.py file:

    ELASTIC_APM = {
        "SERVICE_NAME": "foo",
    }

The precedence is as follows:

- Central configuration (supported options are marked with [dynamic
  config])
- Environment variables
- Inline configuration
- Framework-specific configuration
- Default value

Dynamic configuration [dynamic-configuration]

Configuration options marked with the [dynamic config] badge can be
changed at runtime when set from a supported source.

The Python Agent supports Central configuration, which allows you to
fine-tune certain configurations from in the APM app. This feature is
enabled in the Agent by default with central_config.

Django [django-configuration]

To configure Django, add an ELASTIC_APM dictionary to your settings.py:

    ELASTIC_APM = {
        'SERVICE_NAME': 'my-app',
        'SECRET_TOKEN': 'changeme',
    }

Flask [flask-configuration]

To configure Flask, add an ELASTIC_APM dictionary to your app.config:

    app.config['ELASTIC_APM'] = {
        'SERVICE_NAME': 'my-app',
        'SECRET_TOKEN': 'changeme',
    }

    apm = ElasticAPM(app)

Core options [core-options]

service_name [config-service-name]

  ---------------------------------------------------------------------------------------
  Environment                Django/Flask      Default                  Example
  -------------------------- ----------------- ------------------------ -----------------
  ELASTIC_APM_SERVICE_NAME   SERVICE_NAME      unknown-python-service   my-app

  ---------------------------------------------------------------------------------------

The name of your service. This is used to keep all the errors and
transactions of your service together and is the primary filter in the
Elastic APM user interface.

While a default is provided, it is essential that you override this
default with something more descriptive and unique across your
infrastructure.

The service name must conform to this regular expression:
^[a-zA-Z0-9 _-]+$. In other words, the service name must only contain
characters from the ASCII alphabet, numbers, dashes, underscores, and
spaces. It cannot be an empty string or whitespace-only.

server_url [config-server-url]

  Environment              Django/Flask   Default
  ------------------------ -------------- -------------------------
  ELASTIC_APM_SERVER_URL   SERVER_URL     'http://127.0.0.1:8200'

The URL for your APM Server. The URL must be fully qualified, including
protocol (http or https) and port. Note: Do not set this if you are
using APM in an AWS lambda function. APM Agents are designed to proxy
their calls to the APM Server through the lambda extension. Instead, set
ELASTIC_APM_LAMBDA_APM_SERVER. For more info, see AWS Lambda.

enabled [config-enabled]

  Environment           Django/Flask   Default
  --------------------- -------------- ---------
  ELASTIC_APM_ENABLED   ENABLED        true

Enable or disable the agent. When set to false, the agent will not
collect any data or start any background threads.

recording [config-recording]

[dynamic config]

  Environment             Django/Flask   Default
  ----------------------- -------------- ---------
  ELASTIC_APM_RECORDING   RECORDING      true

Enable or disable recording of events. If set to false, then the Python
agent does not send any events to the Elastic APM server, and
instrumentation overhead is minimized. The agent will continue to poll
the server for configuration changes.

Logging Options [logging-options]

log_level [config-log_level]

[dynamic config]

  Environment             Django/Flask   Default
  ----------------------- -------------- ---------
  ELASTIC_APM_LOG_LEVEL   LOG_LEVEL      

The logging.logLevel at which the elasticapm logger will log. The
available options are:

- "off" (sets logging.logLevel to 1000)
- "critical"
- "error"
- "warning"
- "info"
- "debug"
- "trace" (sets logging.log_level to 5)

Options are case-insensitive

Note that this option doesn’t do anything with logging handlers. In
order for any logs to be visible, you must either configure a handler
(logging.basicConfig will do this for you) or set log_file. This will
also override any log level your app has set for the elasticapm logger.

log_file [config-log_file]

  ------------------------------------------------------------------------------------------
  Environment            Django/Flask      Default           Example
  ---------------------- ----------------- ----------------- -------------------------------
  ELASTIC_APM_LOG_FILE   LOG_FILE          ""                "/var/log/elasticapm/log.txt"

  ------------------------------------------------------------------------------------------

This enables the agent to log to a file. This is disabled by default.
The agent will log at the logging.logLevel configured with log_level.
Use log_file_size to configure the maximum size of the log file. This
log file will automatically rotate.

Note that setting log_level is required for this setting to do anything.

If ecs_logging is installed, the logs will automatically be formatted as
ecs-compatible json.

log_file_size [config-log_file_size]

  Environment                 Django/Flask    Default   Example
  --------------------------- --------------- --------- ---------
  ELASTIC_APM_LOG_FILE_SIZE   LOG_FILE_SIZE   "50mb"    "100mb"

The size of the log file if log_file is set.

The agent always keeps one backup file when rotating, so the maximum
space that the log files will consume is twice the value of this
setting.

log_ecs_reformatting [config-log_ecs_reformatting]

  ----------------------------------------------------------------------------------
  Environment                        Django/Flask            Default
  ---------------------------------- ----------------------- -----------------------
  ELASTIC_APM_LOG_ECS_REFORMATTING   LOG_ECS_REFORMATTING    "off"

  ----------------------------------------------------------------------------------

This functionality is in technical preview and may be changed or removed
in a future release. Elastic will work to fix any issues, but features
in technical preview are not subject to the support SLA of official GA
features.

Valid options:

- "off"
- "override"

If ecs_logging is installed, setting this to "override" will cause the
agent to automatically attempt to enable ecs-formatted logging.

For base logging from the standard library, the agent will get the root
logger, find any attached handlers, and for each, set the formatter to
ecs_logging.StdlibFormatter().

If structlog is installed, the agent will override any configured
processors with ecs_logging.StructlogFormatter().

Note that this is a very blunt instrument that could have unintended
side effects. If problems arise, please apply these formatters manually
and leave this setting as "off". See the ecs_logging docs for more
information about using these formatters.

Also note that this setting does not facilitate shipping logs to
Elasticsearch. We recommend Filebeat for that purpose.

Other options [other-options]

transport_class [config-transport-class]

  -------------------------------------------------------------------------------------------
  Environment                   Django/Flask            Default
  ----------------------------- ----------------------- -------------------------------------
  ELASTIC_APM_TRANSPORT_CLASS   TRANSPORT_CLASS         elasticapm.transport.http.Transport

  -------------------------------------------------------------------------------------------

The transport class to use when sending events to the APM Server.

service_node_name [config-service-node-name]

  ---------------------------------------------------------------------------------------
  Environment                     Django/Flask        Default           Example
  ------------------------------- ------------------- ----------------- -----------------
  ELASTIC_APM_SERVICE_NODE_NAME   SERVICE_NODE_NAME   None              "redis1"

  ---------------------------------------------------------------------------------------

The name of the given service node. This is optional and if omitted, the
APM Server will fall back on system.container.id if available, and
host.name if necessary.

This option allows you to set the node name manually to ensure it is
unique and meaningful.

environment [config-environment]

  Environment               Django/Flask   Default   Example
  ------------------------- -------------- --------- --------------
  ELASTIC_APM_ENVIRONMENT   ENVIRONMENT    None      "production"

The name of the environment this service is deployed in,
e.g. “production” or “staging”.

Environments allow you to easily filter data on a global level in the
APM app. It’s important to be consistent when naming environments across
agents. See environment selector in the APM app for more information.

This feature is fully supported in the APM app in Kibana versions >=
7.2. You must use the query bar to filter for a specific environment in
versions prior to 7.2.

cloud_provider [config-cloud-provider]

  Environment                  Django/Flask     Default   Example
  ---------------------------- ---------------- --------- ---------
  ELASTIC_APM_CLOUD_PROVIDER   CLOUD_PROVIDER   "auto"    "aws"

This config value allows you to specify which cloud provider should be
assumed for metadata collection. By default, the agent will attempt to
detect the cloud provider or, if that fails, will use trial and error to
collect the metadata.

Valid options are "auto", "aws", "gcp", and "azure". If this config
value is set to "none", then no cloud metadata will be collected.

secret_token [config-secret-token]

  --------------------------------------------------------------------------------
  Environment                Django/Flask      Default           Example
  -------------------------- ----------------- ----------------- -----------------
  ELASTIC_APM_SECRET_TOKEN   SECRET_TOKEN      None              A random string

  --------------------------------------------------------------------------------

This string is used to ensure that only your agents can send data to
your APM Server. Both the agents and the APM Server have to be
configured with the same secret token. An example to generate a secure
secret token is:

    python -c "import secrets; print(secrets.token_urlsafe(32))"

Secret tokens only provide any security if your APM Server uses TLS.

api_key [config-api-key]

  Environment           Django/Flask   Default   Example
  --------------------- -------------- --------- -------------------------
  ELASTIC_APM_API_KEY   API_KEY        None      A base64-encoded string

This base64-encoded string is used to ensure that only your agents can
send data to your APM Server. The API key can be created in the
Applications UI.

API keys only provide any real security if your APM Server uses TLS.

service_version [config-service-version]

  -----------------------------------------------------------------------------------
  Environment                   Django/Flask      Default           Example
  ----------------------------- ----------------- ----------------- -----------------
  ELASTIC_APM_SERVICE_VERSION   SERVICE_VERSION   None              A string
                                                                    indicating the
                                                                    version of the
                                                                    deployed service

  -----------------------------------------------------------------------------------

A version string for the currently deployed version of the service. If
youre deploys are not versioned, the recommended value for this field is
the commit identifier of the deployed revision, e.g. the output of
git rev-parse HEAD.

framework_name [config-framework-name]

  ----------------------------------------------------------------------------
  Environment                  Django/Flask            Default
  ---------------------------- ----------------------- -----------------------
  ELASTIC_APM_FRAMEWORK_NAME   FRAMEWORK_NAME          Depending on framework

  ----------------------------------------------------------------------------

The name of the used framework. For Django and Flask, this defaults to
django and flask respectively, otherwise, the default is None.

framework_version [config-framework-version]

  -------------------------------------------------------------------------------
  Environment                     Django/Flask            Default
  ------------------------------- ----------------------- -----------------------
  ELASTIC_APM_FRAMEWORK_VERSION   FRAMEWORK_VERSION       Depending on framework

  -------------------------------------------------------------------------------

The version number of the used framework. For Django and Flask, this
defaults to the used version of the framework, otherwise, the default is
None.

filter_exception_types [config-filter-exception-types]

  -------------------------------------------------------------------------------------------------------------------------------------------
  Environment                          Django/Flask             Default           Example
  ------------------------------------ ------------------------ ----------------- -----------------------------------------------------------
  ELASTIC_APM_FILTER_EXCEPTION_TYPES   FILTER_EXCEPTION_TYPES   []                ['OperationalError', 'mymodule.SomeoneElsesProblemError']

  multiple values separated by commas,                                            
  without spaces                                                                  
  -------------------------------------------------------------------------------------------------------------------------------------------

A list of exception types to be filtered. Exceptions of these types will
not be sent to the APM Server.

transaction_ignore_urls [config-transaction-ignore-urls]

[dynamic config]

  --------------------------------------------------------------------------------------------------------------
  Environment                           Django/Flask              Default           Example
  ------------------------------------- ------------------------- ----------------- ----------------------------
  ELASTIC_APM_TRANSACTION_IGNORE_URLS   TRANSACTION_IGNORE_URLS   []                ['/api/ping', '/static/*']

  multiple values separated by commas,                                              
  without spaces                                                                    
  --------------------------------------------------------------------------------------------------------------

A list of URLs for which the agent should not capture any transaction
data.

Optionally, * can be used to match multiple URLs at once.

transactions_ignore_patterns [config-transactions-ignore-patterns]

  ----------------------------------------------------------------------------------------------------------------------------------
  Environment                                Django/Flask                   Default           Example
  ------------------------------------------ ------------------------------ ----------------- --------------------------------------
  ELASTIC_APM_TRANSACTIONS_IGNORE_PATTERNS   TRANSACTIONS_IGNORE_PATTERNS   []                ['^OPTIONS ', 'myviews.Healthcheck']

  multiple values separated by commas,                                                        
  without spaces                                                                              
  ----------------------------------------------------------------------------------------------------------------------------------

A list of regular expressions. Transactions with a name that matches any
of the configured patterns will be ignored and not sent to the APM
Server.

as the the name of the transaction can only be determined at the end of
the transaction, the agent might still cause overhead for transactions
ignored through this setting. If agent overhead is a concern, we
recommend transaction_ignore_urls instead.

server_timeout [config-server-timeout]

  Environment                  Django/Flask     Default
  ---------------------------- ---------------- ---------
  ELASTIC_APM_SERVER_TIMEOUT   SERVER_TIMEOUT   "5s"

A timeout for requests to the APM Server. The setting has to be provided
in duration format. If a request to the APM Server takes longer than the
configured timeout, the request is cancelled and the event (exception or
transaction) is discarded. Set to None to disable timeouts.

If timeouts are disabled or set to a high value, your app could
experience memory issues if the APM Server times out.

hostname [config-hostname]

  ------------------------------------------------------------------------------------------
  Environment            Django/Flask      Default                Example
  ---------------------- ----------------- ---------------------- --------------------------
  ELASTIC_APM_HOSTNAME   HOSTNAME          socket.gethostname()   app-server01.example.com

  ------------------------------------------------------------------------------------------

The host name to use when sending error and transaction data to the APM
Server.

auto_log_stacks [config-auto-log-stacks]

  Environment                   Django/Flask      Default
  ----------------------------- ----------------- ---------
  ELASTIC_APM_AUTO_LOG_STACKS   AUTO_LOG_STACKS   True
  set to "true" / "false"                         

If set to True (the default), the agent will add a stack trace to each
log event, indicating where the log message has been issued.

This setting can be overridden on an individual basis by setting the
extra-key stack:

    logger.info('something happened', extra={'stack': False})

collect_local_variables [config-collect-local-variables]

  ---------------------------------------------------------------------------------------
  Environment                           Django/Flask              Default
  ------------------------------------- ------------------------- -----------------------
  ELASTIC_APM_COLLECT_LOCAL_VARIABLES   COLLECT_LOCAL_VARIABLES   errors

  ---------------------------------------------------------------------------------------

Possible values: errors, transactions, all, off

The Elastic APM Python agent can collect local variables for stack
frames. By default, this is only done for errors.

Collecting local variables has a non-trivial overhead. Collecting local
variables for transactions in production environments can have adverse
effects for the performance of your service.

local_var_max_length [config-local-var-max-length]

  Environment                        Django/Flask           Default
  ---------------------------------- ---------------------- ---------
  ELASTIC_APM_LOCAL_VAR_MAX_LENGTH   LOCAL_VAR_MAX_LENGTH   200

When collecting local variables, they will be converted to strings. This
setting allows you to limit the length of the resulting string.

local_var_list_max_length [config-local-list-var-max-length]

  --------------------------------------- --------------------------- -----------------------
  Environment                             Django/Flask                Default

  ELASTIC_APM_LOCAL_VAR_LIST_MAX_LENGTH   LOCAL_VAR_LIST_MAX_LENGTH   10
  --------------------------------------- --------------------------- -----------------------

This setting allows you to limit the length of lists in local variables.

local_var_dict_max_length [config-local-dict-var-max-length]

  -------------------------------------------------------------------------------------------
  Environment                             Django/Flask                Default
  --------------------------------------- --------------------------- -----------------------
  ELASTIC_APM_LOCAL_VAR_DICT_MAX_LENGTH   LOCAL_VAR_DICT_MAX_LENGTH   10

  -------------------------------------------------------------------------------------------

This setting allows you to limit the length of dicts in local variables.

source_lines_error_app_frames [config-source-lines-error-app-frames]

source_lines_error_library_frames [config-source-lines-error-library-frames]

source_lines_span_app_frames [config-source-lines-span-app-frames]

source_lines_span_library_frames [config-source-lines-span-library-frames]

  -----------------------------------------------------------------------------------------------------------
  Environment                                     Django/Flask                        Default
  ----------------------------------------------- ----------------------------------- -----------------------
  ELASTIC_APM_SOURCE_LINES_ERROR_APP_FRAMES       SOURCE_LINES_ERROR_APP_FRAMES       5

  ELASTIC_APM_SOURCE_LINES_ERROR_LIBRARY_FRAMES   SOURCE_LINES_ERROR_LIBRARY_FRAMES   5

  ELASTIC_APM_SOURCE_LINES_SPAN_APP_FRAMES        SOURCE_LINES_SPAN_APP_FRAMES        0

  ELASTIC_APM_SOURCE_LINES_SPAN_LIBRARY_FRAMES    SOURCE_LINES_SPAN_LIBRARY_FRAMES    0
  -----------------------------------------------------------------------------------------------------------

By default, the APM agent collects source code snippets for errors. This
setting allows you to modify the number of lines of source code that are
being collected.

We differ between errors and spans, as well as library frames and app
frames.

Especially for spans, collecting source code can have a large impact on
storage use in your Elasticsearch cluster.

capture_body [config-capture-body]

[dynamic config]

  Environment                Django/Flask   Default
  -------------------------- -------------- ---------
  ELASTIC_APM_CAPTURE_BODY   CAPTURE_BODY   off

For transactions that are HTTP requests, the Python agent can optionally
capture the request body (e.g. POST variables).

Possible values: errors, transactions, all, off.

If the request has a body and this setting is disabled, the body will be
shown as [REDACTED].

For requests with a content type of multipart/form-data, any uploaded
files will be referenced in a special _files key. It contains the name
of the field and the name of the uploaded file, if provided.

Request bodies often contain sensitive values like passwords and credit
card numbers. If your service handles data like this, we advise to only
enable this feature with care.

capture_headers [config-capture-headers]

[dynamic config]

  Environment                   Django/Flask      Default
  ----------------------------- ----------------- ---------
  ELASTIC_APM_CAPTURE_HEADERS   CAPTURE_HEADERS   true

For transactions and errors that happen due to HTTP requests, the Python
agent can optionally capture the request and response headers.

Possible values: true, false

Request headers often contain sensitive values like session IDs and
cookies. See sanitizing data for more information on how to filter out
sensitive data.

transaction_max_spans [config-transaction-max-spans]

[dynamic config]

  -----------------------------------------------------------------------------------
  Environment                         Django/Flask            Default
  ----------------------------------- ----------------------- -----------------------
  ELASTIC_APM_TRANSACTION_MAX_SPANS   TRANSACTION_MAX_SPANS   500

  -----------------------------------------------------------------------------------

This limits the amount of spans that are recorded per transaction. This
is helpful in cases where a transaction creates a very high amount of
spans (e.g. thousands of SQL queries). Setting an upper limit will
prevent edge cases from overloading the agent and the APM Server.

stack_trace_limit [config-stack-trace-limit]

[dynamic config]

  Environment                     Django/Flask        Default
  ------------------------------- ------------------- ---------
  ELASTIC_APM_STACK_TRACE_LIMIT   STACK_TRACE_LIMIT   50

This limits the number of frames captured for each stack trace.

Setting the limit to 0 will disable stack trace collection, while any
positive integer value will be used as the maximum number of frames to
collect. To disable the limit and always capture all frames, set the
value to -1.

span_stack_trace_min_duration [config-span-stack-trace-min-duration]

[dynamic config]

  ---------------------------------------------------------------------------------------------------
  Environment                                 Django/Flask                    Default
  ------------------------------------------- ------------------------------- -----------------------
  ELASTIC_APM_SPAN_STACK_TRACE_MIN_DURATION   SPAN_STACK_TRACE_MIN_DURATION   "5ms"

  ---------------------------------------------------------------------------------------------------

By default, the APM agent collects a stack trace with every recorded
span that has a duration equal to or longer than this configured
threshold. While stack traces are very helpful to find the exact place
in your code from which a span originates, collecting this stack trace
does have some overhead. Tune this threshold to ensure that you only
collect stack traces for spans that could be problematic.

To collect traces for all spans, regardless of their length, set the
value to 0.

To disable stack trace collection for spans completely, set the value to
-1.

Except for the special values -1 and 0, this setting should be provided
in duration format.

span_frames_min_duration [config-span-frames-min-duration]

[dynamic config]

  -----------------------------------------------------------------------------------------
  Environment                            Django/Flask               Default
  -------------------------------------- -------------------------- -----------------------
  ELASTIC_APM_SPAN_FRAMES_MIN_DURATION   SPAN_FRAMES_MIN_DURATION   "5ms"

  -----------------------------------------------------------------------------------------

This config value is being deprecated. Use span_stack_trace_min_duration
instead.

span_compression_enabled [config-span-compression-enabled]

[dynamic config]

  -----------------------------------------------------------------------------------------
  Environment                            Django/Flask               Default
  -------------------------------------- -------------------------- -----------------------
  ELASTIC_APM_SPAN_COMPRESSION_ENABLED   SPAN_COMPRESSION_ENABLED   True

  -----------------------------------------------------------------------------------------

Enable/disable span compression.

If enabled, the agent will compress very short, repeated spans into a
single span, which is beneficial for storage and processing
requirements. Some information is lost in this process, e.g. exact
durations of each compressed span.

span_compression_exact_match_max_duration [config-span-compression-exact-match-max_duration]

[dynamic config]

  ---------------------------------------------------------------------------------------------------------------------------
  Environment                                             Django/Flask                                Default
  ------------------------------------------------------- ------------------------------------------- -----------------------
  ELASTIC_APM_SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION   SPAN_COMPRESSION_EXACT_MATCH_MAX_DURATION   "50ms"

  ---------------------------------------------------------------------------------------------------------------------------

Consecutive spans that are exact match and that are under this threshold
will be compressed into a single composite span. This reduces the
collection, processing, and storage overhead, and removes clutter from
the UI. The tradeoff is that the DB statements of all the compressed
spans will not be collected.

Two spans are considered exact matches if the following attributes are
identical: * span name * span type * span subtype * destination resource
(e.g. the Database name)

span_compression_same_kind_max_duration [config-span-compression-same-kind-max-duration]

[dynamic config]

  -----------------------------------------------------------------------------------------------------------------------
  Environment                                           Django/Flask                              Default
  ----------------------------------------------------- ----------------------------------------- -----------------------
  ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION   SPAN_COMPRESSION_SAME_KIND_MAX_DURATION   "0ms" (disabled)

  -----------------------------------------------------------------------------------------------------------------------

Consecutive spans to the same destination that are under this threshold
will be compressed into a single composite span. This reduces the
collection, processing, and storage overhead, and removes clutter from
the UI. The tradeoff is that metadata such as database statements of all
the compressed spans will not be collected.

Two spans are considered to be of the same kind if the following
attributes are identical: * span type * span subtype * destination
resource (e.g. the Database name)

exit_span_min_duration [config-exit-span-min-duration]

[dynamic config]

  -------------------------------------------------------------------------------------
  Environment                          Django/Flask             Default
  ------------------------------------ ------------------------ -----------------------
  ELASTIC_APM_EXIT_SPAN_MIN_DURATION   EXIT_SPAN_MIN_DURATION   "0ms"

  -------------------------------------------------------------------------------------

Exit spans are spans that represent a call to an external service, like
a database. If such calls are very short, they are usually not relevant
and can be ignored.

This feature is disabled by default.

if a span propagates distributed tracing IDs, it will not be ignored,
even if it is shorter than the configured threshold. This is to ensure
that no broken traces are recorded.

api_request_size [config-api-request-size]

[dynamic config]

  Environment                    Django/Flask       Default
  ------------------------------ ------------------ ---------
  ELASTIC_APM_API_REQUEST_SIZE   API_REQUEST_SIZE   "768kb"

The maximum queue length of the request buffer before sending the
request to the APM Server. A lower value will increase the load on your
APM Server, while a higher value can increase the memory pressure of
your app. A higher value also impacts the time until data is indexed and
searchable in Elasticsearch.

This setting is useful to limit memory consumption if you experience a
sudden spike of traffic. It has to be provided in size format.

Due to internal buffering of gzip, the actual request size can be a few
kilobytes larger than the given limit. By default, the APM Server limits
request payload size to 1 MByte.

api_request_time [config-api-request-time]

[dynamic config]

  Environment                    Django/Flask       Default
  ------------------------------ ------------------ ---------
  ELASTIC_APM_API_REQUEST_TIME   API_REQUEST_TIME   "10s"

The maximum queue time of the request buffer before sending the request
to the APM Server. A lower value will increase the load on your APM
Server, while a higher value can increase the memory pressure of your
app. A higher value also impacts the time until data is indexed and
searchable in Elasticsearch.

This setting is useful to limit memory consumption if you experience a
sudden spike of traffic. It has to be provided in duration format.

The actual time will vary between 90-110% of the given value, to avoid
stampedes of instances that start at the same time.

processors [config-processors]

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Environment              Django/Flask            Default
  ------------------------ ----------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  ELASTIC_APM_PROCESSORS   PROCESSORS              ['elasticapm.processors.sanitize_stacktrace_locals',                                              'elasticapm.processors.sanitize_http_request_cookies',                                              'elasticapm.processors.sanitize_http_headers',                                              'elasticapm.processors.sanitize_http_wsgi_env',                                              'elasticapm.processors.sanitize_http_request_body']

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

A list of processors to process transactions and errors. For more
information, see Sanitizing Data.

We recommend always including the default set of validators if you
customize this setting.

sanitize_field_names [config-sanitize-field-names]

[dynamic config]

   
  Environment                        Django/Flask            Default
  ---------------------------------- -----------------------  
  ELASTIC_APM_SANITIZE_FIELD_NAMES   SANITIZE_FIELD_NAMES    ["password",                                                                  "passwd",                                                                  "pwd",                                                                  "secret",                                                                  "*key",                                                                  "*token*",                                                                  "*session*",                                                                  "*credit*",                                                                  "*card*",                                                                  "*auth*",                                                                  "*principal*",                                                                  "set-cookie"]

   

A list of glob-matched field names to match and mask when using
processors. For more information, see Sanitizing Data.

We recommend always including the default set of field name matches if
you customize this setting.

transaction_sample_rate [config-transaction-sample-rate]

[dynamic config]

  ---------------------------------------------------------------------------------------
  Environment                           Django/Flask              Default
  ------------------------------------- ------------------------- -----------------------
  ELASTIC_APM_TRANSACTION_SAMPLE_RATE   TRANSACTION_SAMPLE_RATE   1.0

  ---------------------------------------------------------------------------------------

By default, the agent samples every transaction (e.g. request to your
service). To reduce overhead and storage requirements, set the sample
rate to a value between 0.0 and 1.0. We still record overall time and
the result for unsampled transactions, but no context information,
labels, or spans.

This setting will be automatically rounded to 4 decimals of precision.

include_paths [config-include-paths]

  Environment                                           Django/Flask    Default
  ----------------------------------------------------- --------------- ---------
  ELASTIC_APM_INCLUDE_PATHS                             INCLUDE_PATHS   []
  multiple values separated by commas, without spaces                   

A set of paths, optionally using shell globs (see fnmatch for a
description of the syntax). These are matched against the absolute
filename of every frame, and if a pattern matches, the frame is
considered to be an “in-app frame”.

include_paths takes precedence over exclude_paths.

exclude_paths [config-exclude-paths]

  ---------------------------------------------------------------------------
  Environment                 Django/Flask            Default
  --------------------------- ----------------------- -----------------------
  ELASTIC_APM_EXCLUDE_PATHS   EXCLUDE_PATHS           Varies on Python
                                                      version and
                                                      implementation

  multiple values separated                           
  by commas, without spaces                           
  ---------------------------------------------------------------------------

A set of paths, optionally using shell globs (see fnmatch for a
description of the syntax). These are matched against the absolute
filename of every frame, and if a pattern matches, the frame is
considered to be a “library frame”.

include_paths takes precedence over exclude_paths.

The default value varies based on your Python version and
implementation, e.g.:

- PyPy3: ['\*/lib-python/3/*', '\*/site-packages/*']
- CPython 2.7: ['\*/lib/python2.7/*', '\*/lib64/python2.7/*']

debug [config-debug]

  Environment         Django/Flask   Default
  ------------------- -------------- ---------
  ELASTIC_APM_DEBUG   DEBUG          False

If your app is in debug mode (e.g. in Django with settings.DEBUG = True
or in Flask with app.debug = True), the agent won’t send any data to the
APM Server. You can override it by changing this setting to True.

disable_send [config-disable-send]

  Environment                Django/Flask   Default
  -------------------------- -------------- ---------
  ELASTIC_APM_DISABLE_SEND   DISABLE_SEND   False

If set to True, the agent won’t send any events to the APM Server,
independent of any debug state.

instrument [config-instrument]

  Environment              Django/Flask   Default
  ------------------------ -------------- ---------
  ELASTIC_APM_INSTRUMENT   INSTRUMENT     True

If set to False, the agent won’t instrument any code. This disables most
of the tracing functionality, but can be useful to debug possible
instrumentation issues.

verify_server_cert [config-verify-server-cert]

  Environment                      Django/Flask         Default
  -------------------------------- -------------------- ---------
  ELASTIC_APM_VERIFY_SERVER_CERT   VERIFY_SERVER_CERT   True

By default, the agent verifies the SSL certificate if an HTTPS
connection to the APM Server is used. Verification can be disabled by
changing this setting to False. This setting is ignored when server_cert
is set.

server_cert [config-server-cert]

  Environment               Django/Flask   Default
  ------------------------- -------------- ---------
  ELASTIC_APM_SERVER_CERT   SERVER_CERT    None

If you have configured your APM Server with a self-signed TLS
certificate, or you just wish to pin the server certificate, you can
specify the path to the PEM-encoded certificate via the
ELASTIC_APM_SERVER_CERT configuration.

If this option is set, the agent only verifies that the certificate
provided by the APM Server is identical to the one configured here.
Validity of the certificate is not checked.

server_ca_cert_file [config-server-ca-cert-file]

  Environment                       Django/Flask          Default
  --------------------------------- --------------------- ---------
  ELASTIC_APM_SERVER_CA_CERT_FILE   SERVER_CA_CERT_FILE   None

By default, the agent will validate the TLS/SSL certificate of the APM
Server using the well-known CAs curated by Mozilla, and provided by the
certifi package.

You can set this option to the path of a file containing a CA
certificate that will be used instead.

Specifying this option is required when using self-signed certificates,
unless server certificate validation is disabled.

use_certifi [config-use-certifi]

  Environment               Django/Flask   Default
  ------------------------- -------------- ---------
  ELASTIC_APM_USE_CERTIFI   USE_CERTIFI    True

By default, the Python Agent uses the certifi certificate store. To use
Python’s default mechanism for finding certificates, set this option to
False.

metrics_interval [config-metrics_interval]

  Environment                    Django/Flask       Default
  ------------------------------ ------------------ ---------
  ELASTIC_APM_METRICS_INTERVAL   METRICS_INTERVAL   30s

The interval in which the agent collects metrics. A shorter interval
increases the granularity of metrics, but also increases the overhead of
the agent, as well as storage requirements.

It has to be provided in duration format.

disable_metrics [config-disable_metrics]

  Environment                   Django/Flask      Default
  ----------------------------- ----------------- ---------
  ELASTIC_APM_DISABLE_METRICS   DISABLE_METRICS   None

A comma-separated list of dotted metrics names that should not be sent
to the APM Server. You can use * to match multiple metrics; for example,
to disable all CPU-related metrics, as well as the “total system memory”
metric, set disable_metrics to:

    "*.cpu.*,system.memory.total"

This setting only disables the sending of the given metrics, not
collection.

breakdown_metrics [config-breakdown_metrics]

  Environment                     Django/Flask        Default
  ------------------------------- ------------------- ---------
  ELASTIC_APM_BREAKDOWN_METRICS   BREAKDOWN_METRICS   True

Enable or disable the tracking and collection of breakdown metrics.
Setting this to False disables the tracking of breakdown metrics, which
can reduce the overhead of the agent.

This feature requires APM Server and Kibana >= 7.3.

prometheus_metrics (Beta) [config-prometheus_metrics]

  Environment                      Django/Flask         Default
  -------------------------------- -------------------- ---------
  ELASTIC_APM_PROMETHEUS_METRICS   PROMETHEUS_METRICS   False

Enable/disable the tracking and collection of metrics from
prometheus_client.

See Prometheus metric set (beta) for more information.

This feature is currently in beta status.

prometheus_metrics_prefix (Beta) [config-prometheus_metrics_prefix]

  -------------------------------------------------------------------------------------------
  Environment                             Django/Flask                Default
  --------------------------------------- --------------------------- -----------------------
  ELASTIC_APM_PROMETHEUS_METRICS_PREFIX   PROMETHEUS_METRICS_PREFIX   prometheus.metrics.

  -------------------------------------------------------------------------------------------

A prefix to prepend to Prometheus metrics names.

See Prometheus metric set (beta) for more information.

This feature is currently in beta status.

metrics_sets [config-metrics_sets]

  -------------------------------------------------------------------------------------------------
  Environment                Django/Flask            Default
  -------------------------- ----------------------- ----------------------------------------------
  ELASTIC_APM_METRICS_SETS   METRICS_SETS            [“elasticapm.metrics.sets.cpu.CPUMetricSet”]

  -------------------------------------------------------------------------------------------------

List of import paths for the MetricSets that should be used to collect
metrics.

See Custom Metrics for more information.

central_config [config-central_config]

  Environment                  Django/Flask     Default
  ---------------------------- ---------------- ---------
  ELASTIC_APM_CENTRAL_CONFIG   CENTRAL_CONFIG   True

When enabled, the agent will make periodic requests to the APM Server to
fetch updated configuration.

See Dynamic configuration for more information.

This feature requires APM Server and Kibana >= 7.3.

global_labels [config-global_labels]

  Environment                 Django/Flask    Default
  --------------------------- --------------- ---------
  ELASTIC_APM_GLOBAL_LABELS   GLOBAL_LABELS   None

Labels added to all events, with the format key=value[,key=value[,...]].
Any labels set by application via the API will override global labels
with the same keys.

This feature requires APM Server >= 7.2.

disable_log_record_factory [config-generic-disable-log-record-factory]

  ---------------------------------------------------------------------------------------------
  Environment                              Django/Flask                 Default
  ---------------------------------------- ---------------------------- -----------------------
  ELASTIC_APM_DISABLE_LOG_RECORD_FACTORY   DISABLE_LOG_RECORD_FACTORY   False

  ---------------------------------------------------------------------------------------------

By default in python 3, the agent installs a LogRecord factory that
automatically adds tracing fields to your log records. Disable this
behavior by setting this to True.

use_elastic_traceparent_header [config-use-elastic-traceparent-header]

  -----------------------------------------------------------------------------------------------------
  Environment                                  Django/Flask                     Default
  -------------------------------------------- -------------------------------- -----------------------
  ELASTIC_APM_USE_ELASTIC_TRACEPARENT_HEADER   USE_ELASTIC_TRACEPARENT_HEADER   True

  -----------------------------------------------------------------------------------------------------

To enable distributed tracing, the agent sets a number of HTTP headers
to outgoing requests made with instrumented HTTP libraries. These
headers (traceparent and tracestate) are defined in the W3C Trace
Context specification.

Additionally, when this setting is set to True, the agent will set
elasticapm-traceparent for backwards compatibility.

trace_continuation_strategy [config-trace-continuation-strategy]

[dynamic config]

  -----------------------------------------------------------------------------------------------
  Environment                               Django/Flask                  Default
  ----------------------------------------- ----------------------------- -----------------------
  ELASTIC_APM_TRACE_CONTINUATION_STRATEGY   TRACE_CONTINUATION_STRATEGY   continue

  -----------------------------------------------------------------------------------------------

This option allows some control on how the APM agent handles W3C
trace-context headers on incoming requests. By default, the traceparent
and tracestate headers are used per W3C spec for distributed tracing.
However, in certain cases it can be helpful to not use the incoming
traceparent header. Some example use cases:

- An Elastic-monitored service is receiving requests with traceparent
  headers from unmonitored services.
- An Elastic-monitored service is publicly exposed, and does not want
  tracing data (trace-ids, sampling decisions) to possibly be spoofed by
  user requests.

Valid values are:

- 'continue': The default behavior. An incoming traceparent value is
  used to continue the trace and determine the sampling decision.
- 'restart': Always ignores the traceparent header of incoming requests.
  A new trace-id will be generated and the sampling decision will be
  made based on transaction_sample_rate. A span link will be made to the
  incoming traceparent.
- 'restart_external': If an incoming request includes the es vendor flag
  in tracestate, then any traceparent will be considered internal and
  will be handled as described for 'continue' above. Otherwise, any
  'traceparent' is considered external and will be handled as described
  for 'restart' above.

Starting with Elastic Observability 8.2, span links will be visible in
trace views.

use_elastic_excepthook [config-use-elastic-excepthook]

  -------------------------------------------------------------------------------------
  Environment                          Django/Flask             Default
  ------------------------------------ ------------------------ -----------------------
  ELASTIC_APM_USE_ELASTIC_EXCEPTHOOK   USE_ELASTIC_EXCEPTHOOK   False

  -------------------------------------------------------------------------------------

If set to True, the agent will intercept the default sys.excepthook,
which allows the agent to collect all uncaught exceptions.

include_process_args [config-include-process-args]

  ----------------------------------------------------------------------------------
  Environment                        Django/Flask            Default
  ---------------------------------- ----------------------- -----------------------
  ELASTIC_APM_INCLUDE_PROCESS_ARGS   INCLUDE_PROCESS_ARGS    False

  ----------------------------------------------------------------------------------

Whether each transaction should have the process arguments attached.
Disabled by default to save disk space.

skip_server_info [config-skip-server-info]

{applies_to} apm_agent_python: preview 6.25.0

  Environment                    Django/Flask       Default
  ------------------------------ ------------------ ---------
  ELASTIC_APM_SKIP_SERVER_INFO   SKIP_SERVER_INFO   False

Whether we should skip the server info check to save some latency on
constrained environments like AWS Lambda. Disabled by default.

This functionality is in technical preview and may be changed or removed
in a future release. Elastic will work to fix any issues, but features
in technical preview are not subject to the support SLA of official GA
features.

This requires sending data to an APM Server newer than 8.7.0 in order to
work properly.

Django-specific configuration [config-django-specific]

django_transaction_name_from_route [config-django-transaction-name-from-route]

  -------------------------------------------------------------------------------------------------------------
  Environment                                      Django                               Default
  ------------------------------------------------ ------------------------------------ -----------------------
  ELASTIC_APM_DJANGO_TRANSACTION_NAME_FROM_ROUTE   DJANGO_TRANSACTION_NAME_FROM_ROUTE   False

  -------------------------------------------------------------------------------------------------------------

By default, we use the function or class name of the view as the
transaction name. Starting with Django 2.2, Django makes the route
(e.g. users/<int:user_id>/) available on the request.resolver_match
object. If you want to use the route instead of the view name as the
transaction name, set this config option to true.

in versions previous to Django 2.2, changing this setting will have no
effect.

django_autoinsert_middleware [config-django-autoinsert-middleware]

  -------------------------------------------------------------------------------------------------
  Environment                                Django                         Default
  ------------------------------------------ ------------------------------ -----------------------
  ELASTIC_APM_DJANGO_AUTOINSERT_MIDDLEWARE   DJANGO_AUTOINSERT_MIDDLEWARE   True

  -------------------------------------------------------------------------------------------------

To trace Django requests, the agent uses a middleware,
elasticapm.contrib.django.middleware.TracingMiddleware. By default, this
middleware is inserted automatically as the first item in
settings.MIDDLEWARES. To disable the automatic insertion of the
middleware, change this setting to False.

Generic Environment variables [config-generic-environment]

Some environment variables that are not specific to the APM agent can be
used to configure the agent.

HTTP_PROXY and HTTPS_PROXY [config-generic-http-proxy]

By using HTTP_PROXY and HTTPS_PROXY, the agent can be instructed to use
a proxy to connect to the APM Server. If both are set, HTTPS_PROXY takes
precedence.

The environment variables are case-insensitive.

NO_PROXY [config-generic-no-proxy]

To instruct the agent to not use a proxy, you can use the NO_PROXY
environment variable. You can either set it to a comma-separated list of
hosts for which no proxy should be used (e.g. localhost,example.com) or
use * to match any host.

This is useful if HTTP_PROXY / HTTPS_PROXY is set for other reasons than
agent / APM Server communication.

SSL_CERT_FILE and SSL_CERT_DIR [config-ssl-cert-file]

To tell the agent to use a different SSL certificate, you can use these
environment variables. See also OpenSSL docs.

Please note that these variables may apply to other SSL/TLS
communication in your service, not just related to the APM agent.

These environment variables only take effect if use_certifi is set to
False.

Configuration formats [config-formats]

Some options require a unit, either duration or size. These need to be
provided in a specific format.

Duration format [config-format-duration]

The duration format is used for options like timeouts. The unit is
provided as a suffix directly after the number–without any separation by
whitespace.

Example: 5ms

Supported units

- us (microseconds)
- ms (milliseconds)
- s (seconds)
- m (minutes)

Size format [config-format-size]

The size format is used for options like maximum buffer sizes. The unit
is provided as suffix directly after the number, without and separation
by whitespace.

Example: 10kb

Supported units:

- b (bytes)
- kb (kilobytes)
- mb (megabytes)
- gb (gigabytes)

We use the power-of-two sizing convention, e.g. 1 kilobyte == 1024 bytes

Django support [django-support]

Getting Elastic APM set up for your Django project is easy, and there
are various ways you can tweak it to fit to your needs.

Installation [django-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add it to your project’s requirements.txt file.

For apm-server 6.2+, make sure you use version 2.0 or higher of
elastic-apm.

If you use Django with uwsgi, make sure to enable threads (enabled by
default since 2.0.27) and py-call-uwsgi-fork-hooks.

Setup [django-setup]

Set up the Elastic APM agent in Django with these two steps:

1.  Add elasticapm.contrib.django to INSTALLED_APPS in your settings:

    INSTALLED_APPS = (
       # ...
       'elasticapm.contrib.django',
    )

1.  Choose a service name, and set the secret token if needed.

    ELASTIC_APM = {
       'SERVICE_NAME': '<SERVICE-NAME>',
       'SECRET_TOKEN': '<SECRET-TOKEN>',
    }

or as environment variables:

    ELASTIC_APM_SERVICE_NAME=<SERVICE-NAME>
    ELASTIC_APM_SECRET_TOKEN=<SECRET-TOKEN>

You now have basic error logging set up, and everything resulting in a
500 HTTP status code will be reported to the APM Server.

You can find a list of all available settings in the Configuration page.

The agent only captures and sends data if you have DEBUG = False in your
settings. To force the agent to capture data in Django debug mode, set
the debug configuration option, e.g.:

    ELASTIC_APM = {
       'SERVICE_NAME': '<SERVICE-NAME>',
       'DEBUG': True,
    }

Performance metrics [django-performance-metrics]

In order to collect performance metrics, the agent automatically inserts
a middleware at the top of your middleware list (settings.MIDDLEWARE in
current versions of Django, settings.MIDDLEWARE_CLASSES in some older
versions). To disable the automatic insertion of the middleware, see
django_autoinsert_middleware.

For automatic insertion to work, your list of middlewares
(settings.MIDDLEWARE or settings.MIDDLEWARE_CLASSES) must be of type
list or tuple.

In addition to broad request metrics (what will appear in the APM app as
transactions), the agent also collects fine grained metrics on template
rendering, database queries, HTTP requests, etc. You can find more
information on what we instrument in the Automatic Instrumentation
section.

Instrumenting custom Python code [django-instrumenting-custom-python-code]

To gain further insights into the performance of your code, please see
instrumenting custom code.

Ignoring specific views [django-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific views. The list given should be a list of regular
expressions which are matched against the transaction name as seen in
the Elastic APM user interface:

    ELASTIC_APM['TRANSACTIONS_IGNORE_PATTERNS'] = ['^OPTIONS ', 'views.api.v2']

This example ignores any requests using the OPTIONS method and any
requests containing views.api.v2.

Using the route as transaction name [django-transaction-name-route]

By default, we use the function or class name of the view as the
transaction name. Starting with Django 2.2, Django makes the route
(e.g. users/<int:user_id>/) available on the request.resolver_match
object. If you want to use the route instead of the view name as the
transaction name, you can set the django_transaction_name_from_route
config option to true.

    ELASTIC_APM['DJANGO_TRANSACTION_NAME_FROM_ROUTE'] = True

in versions previous to Django 2.2, changing this setting will have no
effect.

Integrating with the RUM Agent [django-integrating-with-the-rum-agent]

To correlate performance measurement in the browser with measurements in
your Django app, you can help the RUM (Real User Monitoring) agent by
configuring it with the Trace ID and Span ID of the backend request. We
provide a handy template context processor which adds all the necessary
bits into the context of your templates.

To enable this feature, first add the rum_tracing context processor to
your TEMPLATES setting. You most likely already have a list of
context_processors, in which case you can simply append ours to the
list.

    TEMPLATES = [
        {
            'BACKEND': 'django.template.backends.django.DjangoTemplates',
            'OPTIONS': {
                'context_processors': [
                    # ...
                    'elasticapm.contrib.django.context_processors.rum_tracing',
                ],
            },
        },
    ]

Then, update the call to initialize the RUM agent (which probably
happens in your base template) like this:

    elasticApm.init({
        serviceName: "my-frontend-service",
        pageLoadTraceId: "{{ apm.trace_id }}",
        pageLoadSpanId: "{{ apm.span_id }}",
        pageLoadSampled: {{ apm.is_sampled_js }}
    })

See the JavaScript RUM agent documentation for more information.

Enabling and disabling the agent [django-enabling-and-disabling-the-agent]

The easiest way to disable the agent is to set Django’s DEBUG option to
True in your development configuration. No errors or metrics will be
logged to Elastic APM.

However, if during debugging you would like to force logging of errors
to Elastic APM, then you can set DEBUG to True inside of the Elastic APM
configuration dictionary, like this:

    ELASTIC_APM = {
       # ...
       'DEBUG': True,
    }

Integrating with Python logging [django-logging]

To easily send Python logging messages as “error” objects to
Elasticsearch, we provide a LoggingHandler which you can use in your
logging setup. The log messages will be enriched with a stack trace,
data from the request, and more.

the intended use case for this handler is to send high priority log
messages (e.g. log messages with level ERROR) to Elasticsearch. For
normal log shipping, we recommend using filebeat.

If you are new to how the logging module works together with Django,
read more in the Django documentation.

An example of how your LOGGING setting could look:

    LOGGING = {
        'version': 1,
        'disable_existing_loggers': True,
        'formatters': {
            'verbose': {
                'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
            },
        },
        'handlers': {
            'elasticapm': {
                'level': 'WARNING',
                'class': 'elasticapm.contrib.django.handlers.LoggingHandler',
            },
            'console': {
                'level': 'DEBUG',
                'class': 'logging.StreamHandler',
                'formatter': 'verbose'
            }
        },
        'loggers': {
            'django.db.backends': {
                'level': 'ERROR',
                'handlers': ['console'],
                'propagate': False,
            },
            'mysite': {
                'level': 'WARNING',
                'handlers': ['elasticapm'],
                'propagate': False,
            },
            # Log errors from the Elastic APM module to the console (recommended)
            'elasticapm.errors': {
                'level': 'ERROR',
                'handlers': ['console'],
                'propagate': False,
            },
        },
    }

With this configuration, logging can be done like this in any module in
the myapp django app:

You can now use the logger in any module in the myapp Django app, for
instance myapp/views.py:

    import logging
    logger = logging.getLogger('mysite')

    try:
        instance = MyModel.objects.get(pk=42)
    except MyModel.DoesNotExist:
        logger.error(
            'Could not find instance, doing something else',
            exc_info=True
        )

Note that exc_info=True adds the exception information to the data that
gets sent to Elastic APM. Without it, only the message is sent.

Extra data [django-extra-data]

If you want to send more data than what you get with the agent by
default, logging can be done like so:

    import logging
    logger = logging.getLogger('mysite')

    try:
        instance = MyModel.objects.get(pk=42)
    except MyModel.DoesNotExist:
        logger.error(
            'There was some crazy error',
            exc_info=True,
            extra={
                'datetime': str(datetime.now()),
            }
        )

Celery integration [django-celery-integration]

For a general guide on how to set up Django with Celery, head over to
Celery’s Django documentation.

Elastic APM will automatically log errors from your celery tasks, record
performance data and keep the trace.id when the task is launched from an
already started Elastic transaction.

Logging “HTTP 404 Not Found” errors [django-logging-http-404-not-found-errors]

By default, Elastic APM does not log HTTP 404 errors. If you wish to log
these errors, add
'elasticapm.contrib.django.middleware.Catch404Middleware' to MIDDLEWARE
in your settings:

    MIDDLEWARE = (
        # ...
        'elasticapm.contrib.django.middleware.Catch404Middleware',
        # ...
    )

Note that this middleware respects Django’s IGNORABLE_404_URLS setting.

Disable the agent during tests [django-disable-agent-during-tests]

To prevent the agent from sending any data to the APM Server during
tests, set the ELASTIC_APM_DISABLE_SEND environment variable to true,
e.g.:

    ELASTIC_APM_DISABLE_SEND=true python manage.py test

Troubleshooting [django-troubleshooting]

Elastic APM comes with a Django command that helps troubleshooting your
setup. To check your configuration, run

    python manage.py elasticapm check

To send a test exception using the current settings, run

    python manage.py elasticapm test

If the command succeeds in sending a test exception, it will print a
success message:

    python manage.py elasticapm test

    Trying to send a test error using these settings:

    SERVICE_NAME:      <SERVICE_NAME>
    SECRET_TOKEN:      <SECRET_TOKEN>
    SERVER:            http://127.0.0.1:8200

    Success! We tracked the error successfully! You should be able to see it in a few seconds.

Supported Django and Python versions [supported-django-and-python-versions]

A list of supported Django and Python versions can be found on our
Supported Technologies page.

Flask support [flask-support]

Getting Elastic APM set up for your Flask project is easy, and there are
various ways you can tweak it to fit to your needs.

Installation [flask-installation]

Install the Elastic APM agent using pip:

    $ pip install "elastic-apm[flask]"

or add elastic-apm[flask] to your project’s requirements.txt file.

For apm-server 6.2+, make sure you use version 2.0 or higher of
elastic-apm.

If you use Flask with uwsgi, make sure to enable threads (enabled by
default since 2.0.27) and py-call-uwsgi-fork-hooks.

If you see an error log that mentions psutil not found, you can install
psutil using pip install psutil, or add psutil to your requirements.txt
file.

Setup [flask-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, the
application’s settings, or as initialization arguments.

You can find a list of all available settings in the Configuration page.

To initialize the agent for your application using environment
variables:

    from elasticapm.contrib.flask import ElasticAPM

    app = Flask(__name__)

    apm = ElasticAPM(app)

To configure the agent using ELASTIC_APM in your application’s settings:

    from elasticapm.contrib.flask import ElasticAPM

    app.config['ELASTIC_APM'] = {
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
    }
    apm = ElasticAPM(app)

The final option is to initialize the agent with the settings as
arguments:

    from elasticapm.contrib.flask import ElasticAPM

    apm = ElasticAPM(app, service_name='<APP-ID>', secret_token='<SECRET-TOKEN>')

Debug mode [flask-debug-mode]

Please note that errors and transactions will only be sent to the APM
Server if your app is not in Flask debug mode.

To force the agent to send data while the app is in debug mode, set the
value of DEBUG in the ELASTIC_APM dictionary to True:

    app.config['ELASTIC_APM'] = {
            'SERVICE_NAME': '<SERVICE-NAME>',
            'SECRET_TOKEN': '<SECRET-TOKEN>',
            'DEBUG': True
    }

Building applications on the fly? [flask-building-applications-on-the-fly]

You can use the agent’s init_app hook for adding the application on the
fly:

    from elasticapm.contrib.flask import ElasticAPM
    apm = ElasticAPM()

    def create_app():
        app = Flask(__name__)
        apm.init_app(app, service_name='<SERVICE-NAME>', secret_token='<SECRET-TOKEN>')
        return app

Usage [flask-usage]

Once you have configured the agent, it will automatically track
transactions and capture uncaught exceptions within Flask. If you want
to send additional events, a couple of shortcuts are provided on the
ElasticAPM Flask middleware object by raising an exception or logging a
generic message.

Capture an arbitrary exception by calling capture_exception:

    try:
        1 / 0
    except ZeroDivisionError:
        apm.capture_exception()

Log a generic message with capture_message:

    apm.capture_message('hello, world!')

Shipping Logs to Elasticsearch [flask-logging]

This feature has been deprecated and will be removed in a future
version.

Please see our Logging documentation for other supported ways to ship
logs to Elasticsearch.

Note that you can always send exceptions and messages to the APM Server
with capture_exception and and capture_message.

    from elasticapm import get_client

    @app.route('/')
    def bar():
        try:
            1 / 0
        except ZeroDivisionError:
            get_client().capture_exception()

Extra data [flask-extra-data]

In addition to what the agents log by default, you can send extra
information:

    @app.route('/')
    def bar():
        try:
            1 / 0
        except ZeroDivisionError:
            app.logger.error('Math is hard',
                exc_info=True,
                extra={
                    'good_at_math': False,
                }
            )
        )

Celery tasks [flask-celery-tasks]

The Elastic APM agent will automatically send errors and performance
data from your Celery tasks to the APM Server.

Performance metrics [flask-performance-metrics]

If you’ve followed the instructions above, the agent has already hooked
into the right signals and should be reporting performance metrics.

Ignoring specific routes [flask-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific routes. The list given should be a list of regular
expressions which are matched against the transaction name:

    app.config['ELASTIC_APM'] = {
        ...
        'TRANSACTIONS_IGNORE_PATTERNS': ['^OPTIONS ', '/api/']
        ...
    }

This would ignore any requests using the OPTIONS method and any requests
containing /api/.

Integrating with the RUM Agent [flask-integrating-with-the-rum-agent]

To correlate performance measurement in the browser with measurements in
your Flask app, you can help the RUM (Real User Monitoring) agent by
configuring it with the Trace ID and Span ID of the backend request. We
provide a handy template context processor which adds all the necessary
bits into the context of your templates.

The context processor is installed automatically when you initialize
ElasticAPM. All that is left to do is to update the call to initialize
the RUM agent (which probably happens in your base template) like this:

    elasticApm.init({
        serviceName: "my-frontend-service",
        pageLoadTraceId: "{{ apm["trace_id"] }}",
        pageLoadSpanId: "{{ apm["span_id"]() }}",
        pageLoadSampled: {{ apm["is_sampled_js"] }}
    })

See the JavaScript RUM agent documentation for more information.

Supported Flask and Python versions [supported-flask-and-python-versions]

A list of supported Flask and Python versions can be found on our
Supported Technologies page.

How the Agent works [how-the-agent-works]

To gather APM events (called transactions and spans), errors and
metrics, the Python agent instruments your application in a few
different ways. These events, are then sent to the APM Server. The APM
Server converts them to a format suitable for Elasticsearch, and sends
them to an Elasticsearch cluster. You can then use the APM app in Kibana
to gain insight into latency issues and error culprits within your
application.

Broadly, we differentiate between three different approaches to collect
the necessary data: framework integration, instrumentation, and
background collection.

Framework integration [how-it-works-framework-integration]

To collect data about incoming requests and background tasks, we
integrate with frameworks like Django, Flask and Celery. Whenever
possible, framework integrations make use of hooks and signals provided
by the framework. Examples of this are:

- request_started, request_finished, and got_request_exception signals
  from django.core.signals
- request_started, request_finished, and got_request_exception signals
  from flask.signals
- task_prerun, task_postrun, and task_failure signals from
  celery.signals

Framework integrations require some limited code changes in your app.
E.g. for Django, you need to add elasticapm.contrib.django to
INSTALLED_APPS.

What if you are not using a framework [how-it-works-no-framework]

If you’re not using a supported framework, for example, a simple Python
script, you can still leverage the agent’s automatic instrumentation.
Check out our docs on instrumenting custom code.

Instrumentation [how-it-works-instrumentation]

To collect data from database drivers, HTTP libraries etc., we
instrument certain functions and methods in these libraries. Our
instrumentation wraps these callables and collects additional data, like

- time spent in the call
- the executed query for database drivers
- the fetched URL for HTTP libraries

We use a 3rd party library, wrapt, to wrap the callables. You can read
more on how wrapt works in Graham Dumpleton’s excellent series of blog
posts.

Instrumentations are set up automatically and do not require any code
changes. See Automatic Instrumentation to learn more about which
libraries we support.

Background collection [how-it-works-background-collection]

In addition to APM and error data, the Python agent also collects system
and application metrics in regular intervals. This collection happens in
a background thread that is started by the agent.

In addition to the metrics collection background thread, the agent
starts two additional threads per process:

- a thread to regularly fetch remote configuration from the APM Server
- a thread to process the collected data and send it to the APM Server
  via HTTP.

Note that every process that instantiates the agent will have these
three threads. This means that when you e.g. use gunicorn or uwsgi
workers, each worker will have three threads started by the Python
agent.

APM Python agent [getting-started]

The Elastic APM Python agent sends performance metrics and error logs to
the APM Server. It has built-in support for Django and Flask performance
metrics and error logging, as well as generic support of other WSGI
frameworks for error logging.

Elastic supports OpenTelemetry, which allows logs, metrics, and trace
signal collection for many of the supported technologies below. Consider
using the EDOT Python SDK for observability data so you continue to get
the full power of Elastic’s platform.

How does the Agent work? [how-it-works]

The Python Agent instruments your application to collect APM events in a
few different ways:

To collect data about incoming requests and background tasks, the Agent
integrates with supported technologies to make use of hooks and signals
provided by the framework. These framework integrations require limited
code changes in your application.

To collect data from database drivers, HTTP libraries etc., we
instrument certain functions and methods in these libraries.
Instrumentations are set up automatically and do not require any code
changes.

In addition to APM and error data, the Python agent also collects system
and application metrics in regular intervals. This collection happens in
a background thread that is started by the agent.

More detailed information on how the Agent works can be found in the
advanced topics.

Additional components [additional-components]

APM Agents work in conjunction with the APM Server, Elasticsearch, and
Kibana. The APM documentation provides details on how these components
work together, and provides a matrix outlining Agent and Server
compatibility.

Troubleshooting

If you’re experiencing issues with the APM Python agent, refer to
Troubleshoot APM Python Agent.

Instrumenting custom code [instrumenting-custom-code]

Creating Additional Spans in a Transaction [instrumenting-custom-code-spans]

Elastic APM instruments a variety of libraries out of the box, but
sometimes you need to know how long a specific function took or how
often it gets called.

Assuming you’re using one of our supported frameworks, you can apply the
@elasticapm.capture_span() decorator to achieve exactly that. If you’re
not using a supported framework, see Creating New Transactions.

elasticapm.capture_span can be used either as a decorator or as a
context manager. The following example uses it both ways:

    import elasticapm

    @elasticapm.capture_span()
    def coffee_maker(strength):
        fetch_water()

        with elasticapm.capture_span('near-to-machine'):
            insert_filter()
            for i in range(strength):
                pour_coffee()

            start_drip()

        fresh_pots()

Similarly, you can use elasticapm.async_capture_span for instrumenting
async workloads:

    import elasticapm

    @elasticapm.async_capture_span()
    async def coffee_maker(strength):
        await fetch_water()

        async with elasticapm.async_capture_span('near-to-machine'):
            await insert_filter()
            async for i in range(strength):
                await pour_coffee()

            start_drip()

        fresh_pots()

asyncio support is only available in Python 3.7+.

See the API docs for more information on capture_span.

Creating New Transactions [instrumenting-custom-code-transactions]

It’s important to note that elasticapm.capture_span only works if there
is an existing transaction. If you’re not using one of our supported
frameworks, you need to create a Client object and begin and end the
transactions yourself. You can even utilize the agent’s automatic
instrumentation!

To collect the spans generated by the supported libraries, you need to
invoke elasticapm.instrument() (just once, at the initialization stage
of your application) and create at least one transaction. It is up to
you to determine what you consider a transaction within your application
— it can be the whole execution of the script or a part of it.

The example below will consider the whole execution as a single
transaction with two HTTP request spans in it. The config for
elasticapm.Client can be passed in programmatically, and it will also
utilize any config environment variables available to it automatically.

    import requests
    import time
    import elasticapm

    def main():
        sess = requests.Session()
        for url in [ 'https://www.elastic.co', 'https://benchmarks.elastic.co' ]:
            resp = sess.get(url)
            time.sleep(1)

    if __name__ == '__main__':
        client = elasticapm.Client(service_name="foo", server_url="https://example.com:8200")
        elasticapm.instrument()  # Only call this once, as early as possible.
        client.begin_transaction(transaction_type="script")
        main()
        client.end_transaction(name=__name__, result="success")

Note that you don’t need to do anything to send the data — the Client
object will handle that before the script exits. Additionally, the
Client object should be treated as a singleton — you should only create
one instance and store/pass around that instance for all transaction
handling.

Distributed Tracing [instrumenting-custom-code-distributed-tracing]

When instrumenting custom code across multiple services, you should
propagate the TraceParent where possible. This allows Elastic APM to
bundle the various transactions into a single distributed trace. The
Python Agent will automatically add TraceParent information to the
headers of outgoing HTTP requests, which can then be used on the
receiving end to add that TraceParent information to new
manually-created transactions.

Additionally, the Python Agent provides utilities for propagating the
TraceParent in string format.

    import elasticapm

    client = elasticapm.Client(service_name="foo", server_url="https://example.com:8200")

    # Retrieve the current TraceParent as a string, requires active transaction
    traceparent_string = elasticapm.get_trace_parent_header()

    # Create a TraceParent object from a string and use it for a new transaction
    parent = elasticapm.trace_parent_from_string(traceparent_string)
    client.begin_transaction(transaction_type="script", trace_parent=parent)
    # Do some work
    client.end_transaction(name=__name__, result="success")

    # Create a TraceParent object from a dictionary of headers, provided
    # automatically by the sending service if it is using an Elastic APM Agent.
    parent = elasticapm.trace_parent_from_headers(headers_dict)
    client.begin_transaction(transaction_type="script", trace_parent=parent)
    # Do some work
    client.end_transaction(name=__name__, result="success")

Monitoring AWS Lambda Python Functions [lambda-support]

The Python APM Agent can be used with AWS Lambda to monitor the
execution of your AWS Lambda functions.

The Centralized Agent Configuration on the Elasticsearch APM currently
does NOT support AWS Lambda.

Prerequisites [_prerequisites]

You need an APM Server to send APM data to. Follow the APM Quick start
if you have not set one up yet. For the best-possible performance, we
recommend setting up APM on {{ecloud}} in the same AWS region as your
AWS Lambda functions.

Step 1: Add the APM Layers to your Lambda function [add_the_apm_layers_to_your_lambda_function]

Both the {{apm-lambda-ext}} and the Python APM Agent are added to your
Lambda function as AWS Lambda Layers. Therefore, you need to add the
corresponding Layer ARNs (identifiers) to your Lambda function.

::::::{tab-item} AWS Web Console To add the layers to your Lambda
function through the AWS Management Console:

1.  Navigate to your function in the AWS Management Console
2.  Scroll to the Layers section and click the Add a layer button [image
    of layer configuration section in AWS Console]
3.  Choose the Specify an ARN radio button
4.  Copy and paste the following ARNs of the {{apm-lambda-ext}} layer
    and the APM agent layer in the Specify an ARN text input:
    - APM Extension layer:
      bash subs=true arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-extension-{{apm-lambda-ext-v}}-{ARCHITECTURE}:1 <1>
      1.  Replace {AWS_REGION} with the AWS region of your Lambda
          function and {ARCHITECTURE} with its architecture.
    - APM agent layer:
      bash subs=true arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-python-{{apm-python-v}}:{{apm-python-layer-v}} <1>
      1.  Replace {AWS_REGION} with the AWS region of your Lambda
          function.

    [image of choosing a layer in AWS Console]
5.  Click the Add button

::::::{tab-item} AWS CLI To add the Layer ARNs of the {{apm-lambda-ext}}
and the APM agent through the AWS command line interface execute the
following command:

bash subs=true aws lambda update-function-configuration --function-name yourLambdaFunctionName \ --layers arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-extension-{{apm-lambda-ext-v}}-{ARCHITECTURE}:1 \ <1> arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-python-{{apm-python-v}}:{{apm-python-layer-v}} <2> 1.
Replace {AWS_REGION} with the AWS region of your Lambda function and
{ARCHITECTURE} with its architecture. 2. Replace {AWS_REGION} with the
AWS region of your Lambda function. ::::::

::::::{tab-item} SAM In your SAM template.yml file add the Layer ARNs of
the {{apm-lambda-ext}} and the APM agent as follows:

yaml subs=true ... Resources:   yourLambdaFunction:     Type: AWS::Serverless::Function     Properties:       ...       Layers:           - arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-extension-{{apm-lambda-ext-v}}-{ARCHITECTURE}:1 <1>           - arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-python-{{apm-python-v}}:{{apm-python-layer-v}} <2> ... 1.
Replace {AWS_REGION} with the AWS region of your Lambda function and
{ARCHITECTURE} with its architecture. 2. Replace {AWS_REGION} with the
AWS region of your Lambda function. ::::::

::::::{tab-item} Serverless In your serverless.yml file add the Layer
ARNs of the {{apm-lambda-ext}} and the APM agent to your function as
follows:

yaml subs=true ... functions:   yourLambdaFunction:     handler: ...     layers:       - arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-extension-{{apm-lambda-ext-v}}-{ARCHITECTURE}:1 <1>       - arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-python-{{apm-python-v}}:{{apm-python-layer-v}} <2> ... 1.
Replace {AWS_REGION} with the AWS region of your Lambda function and
{ARCHITECTURE} with its architecture. 2. Replace {AWS_REGION} with the
AWS region of your Lambda function. ::::::

::::::{tab-item} Terraform To add the{{apm-lambda-ext}} and the APM
agent to your function add the ARNs to the layers property in your
Terraform file:

yaml subs=true ... resource "aws_lambda_function" "your_lambda_function" {   ...   layers = ["arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-extension-{{apm-lambda-ext-v}}-{ARCHITECTURE}:1", "arn:aws:lambda:{AWS_REGION}:267093732750:layer:elastic-apm-python-{{apm-python-v}}:{{apm-python-layer-v}}"] <1> } ... 1.
Replace {AWS_REGION} with the AWS region of your Lambda function and
{ARCHITECTURE} with its architecture. ::::::

::::::{tab-item} Container Image To add the {{apm-lambda-ext}} and the
APM agent to your container-based function extend the Dockerfile of your
function image as follows:

    FROM docker.elastic.co/observability/apm-lambda-extension-{IMAGE_ARCH}:latest AS lambda-extension <1>
    FROM docker.elastic.co/observability/apm-agent-python:latest AS python-agent

    # FROM ...  <-- this is the base image of your Lambda function

    COPY --from=lambda-extension /opt/elastic-apm-extension /opt/extensions/elastic-apm-extension
    COPY --from=python-agent /opt/python/ /opt/python/

    # ...

1.  Replace {IMAGE_ARCH} with the architecture of the image. ::::::

:::::::

Step 2: Configure APM on AWS Lambda [configure_apm_on_aws_lambda]

The {{apm-lambda-ext}} and the APM Python agent are configured through
environment variables on the AWS Lambda function.

For the minimal configuration, you will need the APM Server URL to set
the destination for APM data and an APM Secret Token. If you prefer to
use an APM API key instead of the APM secret token, use the
ELASTIC_APM_API_KEY environment variable instead of
ELASTIC_APM_SECRET_TOKEN in the following configuration.

For production environments, we recommend using the AWS Secrets Manager
to store your APM authentication key instead of providing the secret
value as plaintext in the environment variables.

::::::{tab-item} AWS Web Console To configure APM through the AWS
Management Console:

1.  Navigate to your function in the AWS Management Console
2.  Click on the Configuration tab
3.  Click on Environment variables
4.  Add the following required variables:

    AWS_LAMBDA_EXEC_WRAPPER       = /opt/python/bin/elasticapm-lambda  <1>
    ELASTIC_APM_LAMBDA_APM_SERVER = <YOUR-APM-SERVER-URL>              <2>
    ELASTIC_APM_SECRET_TOKEN      = <YOUR-APM-SECRET-TOKEN>            <3>
    ELASTIC_APM_SEND_STRATEGY     = background                         <4>

1.  Use this exact fixed value.
2.  This is your APM Server URL.
3.  This is your APM secret token.
4.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios.

[Python environment variables configuration section in AWS Console]

::::::{tab-item} AWS CLI To configure APM through the AWS command line
interface execute the following command:

    aws lambda update-function-configuration --function-name yourLambdaFunctionName \
        --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/python/bin/elasticapm-lambda,ELASTIC_APM_LAMBDA_APM_SERVER=<YOUR-APM-SERVER-URL>,ELASTIC_APM_SECRET_TOKEN=<YOUR-APM-SECRET-TOKEN>,ELASTIC_APM_SEND_STRATEGY=background}" <1>

1.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios. ::::::

::::::{tab-item} SAM In your SAM template.yml file configure the
following environment variables:

    ...
    Resources:
      yourLambdaFunction:
        Type: AWS::Serverless::Function
        Properties:
          ...
          Environment:
              Variables:
                AWS_LAMBDA_EXEC_WRAPPER: /opt/python/bin/elasticapm-lambda
                ELASTIC_APM_LAMBDA_APM_SERVER: <YOUR-APM-SERVER-URL>
                ELASTIC_APM_SECRET_TOKEN: <YOUR-APM-SECRET-TOKEN>
                ELASTIC_APM_SEND_STRATEGY: background <1>
    ...

1.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios.

::::::

::::::{tab-item} Serverless In your serverless.yml file configure the
following environment variables:

    ...
    functions:
      yourLambdaFunction:
        ...
        environment:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/python/bin/elasticapm-lambda
          ELASTIC_APM_LAMBDA_APM_SERVER: <YOUR-APM-SERVER-URL>
          ELASTIC_APM_SECRET_TOKEN: <YOUR-APM-SECRET-TOKEN>
          ELASTIC_APM_SEND_STRATEGY: background <1>
    ...

1.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios.

::::::

::::::{tab-item} Terraform In your Terraform file configure the
following environment variables:

    ...
    resource "aws_lambda_function" "your_lambda_function" {
      ...
      environment {
        variables = {
          AWS_LAMBDA_EXEC_WRAPPER       = /opt/python/bin/elasticapm-lambda
          ELASTIC_APM_LAMBDA_APM_SERVER = "<YOUR-APM-SERVER-URL>"
          ELASTIC_APM_SECRET_TOKEN      = "<YOUR-APM-SECRET-TOKEN>"
          ELASTIC_APM_SEND_STRATEGY     = "background" <1>
        }
      }
    }
    ...

1.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios.

::::::

::::::{tab-item} Container Image Environment variables configured for an
AWS Lambda function are passed to the container running the lambda
function. You can use one of the other options (through AWS Web Console,
AWS CLI, etc.) to configure the following environment variables:

    AWS_LAMBDA_EXEC_WRAPPER       = /opt/python/bin/elasticapm-lambda  <1>
    ELASTIC_APM_LAMBDA_APM_SERVER = <YOUR-APM-SERVER-URL>              <2>
    ELASTIC_APM_SECRET_TOKEN      = <YOUR-APM-SECRET-TOKEN>            <3>
    ELASTIC_APM_SEND_STRATEGY     = background                         <4>

1.  Use this exact fixed value.
2.  This is your APM Server URL.
3.  This is your APM secret token.
4.  The ELASTIC_APM_SEND_STRATEGY defines when APM data is sent to your
    Elastic APM backend. To reduce the execution time of your lambda
    functions, we recommend to use the background strategy in production
    environments with steady load scenarios.

::::::

:::::::

You can optionally fine-tune the Python agent or the configuration of
the {{apm-lambda-ext}}.

That’s it. After following the steps above, you’re ready to go! Your
Lambda function invocations should be traced from now on. Spans will be
captured for supported technologies. You can also use capture_span to
capture custom spans, and you can retrieve the Client object for
capturing exceptions/messages using get_client.

Logs [logs]

Elastic Python APM Agent provides the following log features:

- Log correlation : Automatically inject correlation IDs that allow
  navigation between logs, traces and services.
- Log reformatting (experimental) : Automatically reformat plaintext
  logs in ECS logging format.

Elastic Python APM Agent does not send the logs to Elasticsearch. It
only injects correlation IDs and reformats the logs. You must use
another ingestion strategy. We recommend Filebeat for that purpose.

Those features are part of Application log ingestion strategies.

The ecs-logging-python library can also be used to use the ECS logging
format without an APM agent. When deployed with the Python APM agent,
the agent will provide log correlation IDs.

Log correlation [log-correlation-ids]

Log correlation allows you to navigate to all logs belonging to a
particular trace and vice-versa: for a specific log, see in which
context it has been logged and which parameters the user provided.

The Agent provides integrations with both the default Python logging
library, as well as structlog.

- Logging integrations
- Log correlation in Elasticsearch

Logging integrations [logging-integrations]

logging [logging]

We use logging.setLogRecordFactory() to decorate the default
LogRecordFactory to automatically add new attributes to each LogRecord
object:

- elasticapm_transaction_id
- elasticapm_trace_id
- elasticapm_span_id

This factory also adds these fields to a dictionary attribute,
elasticapm_labels, using the official ECS tracing fields.

You can disable this automatic behavior by using the
disable_log_record_factory setting in your configuration.

structlog [structlog]

We provide a processor for structlog which will add three new keys to
the event_dict of any processed event:

- transaction.id
- trace.id
- span.id

    from structlog import PrintLogger, wrap_logger
    from structlog.processors import JSONRenderer
    from elasticapm.handlers.structlog import structlog_processor

    wrapped_logger = PrintLogger()
    logger = wrap_logger(wrapped_logger, processors=[structlog_processor, JSONRenderer()])
    log = logger.new()
    log.msg("some_event")

Use structlog for agent-internal logging [_use_structlog_for_agent_internal_logging]

The Elastic APM Python agent uses logging to log internal events and
issues. By default, it will use a logging logger. If your project uses
structlog, you can tell the agent to use a structlog logger by setting
the environment variable ELASTIC_APM_USE_STRUCTLOG to true.

Log correlation in Elasticsearch [log-correlation-in-es]

In order to correlate logs from your app with transactions captured by
the Elastic APM Python Agent, your logs must contain one or more of the
following identifiers:

- transaction.id
- trace.id
- span.id

If you’re using structured logging, either with a custom solution or
with structlog (recommended), then this is fairly easy. Throw the
JSONRenderer in, and use Filebeat to pull these logs into Elasticsearch.

Without structured logging the task gets a little trickier. Here we
recommend first making sure your LogRecord objects have the elasticapm
attributes (see logging), and then you’ll want to combine some specific
formatting with a Grok pattern, either in Elasticsearch using the grok
processor, or in logstash with a plugin.

Say you have a Formatter that looks like this:

    import logging

    fh = logging.FileHandler('spam.log')
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    fh.setFormatter(formatter)

You can add the APM identifiers by simply switching out the Formatter
object for the one that we provide:

    import logging
    from elasticapm.handlers.logging import Formatter

    fh = logging.FileHandler('spam.log')
    formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    fh.setFormatter(formatter)

This will automatically append apm-specific fields to your format
string:

    formatstring = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    formatstring = formatstring + " | elasticapm " \
                                  "transaction.id=%(elasticapm_transaction_id)s " \
                                  "trace.id=%(elasticapm_trace_id)s " \
                                  "span.id=%(elasticapm_span_id)s"

Then, you could use a grok pattern like this (for the Elasticsearch Grok
Processor):

    {
      "description" : "...",
      "processors": [
        {
          "grok": {
            "field": "message",
            "patterns": ["%{GREEDYDATA:msg} | elasticapm transaction.id=%{DATA:transaction.id} trace.id=%{DATA:trace.id} span.id=%{DATA:span.id}"]
          }
        }
      ]
    }

Log reformatting (experimental) [log-reformatting]

Starting in version 6.16.0, the agent can automatically reformat
application logs to ECS format with no changes to dependencies. Prior
versions must install the ecs_logging dependency.

Log reformatting is controlled by the log_ecs_reformatting configuration
option, and is disabled by default.

The reformatted logs will include both the trace and service correlation
IDs.

Metrics [metrics]

With Elastic APM, you can capture system and process metrics. These
metrics will be sent regularly to the APM Server and from there to
Elasticsearch

Metric sets [metric-sets]

- CPU/Memory metric set
- Breakdown metric set
- Prometheus metric set (beta)
- Custom Metrics

CPU/Memory metric set [cpu-memory-metricset]

elasticapm.metrics.sets.cpu.CPUMetricSet

This metric set collects various system metrics and metrics of the
current process.

if you do not use Linux, you need to install psutil for this metric set.

system.cpu.total.norm.pct
  type: scaled_float

format: percent

The percentage of CPU time in states other than Idle and IOWait,
normalized by the number of cores.

system.process.cpu.total.norm.pct
  type: scaled_float

format: percent

The percentage of CPU time spent by the process since the last event.
This value is normalized by the number of CPU cores and it ranges from 0
to 100%.

system.memory.total
  type: long

format: bytes

Total memory.

system.memory.actual.free
  type: long

format: bytes

Actual free memory in bytes.

system.process.memory.size
  type: long

format: bytes

The total virtual memory the process has.

system.process.memory.rss.bytes
  type: long

format: bytes

The Resident Set Size. The amount of memory the process occupied in main
memory (RAM).

Linux’s cgroup metrics [cpu-memory-cgroup-metricset]

system.process.cgroup.memory.mem.limit.bytes
  type: long

format: bytes

Memory limit for current cgroup slice.

system.process.cgroup.memory.mem.usage.bytes
  type: long

format: bytes

Memory usage in current cgroup slice.

Breakdown metric set [breakdown-metricset]

Tracking and collection of this metric set can be disabled using the
breakdown_metrics setting.

span.self_time
  type: simple timer

This timer tracks the span self-times and is the basis of the
transaction breakdown visualization.

Fields:

- sum: The sum of all span self-times in ms since the last report (the
  delta)
- count: The count of all span self-times since the last report (the
  delta)

You can filter and group by these dimensions:

- transaction.name: The name of the transaction
- transaction.type: The type of the transaction, for example request
- span.type: The type of the span, for example app, template or db
- span.subtype: The sub-type of the span, for example mysql (optional)

Prometheus metric set (beta) [prometheus-metricset]

This functionality is in beta and is subject to change. The design and
code is less mature than official GA features and is being provided
as-is with no warranties. Beta features are not subject to the support
SLA of official GA features.

If you use prometheus_client to collect metrics, the agent can collect
them as well and make them available in Elasticsearch.

The following types of metrics are supported:

- Counters
- Gauges
- Summaries
- Histograms (requires APM Server / Elasticsearch / Kibana 7.14+)

To use the Prometheus metric set, you have to enable it with the
prometheus_metrics configuration option.

All metrics collected from prometheus_client are prefixed with
"prometheus.metrics.". This can be changed using the
prometheus_metrics_prefix configuration option.

Beta limitations [prometheus-metricset-beta]

- The metrics format may change without backwards compatibility in
  future releases.

Custom Metrics [custom-metrics]

Custom metrics allow you to send your own metrics to Elasticsearch.

The most common way to send custom metrics is with the Prometheus metric
set. However, you can also use your own metric set. If you collect the
metrics manually in your code, you can use the base MetricSet class:

    from elasticapm.metrics.base_metrics import MetricSet

    client = elasticapm.Client()
    metricset = client.metrics.register(MetricSet)

    for x in range(10):
        metricset.counter("my_counter").inc()

Alternatively, you can create your own MetricSet class which inherits
from the base class. In this case, you’ll usually want to override the
before_collect method, where you can gather and set metrics before they
are collected and sent to Elasticsearch.

You can add your MetricSet class as shown in the example above, or you
can add an import string for your class to the metrics_sets
configuration option:

    ELASTIC_APM_METRICS_SETS="elasticapm.metrics.sets.cpu.CPUMetricSet,myapp.metrics.MyMetricSet"

Your MetricSet might look something like this:

    from elasticapm.metrics.base_metrics import MetricSet

    class MyAwesomeMetricSet(MetricSet):
        def before_collect(self):
            self.gauge("my_gauge").set(myapp.some_value)

In the example above, the MetricSet would look up myapp.some_value and
set the metric my_gauge to that value. This would happen whenever
metrics are collected/sent, which is controlled by the metrics_interval
setting.

OpenTelemetry API Bridge [opentelemetry-bridge]

The Elastic APM OpenTelemetry bridge allows you to create Elastic APM
Transactions and Spans, using the OpenTelemetry API. This allows users
to utilize the Elastic APM agent’s automatic instrumentations, while
keeping custom instrumentations vendor neutral.

If a span is created while there is no transaction active, it will
result in an Elastic APM Transaction. Inner spans are mapped to Elastic
APM Span.

Getting started [opentelemetry-getting-started]

The first step in getting started with the OpenTelemetry bridge is to
install the opentelemetry libraries:

    pip install elastic-apm[opentelemetry]

Or if you already have installed elastic-apm:

    pip install opentelemetry-api opentelemetry-sdk

Usage [opentelemetry-usage]

    from elasticapm.contrib.opentelemetry import Tracer

    tracer = Tracer(__name__)
    with tracer.start_as_current_span("test"):
        # Do some work

or

    from elasticapm.contrib.opentelemetry import trace

    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("test"):
        # Do some work

Tracer and get_tracer() accept the following optional arguments:

- elasticapm_client: an already instantiated Elastic APM client
- config: a configuration dictionary, which will be used to instantiate
  a new Elastic APM client, e.g. {"SERVER_URL": "https://example.org"}.
  See configuration for more information.

The Tracer object mirrors the upstream interface on the OpenTelemetry
Tracer object.

Caveats [opentelemetry-caveats]

Not all features of the OpenTelemetry API are supported.

Processors, exporters, metrics, logs, span events, and span links are
not supported.

Additionally, due to implementation details, the global context API only
works when a span is included in the activated context, and tokens are
not used. Instead, the global context works as a stack, and when a
context is detached the previously-active context will automatically be
activated.

Performance tuning [tuning-and-overhead]

Using an APM solution comes with certain trade-offs, and the Python
agent for Elastic APM is no different. Instrumenting your code,
measuring timings, recording context data, etc., all need resources:

- CPU time
- memory
- bandwidth use
- Elasticsearch storage

We invested and continue to invest a lot of effort to keep the overhead
of using Elastic APM as low as possible. But because every deployment is
different, there are some knobs you can turn to adapt it to your
specific needs.

Transaction Sample Rate [tuning-sample-rate]

The easiest way to reduce the overhead of the agent is to tell the agent
to do less. If you set the transaction_sample_rate to a value below 1.0,
the agent will randomly sample only a subset of transactions. Unsampled
transactions only record the name of the transaction, the overall
transaction time, and the result:

  Field              Sampled   Unsampled
  ------------------ --------- -----------
  Transaction name   yes       yes
  Duration           yes       yes
  Result             yes       yes
  Context            yes       no
  Tags               yes       no
  Spans              yes       no

Reducing the sample rate to a fraction of all transactions can make a
huge difference in all four of the mentioned resource types.

Transaction Queue [tuning-queue]

To reduce the load on the APM Server, the agent does not send every
transaction up as it happens. Instead, it queues them up and flushes the
queue periodically, or when it reaches a maximum size, using a
background thread.

While this reduces the load on the APM Server (and to a certain extent
on the agent), holding on to the transaction data in a queue uses
memory. If you notice that using the Python agent results in a large
increase of memory use, you can use these settings:

- api_request_time to reduce the time between queue flushes
- api_request_size to reduce the maximum size of the queue

The first setting, api_request_time, is helpful if you have a sustained
high number of transactions. The second setting, api_request_size, can
help if you experience peaks of transactions (a large number of
transactions in a short period of time).

Keep in mind that reducing the value of either setting will cause the
agent to send more HTTP requests to the APM Server, potentially causing
a higher load.

Spans per transaction [tuning-max-spans]

The average amount of spans per transaction can influence how much time
the agent spends in each transaction collecting contextual data for each
span, and the storage space needed in Elasticsearch. In our experience,
most usual transactions should have well below 100 spans. In some cases,
however, the number of spans can explode:

- long-running transactions
- unoptimized code, e.g. doing hundreds of SQL queries in a loop

To avoid these edge cases overloading both the agent and the APM Server,
the agent stops recording spans when a specified limit is reached. You
can configure this limit by changing the transaction_max_spans setting.

Span Stack Trace Collection [tuning-span-stack-trace-collection]

Collecting stack traces for spans can be fairly costly from a
performance standpoint. Stack traces are very useful for pinpointing
which part of your code is generating a span; however, these stack
traces are less useful for very short spans (as problematic spans tend
to be longer).

You can define a minimal threshold for span duration using the
span_stack_trace_min_duration setting. If a span’s duration is less than
this config value, no stack frames will be collected for this span.

Collecting Frame Context [tuning-frame-context]

When a stack trace is captured, the agent will also capture several
lines of source code around each frame location in the stack trace. This
allows the APM app to give greater insight into where exactly the error
or span happens.

There are four settings you can modify to control this behavior:

- source_lines_error_app_frames
- source_lines_error_library_frames
- source_lines_span_app_frames
- source_lines_span_library_frames

As you can see, these settings are divided between app frames, which
represent your application code, and library frames, which represent the
code of your dependencies. Each of these categories are also split into
separate error and span settings.

Reading source files inside a running application can cause a lot of
disk I/O, and sending up source lines for each frame will have a network
and storage cost that is quite high. Turning down these limits will help
prevent excessive memory usage.

Collecting headers and request body [tuning-body-headers]

You can configure the Elastic APM agent to capture headers of both
requests and responses (capture_headers), as well as request bodies
(capture_body). By default, capturing request bodies is disabled.
Enabling it for transactions may introduce noticeable overhead, as well
as increased storage use, depending on the nature of your POST requests.
In most scenarios, we advise against enabling request body capturing for
transactions, and only enable it if necessary for errors.

Capturing request/response headers has less overhead on the agent, but
can have an impact on storage use. If storage use is a problem for you,
it might be worth disabling.

Run Tests Locally [run-tests-locally]

To run tests locally you can make use of the docker images also used
when running the whole test suite with Jenkins. Running the full test
suite first does some linting and then runs the actual tests with
different versions of Python and different web frameworks. For a full
overview of the test matrix and supported versions have a look at
Jenkins Configuration.

Pre Commit [pre-commit]

We run our git hooks on every commit to automatically point out issues
in code. Those issues are also detected within the GitHub actions.
Please follow the installation steps stated in
https://pre-commit.com/#install.

Code Linter [coder-linter]

We run two code linters isort and flake8. You can trigger each single
one locally by running:

    $ pre-commit run -a isort

    $ pre-commit run -a flake8

Code Formatter [coder-formatter]

We test that the code is formatted using black. You can trigger this
check by running:

    $ pre-commit run -a black

Test Documentation [test-documentation]

We test that the documentation can be generated without errors. You can
trigger this check by running:

    $ ./tests/scripts/docker/docs.sh

Running Tests [running-tests]

We run the test suite on different combinations of Python versions and
web frameworks. For triggering the test suite for a specific combination
locally you can run:

    $ ./tests/scripts/docker/run_tests.sh python-version framework-version <pip-cache-dir>

The python-version must be of format python-version, e.g. python-3.6 or
pypy-2. The framework must be of format framework-version,
e.g. django-1.10 or flask-0.12.

You can also run the unit tests outside of docker, by installing the
relevant requirements file and then running py.test from the project
root.

Integration testing [_integration_testing]

Check out https://github.com/elastic/apm-integration-testing for
resources for setting up full end-to-end testing environments. For
example, to spin up an environment with the opbeans Django app, with
version 7.3 of the elastic stack and the apm-python-agent from your
local checkout, you might do something like this:

    $ ./scripts/compose.py start 7.3 \
        --with-agent-python-django --with-opbeans-python \
        --opbeans-python-agent-local-repo=~/elastic/apm-agent-python

Sanic Support [sanic-support]

Incorporating Elastic APM into your Sanic project only requires a few
easy steps.

Installation [sanic-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add elastic-apm to your project’s requirements.txt file.

Setup [sanic-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, or as
initialization arguments.

You can find a list of all available settings in the Configuration page.

To initialize the agent for your application using environment
variables:

    from sanic import Sanic
    from elasticapm.contrib.sanic import ElasticAPM

    app = Sanic(name="elastic-apm-sample")
    apm = ElasticAPM(app=app)

To configure the agent using initialization arguments and Sanic’s
Configuration infrastructure:

    # Create a file named external_config.py in your application
    # If you want this module based configuration to be used for APM, prefix them with ELASTIC_APM_
    ELASTIC_APM_SERVER_URL = "https://serverurl.example.com:443"
    ELASTIC_APM_SECRET_TOKEN = "sometoken"

    from sanic import Sanic
    from elasticapm.contrib.sanic import ElasticAPM

    app = Sanic(name="elastic-apm-sample")
    app.config.update_config("path/to/external_config.py")
    apm = ElasticAPM(app=app)

Usage [sanic-usage]

Once you have configured the agent, it will automatically track
transactions and capture uncaught exceptions within sanic.

Capture an arbitrary exception by calling capture_exception:

    from sanic import Sanic
    from elasticapm.contrib.sanic import ElasticAPM

    app = Sanic(name="elastic-apm-sample")
    apm = ElasticAPM(app=app)

    try:
        1 / 0
    except ZeroDivisionError:
        apm.capture_exception()

Log a generic message with capture_message:

    from sanic import Sanic
    from elasticapm.contrib.sanic import ElasticAPM

    app = Sanic(name="elastic-apm-sample")
    apm = ElasticAPM(app=app)

    apm.capture_message('hello, world!')

Performance metrics [sanic-performance-metrics]

If you’ve followed the instructions above, the agent has installed our
instrumentation middleware which will process all requests through your
app. This will measure response times, as well as detailed performance
data for all supported technologies.

Due to the fact that asyncio drivers are usually separate from their
synchronous counterparts, specific instrumentation is needed for all
drivers. The support for asynchronous drivers is currently quite
limited.

Ignoring specific routes [sanic-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific routes. The list given should be a list of regular
expressions which are matched against the transaction name:

    from sanic import Sanic
    from elasticapm.contrib.sanic import ElasticAPM

    app = Sanic(name="elastic-apm-sample")
    apm = ElasticAPM(app=app, config={
        'TRANSACTIONS_IGNORE_PATTERNS': ['^GET /secret', '/extra_secret'],
    })

This would ignore any requests using the GET /secret route and any
requests containing /extra_secret.

Extended Sanic APM Client Usage [extended-sanic-usage]

Sanic’s contributed APM client also provides a few extendable way to
configure selective behaviors to enhance the information collected as
part of the transactions being tracked by the APM.

In order to enable this behavior, the APM Client middleware provides a
few callback functions that you can leverage in order to simplify the
process of generating additional contexts into the traces being
collected.

  -------------------------------------------------------------------------------------------------------
  Callback Name               Callback Invocation Format           Expected Return      Is Async
                                                                   Format               
  --------------------------- ------------------------------------ -------------------- -----------------
  transaction_name_callback   transaction_name_callback(request)   string               false

  user_context_callback       user_context_callback(request)       (username_string,    true
                                                                   user_email_string,   
                                                                   userid_string)       

  custom_context_callback     custom_context_callback(request) or  dict(str=str)        true
                              custom_context_callback(response)                         

  label_info_callback         label_info_callback()                dict(str=str)        true
  -------------------------------------------------------------------------------------------------------

Supported Sanic and Python versions [supported-stanic-and-python-versions]

A list of supported Sanic and Python versions can be found on our
Supported Technologies page.

Elastic APM only supports asyncio when using Python 3.7+

Sanitizing data [sanitizing-data]

Sometimes it is necessary to sanitize the data sent to Elastic APM,
e.g. remove sensitive data.

To do this with the Elastic APM module, you create a processor. A
processor is a function that takes a client instance as well as an event
(an error, a transaction, a span, or a metricset), and returns the
modified event.

To completely drop an event, your processor should return False (or any
other “falsy” value) instead of the event.

An event will also be dropped if any processor raises an exception while
processing it. A log message with level WARNING will be issued in this
case.

This is an example of a processor that removes the exception stacktrace
from an error:

    from elasticapm.conf.constants import ERROR
    from elasticapm.processors import for_events

    @for_events(ERROR)
    def my_processor(client, event):
        if 'exception' in event and 'stacktrace' in event['exception']:
            event['exception'].pop('stacktrace')
        return event

You can use the @for_events decorator to limit for which event type the
processor should be called. Possible choices are ERROR, TRANSACTION,
SPAN and METRICSET, all of which are defined in
elasticapm.conf.constants.

To use this processor, update your ELASTIC_APM settings like this:

    ELASTIC_APM = {
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
        'PROCESSORS': (
            'path.to.my_processor',
            'elasticapm.processors.sanitize_stacktrace_locals',
            'elasticapm.processors.sanitize_http_request_cookies',
            'elasticapm.processors.sanitize_http_headers',
            'elasticapm.processors.sanitize_http_wsgi_env',
            'elasticapm.processors.sanitize_http_request_body',
        ),
    }

We recommend using the above list of processors that sanitize passwords
and secrets in different places of the event object.

The default set of processors sanitize fields based on a set of defaults
defined in elasticapm.conf.constants. This set can be configured with
the SANITIZE_FIELD_NAMES configuration option. For example, if your
application produces a sensitive field called My-Sensitive-Field, the
default processors can be used to automatically sanitize this field. You
can specify what fields to sanitize within default processors like this:

    ELASTIC_APM = {
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
        'SANITIZE_FIELD_NAMES': (
            "password",
            "passwd",
            "pwd",
            "secret",
            "*key",
            "*token*",
            "*session*",
            "*credit*",
            "*card*",
            "*auth*",
            "set-cookie",
        ),
    }

We recommend to use the above list of fields to sanitize various parts
of the event object in addition to your specified fields.

When choosing fields names to sanitize, you can specify values that will
match certain wildcards. For example, passing base as a field name to be
sanitized will also sanitize all fields whose names match the regex
pattern \*base*.

Set up the APM Python Agent [set-up]

To get you off the ground, we’ve prepared guides for setting up the
Agent with different frameworks:

- Django
- Flask
- aiohttp
- Tornado
- Starlette/FastAPI
- Sanic
- AWS Lambda
- Azure Functions
- Wrapper (Experimental)
- ASGI Middleware

For custom instrumentation, see Instrumenting Custom Code.

Starlette/FastAPI Support [starlette-support]

Incorporating Elastic APM into your Starlette project only requires a
few easy steps.

Installation [starlette-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add elastic-apm to your project’s requirements.txt file.

Setup [starlette-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, or as
initialization arguments.

You can find a list of all available settings in the Configuration page.

To initialize the agent for your application using environment
variables, add the ElasticAPM middleware to your Starlette application:

    from starlette.applications import Starlette
    from elasticapm.contrib.starlette import ElasticAPM

    app = Starlette()
    app.add_middleware(ElasticAPM)

BaseHTTPMiddleware breaks contextvar propagation, as noted here. This
means the ElasticAPM middleware must be above any BaseHTTPMiddleware in
the final middleware list. If you’re calling add_middleware repeatedly,
add the ElasticAPM middleware last. If you’re passing in a list of
middleware, ElasticAPM should be first on that list.

To configure the agent using initialization arguments:

    from starlette.applications import Starlette
    from elasticapm.contrib.starlette import make_apm_client, ElasticAPM

    apm = make_apm_client({
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
        'SERVER_URL': '<SERVER-URL>',
    })
    app = Starlette()
    app.add_middleware(ElasticAPM, client=apm)

FastAPI [starlette-fastapi]

Because FastAPI supports Starlette middleware, using the agent with
FastAPI is almost exactly the same as with Starlette:

    from fastapi import FastAPI
    from elasticapm.contrib.starlette import ElasticAPM

    app = FastAPI()
    app.add_middleware(ElasticAPM)

Usage [starlette-usage]

Once you have configured the agent, it will automatically track
transactions and capture uncaught exceptions within starlette.

Capture an arbitrary exception by calling capture_exception:

    try:
        1 / 0
    except ZeroDivisionError:
        apm.capture_exception()

Log a generic message with capture_message:

    apm.capture_message('hello, world!')

Performance metrics [starlette-performance-metrics]

If you’ve followed the instructions above, the agent has installed our
instrumentation middleware which will process all requests through your
app. This will measure response times, as well as detailed performance
data for all supported technologies.

Due to the fact that asyncio drivers are usually separate from their
synchronous counterparts, specific instrumentation is needed for all
drivers. The support for asynchronous drivers is currently quite
limited.

Ignoring specific routes [starlette-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific routes. The list given should be a list of regular
expressions which are matched against the transaction name:

    apm = make_apm_client({
        # ...
        'TRANSACTIONS_IGNORE_PATTERNS': ['^GET /secret', '/extra_secret']
        # ...
    })

This would ignore any requests using the GET /secret route and any
requests containing /extra_secret.

Supported Starlette and Python versions [supported-starlette-and-python-versions]

A list of supported Starlette and Python versions can be found on our
Supported Technologies page.

Elastic APM only supports asyncio when using Python 3.7+

Supported technologies [supported-technologies]

The Elastic APM Python Agent supports the technologies listed below.

Elastic supports OpenTelemetry, which allows logs, metrics, and trace
signal collection for many of these technologies. Consider using the
EDOT Python SDK for observability data so you continue to get the full
power of Elastic’s platform.

$$$framework-support$$
$ The Elastic APM Python Agent comes with support for the following
frameworks:

- Django
- Flask
- Aiohttp Server
- Tornado
- Starlette/FastAPI
- Sanic
- GRPC

For other frameworks and custom Python code, the agent exposes a set of
APIs for integration.

Python [supported-python]

The following Python versions are supported:

- 3.6
- 3.7
- 3.8
- 3.9
- 3.10
- 3.11
- 3.12
- 3.13

Django [supported-django]

We support these Django versions:

- 1.11
- 2.0
- 2.1
- 2.2
- 3.0
- 3.1
- 3.2
- 4.0
- 4.2
- 5.0

For upcoming Django versions, we generally aim to ensure compatibility
starting with the first Release Candidate.

we currently don’t support Django running in ASGI mode.

Flask [supported-flask]

We support these Flask versions:

- 0.10 (Deprecated)
- 0.11 (Deprecated)
- 0.12 (Deprecated)
- 1.0
- 1.1
- 2.0
- 2.1
- 2.2
- 2.3
- 3.0

Aiohttp Server [supported-aiohttp]

We support these aiohttp versions:

- 3.x

Tornado [supported-tornado]

We support these tornado versions:

- 6.x

Sanic [supported-sanic]

We support these sanic versions:

-   20.12.2,<26

Starlette/FastAPI [supported-starlette]

We support these Starlette versions:

-   0.13.0,<1

Any FastAPI version which uses a supported Starlette version should also
be supported.

GRPC [supported-grpc]

We support these grpcio versions:

-   1.24.0,<2

Automatic Instrumentation [automatic-instrumentation]

The Python APM agent comes with automatic instrumentation of various 3rd
party modules and standard library modules.

Scheduling [automatic-instrumentation-scheduling]

Celery [automatic-instrumentation-scheduling-celery]

We support these Celery versions:

- 4.x (deprecated)
- 5.x

Celery tasks will be recorded automatically with Django and Flask only.

Databases [automatic-instrumentation-db]

Elasticsearch [automatic-instrumentation-db-elasticsearch]

Instrumented methods:

- elasticsearch.transport.Transport.perform_request
- elasticsearch.connection.http_urllib3.Urllib3HttpConnection.perform_request
- elasticsearch.connection.http_requests.RequestsHttpConnection.perform_request
- elasticsearch._async.transport.AsyncTransport.perform_request
- elasticsearch_async.connection.AIOHttpConnection.perform_request

Additionally, the instrumentation wraps the following methods of the
Elasticsearch client class:

- elasticsearch.client.Elasticsearch.delete_by_query
- elasticsearch.client.Elasticsearch.search
- elasticsearch.client.Elasticsearch.count
- elasticsearch.client.Elasticsearch.update

Collected trace data:

- the query string (if available)
- the query element from the request body (if available)
- the response status code
- the count of affected rows (if available)

We recommend using keyword arguments only with elasticsearch-py, as
recommended by the elasticsearch-py docs. If you are using positional
arguments, we will be unable to gather the query element from the
request body.

SQLite [automatic-instrumentation-db-sqlite]

Instrumented methods:

- sqlite3.connect
- sqlite3.dbapi2.connect
- pysqlite2.dbapi2.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

MySQLdb [automatic-instrumentation-db-mysql]

Library: MySQLdb (<2)

Instrumented methods:

- MySQLdb.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

mysql-connector [automatic-instrumentation-db-mysql-connector]

Library: mysql-connector-python (<9)

Instrumented methods:

- mysql.connector.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

pymysql [automatic-instrumentation-db-pymysql]

Library: pymysql (<2)

Instrumented methods:

- pymysql.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

aiomysql [automatic-instrumentation-db-aiomysql]

Library: aiomysql (<1)

Instrumented methods:

- aiomysql.cursors.Cursor.execute

Collected trace data:

- parametrized SQL query

PostgreSQL Psycopg2 [automatic-instrumentation-db-postgres]

Library: psycopg2, psycopg2-binary (>=2.9,<3)

Instrumented methods:

- psycopg2.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

PostgreSQL Psycopg [automatic-instrumentation-db-postgres-psycopg]

Library: psycopg, psycopg-binary (>3.0.0,<4)

Instrumented methods:

- psycopg.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

aiopg [automatic-instrumentation-db-aiopg]

Library: aiopg (>=1.0,<2)

Instrumented methods:

- aiopg.cursor.Cursor.execute
- aiopg.cursor.Cursor.callproc

Collected trace data:

- parametrized SQL query

asyncpg [automatic-instrumentation-db-asyncg]

Library: asyncpg (>=0.20,<2)

Instrumented methods:

- asyncpg.connection.Connection.execute
- asyncpg.connection.Connection.executemany

Collected trace data:

- parametrized SQL query

PyODBC [automatic-instrumentation-db-pyodbc]

Library: pyodbc (>=4.0,<6)

Instrumented methods:

- pyodbc.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

MS-SQL [automatic-instrumentation-db-mssql]

Library: pymssql (>=2.1.0,<3)

Instrumented methods:

- pymssql.connect

The instrumented connect method returns a wrapped connection/cursor
which instruments the actual Cursor.execute calls.

Collected trace data:

- parametrized SQL query

MongoDB [automatic-instrumentation-db-mongodb]

Library: pymongo (>=2.9,<5)

Instrumented methods:

- pymongo.collection.Collection.aggregate
- pymongo.collection.Collection.bulk_write
- pymongo.collection.Collection.count
- pymongo.collection.Collection.create_index
- pymongo.collection.Collection.create_indexes
- pymongo.collection.Collection.delete_many
- pymongo.collection.Collection.delete_one
- pymongo.collection.Collection.distinct
- pymongo.collection.Collection.drop
- pymongo.collection.Collection.drop_index
- pymongo.collection.Collection.drop_indexes
- pymongo.collection.Collection.ensure_index
- pymongo.collection.Collection.find_and_modify
- pymongo.collection.Collection.find_one
- pymongo.collection.Collection.find_one_and_delete
- pymongo.collection.Collection.find_one_and_replace
- pymongo.collection.Collection.find_one_and_update
- pymongo.collection.Collection.group
- pymongo.collection.Collection.inline_map_reduce
- pymongo.collection.Collection.insert
- pymongo.collection.Collection.insert_many
- pymongo.collection.Collection.insert_one
- pymongo.collection.Collection.map_reduce
- pymongo.collection.Collection.reindex
- pymongo.collection.Collection.remove
- pymongo.collection.Collection.rename
- pymongo.collection.Collection.replace_one
- pymongo.collection.Collection.save
- pymongo.collection.Collection.update
- pymongo.collection.Collection.update_many
- pymongo.collection.Collection.update_one

Collected trace data:

- database name
- method name

Redis [automatic-instrumentation-db-redis]

Library: redis (>=2.8,<8)

Instrumented methods:

- redis.client.Redis.execute_command
- redis.client.Pipeline.execute

Collected trace data:

- Redis command name

aioredis [automatic-instrumentation-db-aioredis]

Library: aioredis (<=2.0.1)

Instrumented methods:

- aioredis.pool.ConnectionsPool.execute
- aioredis.commands.transaction.Pipeline.execute
- aioredis.connection.RedisConnection.execute

Collected trace data:

- Redis command name

Cassandra [automatic-instrumentation-db-cassandra]

Library: cassandra-driver (>=3.24,<4.0)

Instrumented methods:

- cassandra.cluster.Session.execute
- cassandra.cluster.Cluster.connect

Collected trace data:

- CQL query

Python Memcache [automatic-instrumentation-db-python-memcache]

Library: python-memcached (>=1.51,<2)

Instrumented methods:

- memcache.Client.add
- memcache.Client.append
- memcache.Client.cas
- memcache.Client.decr
- memcache.Client.delete
- memcache.Client.delete_multi
- memcache.Client.disconnect_all
- memcache.Client.flush_all
- memcache.Client.get
- memcache.Client.get_multi
- memcache.Client.get_slabs
- memcache.Client.get_stats
- memcache.Client.gets
- memcache.Client.incr
- memcache.Client.prepend
- memcache.Client.replace
- memcache.Client.set
- memcache.Client.set_multi
- memcache.Client.touch

Collected trace data:

- Destination (address and port)

pymemcache [automatic-instrumentation-db-pymemcache]

Library: pymemcache (>=3.0,<4.1)

Instrumented methods:

- pymemcache.client.base.Client.add
- pymemcache.client.base.Client.append
- pymemcache.client.base.Client.cas
- pymemcache.client.base.Client.decr
- pymemcache.client.base.Client.delete
- pymemcache.client.base.Client.delete_many
- pymemcache.client.base.Client.delete_multi
- pymemcache.client.base.Client.flush_all
- pymemcache.client.base.Client.get
- pymemcache.client.base.Client.get_many
- pymemcache.client.base.Client.get_multi
- pymemcache.client.base.Client.gets
- pymemcache.client.base.Client.gets_many
- pymemcache.client.base.Client.incr
- pymemcache.client.base.Client.prepend
- pymemcache.client.base.Client.quit
- pymemcache.client.base.Client.replace
- pymemcache.client.base.Client.set
- pymemcache.client.base.Client.set_many
- pymemcache.client.base.Client.set_multi
- pymemcache.client.base.Client.stats
- pymemcache.client.base.Client.touch

Collected trace data:

- Destination (address and port)

kafka-python [automatic-instrumentation-db-kafka-python]

Library: kafka-python (>=2.0,<3)

Instrumented methods:

- kafka.KafkaProducer.send,
- kafka.KafkaConsumer.poll,
- kafka.KafkaConsumer.__next__

Collected trace data:

- Destination (address and port)
- topic (if applicable)

External HTTP requests [automatic-instrumentation-http]

Standard library [automatic-instrumentation-stdlib-urllib]

Library: urllib.request (Python 3)

Instrumented methods:

- urllib.request.AbstractHTTPHandler.do_open

Collected trace data:

- HTTP method
- requested URL

urllib3 [automatic-instrumentation-urllib3]

Library: urllib3 (<3)

Instrumented methods:

- urllib3.connectionpool.HTTPConnectionPool.urlopen

Additionally, we instrumented vendored instances of urllib3 in the
following libraries:

- requests
- botocore

Both libraries have “unvendored” urllib3 in more recent versions, we
recommend to use the newest versions.

Collected trace data:

- HTTP method
- requested URL

requests [automatic-instrumentation-requests]

Library: requests (<3)

Instrumented methods:

- requests.sessions.Session.send

Collected trace data:

- HTTP method
- requested URL

AIOHTTP Client [automatic-instrumentation-aiohttp-client]

Library: aiohttp (>=3,<4)

Instrumented methods:

- aiohttp.client.ClientSession._request

Collected trace data:

- HTTP method
- requested URL

httpx [automatic-instrumentation-httpx]

Library: httpx (<1)

Instrumented methods:

- `httpx.Client.send

Collected trace data:

- HTTP method
- requested URL

Services [automatic-instrumentation-services]

AWS Boto3 / Botocore [automatic-instrumentation-boto3]

Library: boto3 (>=1.0,<2)

Instrumented methods:

- botocore.client.BaseClient._make_api_call

Collected trace data for all services:

- AWS region (e.g. eu-central-1)
- AWS service name (e.g. s3)
- operation name (e.g. ListBuckets)

Additionally, some services collect more specific data

AWS Aiobotocore [automatic-instrumentation-aiobotocore]

Library: aiobotocore (>=2.2.0,<3)

Instrumented methods:

- aiobotocore.client.BaseClient._make_api_call

Collected trace data for all services:

- AWS region (e.g. eu-central-1)
- AWS service name (e.g. s3)
- operation name (e.g. ListBuckets)

Additionally, some services collect more specific data

S3 [automatic-instrumentation-s3]

- Bucket name

DynamoDB [automatic-instrumentation-dynamodb]

- Table name

SNS [automatic-instrumentation-sns]

- Topic name

SQS [automatic-instrumentation-sqs]

- Queue name

Template Engines [automatic-instrumentation-template-engines]

Django Template Language [automatic-instrumentation-dtl]

Library: Django (see Django for supported versions)

Instrumented methods:

- django.template.Template.render

Collected trace data:

- template name

Jinja2 [automatic-instrumentation-jinja2]

Library: jinja2

Instrumented methods:

- jinja2.Template.render

Collected trace data:

- template name

Tornado Support [tornado-support]

Incorporating Elastic APM into your Tornado project only requires a few
easy steps.

Installation [tornado-installation]

Install the Elastic APM agent using pip:

    $ pip install elastic-apm

or add elastic-apm to your project’s requirements.txt file.

Setup [tornado-setup]

To set up the agent, you need to initialize it with appropriate
settings.

The settings are configured either via environment variables, the
application’s settings, or as initialization arguments.

You can find a list of all available settings in the Configuration page.

To initialize the agent for your application using environment
variables:

    import tornado.web
    from elasticapm.contrib.tornado import ElasticAPM

    app = tornado.web.Application()
    apm = ElasticAPM(app)

To configure the agent using ELASTIC_APM in your application’s settings:

    import tornado.web
    from elasticapm.contrib.tornado import ElasticAPM

    app = tornado.web.Application()
    app.settings['ELASTIC_APM'] = {
        'SERVICE_NAME': '<SERVICE-NAME>',
        'SECRET_TOKEN': '<SECRET-TOKEN>',
    }
    apm = ElasticAPM(app)

Usage [tornado-usage]

Once you have configured the agent, it will automatically track
transactions and capture uncaught exceptions within tornado.

Capture an arbitrary exception by calling capture_exception:

    try:
        1 / 0
    except ZeroDivisionError:
        apm.client.capture_exception()

Log a generic message with capture_message:

    apm.client.capture_message('hello, world!')

Performance metrics [tornado-performance-metrics]

If you’ve followed the instructions above, the agent has installed our
instrumentation within the base RequestHandler class in tornado.web.
This will measure response times, as well as detailed performance data
for all supported technologies.

Due to the fact that asyncio drivers are usually separate from their
synchronous counterparts, specific instrumentation is needed for all
drivers. The support for asynchronous drivers is currently quite
limited.

Ignoring specific routes [tornado-ignoring-specific-views]

You can use the TRANSACTIONS_IGNORE_PATTERNS configuration option to
ignore specific routes. The list given should be a list of regular
expressions which are matched against the transaction name:

    app.settings['ELASTIC_APM'] = {
        # ...
        'TRANSACTIONS_IGNORE_PATTERNS': ['^GET SecretHandler', 'MainHandler']
        # ...
    }

This would ignore any requests using the GET SecretHandler route and any
requests containing MainHandler.

Supported tornado and Python versions [supported-tornado-and-python-versions]

A list of supported tornado and Python versions can be found on our
Supported Technologies page.

Elastic APM only supports asyncio when using Python 3.7+

Upgrading to version 4 of the agent [upgrading-4-x]

4.0 of the Elastic APM Python Agent comes with several backwards
incompatible changes.

APM Server 6.5 required [upgrading-4-x-apm-server]

This version of the agent is only compatible with APM Server 6.5+. To
upgrade, we recommend to first upgrade APM Server, and then the agent.
APM Server 6.5+ is backwards compatible with versions 2.x and 3.x of the
agent.

Configuration options [upgrading-4-x-configuration]

Several configuration options have been removed, or renamed

- flush_interval has been removed
- the flush_interval and max_queue_size settings have been removed.
- new settings introduced: api_request_time and api_request_size.
- Some settings now require a unit for duration or size. See size format
  and duration format.

Processors [upgrading-4-x-processors]

The method to write processors for sanitizing events has been changed.
It will now be called for every type of event (transactions, spans and
errors), unless the event types are limited using a decorator. See
Sanitizing data for more information.

Upgrading to version 5 of the agent [upgrading-5-x]

APM Server 7.3 required for some features [_apm_server_7_3_required_for_some_features]

APM Server and Kibana 7.3 introduced support for collecting breakdown
metrics, and central configuration of APM agents. To use these features,
please update the Python agent to 5.0+ and APM Server / Kibana to 7.3+

Tags renamed to Labels [_tags_renamed_to_labels]

To better align with other parts of the Elastic Stack and the Elastic
Common Schema, we renamed “tags” to “labels”, and introduced limited
support for typed labels. While tag values were only allowed to be
strings, label values can be strings, booleans, or numerical.

To benefit from this change, ensure that you run at least APM Server
6.7, and use elasticapm.label() instead of elasticapm.tag(). The tag()
API will continue to work as before, but emit a DeprecationWarning. It
will be removed in 6.0 of the agent.

Upgrading to version 6 of the agent [upgrading-6-x]

Python 2 no longer supported [_python_2_no_longer_supported]

Please upgrade to Python 3.6+ to continue to receive regular updates.

SANITIZE_FIELD_NAMES changes [_sanitize_field_names_changes]

If you are using a non-default sanitize_field_names config, please note
that your entries must be surrounded with stars (e.g. *secret*) in order
to maintain previous behavior.

Tags removed (in favor of labels) [_tags_removed_in_favor_of_labels]

Tags were deprecated in the 5.x release (in favor of labels). They have
now been removed.

Upgrading [upgrading]

Upgrades between minor versions of the agent, like from 3.1 to 3.2 are
always backwards compatible. Upgrades that involve a major version bump
often come with some backwards incompatible changes.

We highly recommend to always pin the version of elastic-apm in your
requirements.txt or Pipfile. This avoids automatic upgrades to
potentially incompatible versions.

End of life dates [end-of-life-dates]

We love all our products, but sometimes we must say goodbye to a release
so that we can continue moving forward on future development and
innovation. Our End of life policy defines how long a given release is
considered supported, as well as how long a release is considered still
in active development or maintenance.

Wrapper Support [wrapper-support]

This functionality is in technical preview and may be changed or removed
in a future release. Elastic will work to fix any issues, but features
in technical preview are not subject to the support SLA of official GA
features.

The following frameworks are supported using our new wrapper script for
no-code-changes instrumentation:

- Django
- Flask
- Starlette

Please keep in mind that these instrumentations are a work in progress!
We’d love to have feedback on our issue tracker.

Usage [wrapper-usage]

When installing the agent, an entrypoint script, elasticapm-run is
installed as well. You can use this script to instrument your app
(assuming it’s using a supported framework) without changing your code!

    $ elasticapm-run --version
    elasticapm-run 6.14.0

Alternatively, you can run the entrypoint directly:

    $ python -m elasticapm.instrumentation.wrapper --version
    elasticapm-run 6.14.0

The elasticapm-run script can be used to run any Python script or
module:

    $ elasticapm-run flask run
    $ elasticapm-run python myapp.py

Generally, config should be passed in via environment variables. For
example,

    $ ELASTIC_APM_SERVICE_NAME=my_flask_app elasticapm-run flask run

You can also pass config options as arguments to the script:

    $ elasticapm-run --config "service_name=my_flask_app" --config "debug=true" flask run
