
# Zookeeper by HTTP

## Overview

This template is designed for the effortless deployment of Apache Zookeeper monitoring by Zabbix via HTTP and doesn't require any external scripts.

This template works with standalone and cluster instances. Metrics are collected from each Zookeeper node by requests to AdminServer.

## Requirements

Zabbix version: 7.0 and higher.

## Tested versions

This template has been tested on:
- Apache Zookeeper, version 3.6+, 3.8+

## Configuration

> Zabbix should be configured according to the instructions in the [Templates out of the box](https://www.zabbix.com/documentation/7.0/manual/config/templates_out_of_the_box) section.

## Setup

1. Enable the AdminServer and configure the parameters according to the [`official documentation`](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_adminserver_config).

2. Set the hostname or IP address of the Apache Zookeeper host in the `{$ZOOKEEPER.HOST}` macro. You can also change the `{$ZOOKEEPER.COMMAND_URL}`, `{$ZOOKEEPER.PORT}` and `{$ZOOKEEPER.SCHEME}` macros if necessary.

### Macros used

|Name|Description|Default|
|----|-----------|-------|
|{$ZOOKEEPER.HOST}|
The hostname or IP address of the Apache Zookeeper host.
|`<SET ZOOKEEPER HOST>`|
|{$ZOOKEEPER.PORT}|
The port the embedded Jetty server listens on (admin.serverPort).
|`8080`|
|{$ZOOKEEPER.COMMAND_URL}|
The URL for listing and issuing commands relative to the root URL (admin.commandURL).
|`commands`|
|{$ZOOKEEPER.SCHEME}|
Request scheme which may be http or https
|`http`|
|{$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN}|
Maximum percentage of file descriptors usage alert threshold (for trigger expression).
|`85`|
|{$ZOOKEEPER.OUTSTANDING_REQ.MAX.WARN}|
Maximum number of outstanding requests (for trigger expression).
|`10`|
|{$ZOOKEEPER.PENDING_SYNCS.MAX.WARN}|
Maximum number of pending syncs from the followers (for trigger expression).
|`10`|

### Items

|Name|Description|Type|Key and additional info|
|----|-----------|----|-----------------------|
|Get server metrics||HTTP agent|zookeeper.get_metrics|
|Get connections stats|
Get information on client connections to server. Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance).
|HTTP agent|zookeeper.get_connections_stats|
|Server mode|
Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone
|Dependent item|zookeeper.server_state
**Preprocessing**

- 
JSON Path: `$.server_state`

- 
Discard unchanged with heartbeat: `1h`

|
|Uptime|
Uptime that a peer has been in a table leading/following/observing state.
|Dependent item|zookeeper.uptime
**Preprocessing**

- 
JSON Path: `$.uptime`

- 
Custom multiplier: `0.001`

|
|Version|
Version of Zookeeper server.
|Dependent item|zookeeper.version
**Preprocessing**

- 
JSON Path: `$.version`

- 
Regular expression: `^([0-9\.]+) \1`

- 
Discard unchanged with heartbeat: `3h`

|
|Approximate data size|
Data tree size in bytes.The size includes the znode path and its value.
|Dependent item|zookeeper.approximate_data_size
**Preprocessing**

- 
JSON Path: `$.approximate_data_size`

|
|File descriptors, max|
Maximum number of file descriptors that a zookeeper server can open.
|Dependent item|zookeeper.max_file_descriptor_count
**Preprocessing**

- 
JSON Path: `$.max_file_descriptor_count`

- 
Discard unchanged with heartbeat: `1h`

|
|File descriptors, open|
Number of file descriptors that a zookeeper server has open.
|Dependent item|zookeeper.open_file_descriptor_count
**Preprocessing**

- 
JSON Path: `$.open_file_descriptor_count`

|
|Outstanding requests|
The number of queued requests when the server is under load and is receiving more sustained requests than it can process.
|Dependent item|zookeeper.outstanding_requests
**Preprocessing**

- 
JSON Path: `$.outstanding_requests`

|
|Commit per sec|
The number of commits performed per second
|Dependent item|zookeeper.commit_count.rate
**Preprocessing**

- 
JSON Path: `$.commit_count`

- Change per second
|
|Diff syncs per sec|
Number of diff syncs performed per second
|Dependent item|zookeeper.diff_count.rate
**Preprocessing**

- 
JSON Path: `$.diff_count`

- Change per second
|
|Snap syncs per sec|
Number of snap syncs performed per second
|Dependent item|zookeeper.snap_count.rate
**Preprocessing**

- 
JSON Path: `$.snap_count`

- Change per second
|
|Looking per sec|
Rate of transitions into looking state.
|Dependent item|zookeeper.looking_count.rate
**Preprocessing**

- 
JSON Path: `$.looking_count`

- Change per second
|
|Alive connections|
Number of active clients connected to a zookeeper server.
|Dependent item|zookeeper.num_alive_connections
**Preprocessing**

- 
JSON Path: `$.num_alive_connections`

|
|Global sessions|
Number of global sessions.
|Dependent item|zookeeper.global_sessions
**Preprocessing**

- 
JSON Path: `$.global_sessions`

|
|Local sessions|
Number of local sessions.
|Dependent item|zookeeper.local_sessions
**Preprocessing**

- 
JSON Path: `$.local_sessions`

|
|Drop connections per sec|
Rate of connection drops.
|Dependent item|zookeeper.connection_drop_count.rate
**Preprocessing**

- 
JSON Path: `$.connection_drop_count`

- Change per second
|
|Rejected connections per sec|
Rate of connection rejected.
|Dependent item|zookeeper.connection_rejected.rate
**Preprocessing**

- 
JSON Path: `$.connection_rejected`

- Change per second
|
|Revalidate connections per sec|
Rate of connection revalidations.
|Dependent item|zookeeper.connection_revalidate_count.rate
**Preprocessing**

- 
JSON Path: `$.connection_revalidate_count`

- Change per second
|
|Revalidate per sec|
Rate of revalidations.
|Dependent item|zookeeper.revalidate_count.rate
**Preprocessing**

- 
JSON Path: `$.revalidate_count`

- Change per second
|
|Latency, max|
The maximum amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.max_latency
**Preprocessing**

- 
JSON Path: `$.max_latency`

|
|Latency, min|
The minimum amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.min_latency
**Preprocessing**

- 
JSON Path: `$.min_latency`

|
|Latency, avg|
The average amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.avg_latency
**Preprocessing**

- 
JSON Path: `$.avg_latency`

|
|Znode count|
The number of znodes in the ZooKeeper namespace (the data)
|Dependent item|zookeeper.znode_count
**Preprocessing**

- 
JSON Path: `$.znode_count`

- 
Discard unchanged with heartbeat: `1h`

|
|Ephemeral nodes count|
Number of ephemeral nodes that a zookeeper server has in its data tree.
|Dependent item|zookeeper.ephemerals_count
**Preprocessing**

- 
JSON Path: `$.ephemerals_count`

|
|Watch count|
Number of watches currently set on the local ZooKeeper process.
|Dependent item|zookeeper.watch_count
**Preprocessing**

- 
JSON Path: `$.watch_count`

|
|Packets sent per sec|
The number of zookeeper packets sent from a server per second.
|Dependent item|zookeeper.packets_sent
**Preprocessing**

- 
JSON Path: `$.packets_sent`

- Change per second
|
|Packets received per sec|
The number of zookeeper packets received by a server per second.
|Dependent item|zookeeper.packets_received.rate
**Preprocessing**

- 
JSON Path: `$.packets_received`

- Change per second
|
|Bytes received per sec|
Number of bytes received per second.
|Dependent item|zookeeper.bytes_received_count.rate
**Preprocessing**

- 
JSON Path: `$.bytes_received_count`

- Change per second
|
|Election time, avg|
Time between entering and leaving election.
|Dependent item|zookeeper.avg_election_time
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|
|Elections|
Number of elections happened.
|Dependent item|zookeeper.cnt_election_time
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|
|Fsync time, avg|
Time to fsync transaction log.
|Dependent item|zookeeper.avg_fsynctime
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|
|Fsync|
Count of performed fsyncs.
|Dependent item|zookeeper.cnt_fsynctime
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|
|Snapshot write time, avg|
Average time to write a snapshot.
|Dependent item|zookeeper.avg_snapshottime
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|
|Snapshot writes|
Count of performed snapshot writes.
|Dependent item|zookeeper.cnt_snapshottime
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|

### Triggers

|Name|Description|Expression|Severity|Dependencies and additional info|
|----|-----------|----------|--------|--------------------------------|
|Zookeeper: Server mode has changed|
Zookeeper node state has changed. Acknowledge to close the problem manually.
|`last(/Zookeeper by HTTP/zookeeper.server_state,#1)<>last(/Zookeeper by HTTP/zookeeper.server_state,#2) and length(last(/Zookeeper by HTTP/zookeeper.server_state))>0`|Info|**Manual close**: Yes|
|Zookeeper: Failed to fetch info data|
Zabbix has not received data for items for the last 10 minutes
|`nodata(/Zookeeper by HTTP/zookeeper.uptime,10m)=1`|Warning|**Manual close**: Yes|
|Zookeeper: Version has changed|
Zookeeper version has changed. Acknowledge to close the problem manually.
|`last(/Zookeeper by HTTP/zookeeper.version,#1)<>last(/Zookeeper by HTTP/zookeeper.version,#2) and length(last(/Zookeeper by HTTP/zookeeper.version))>0`|Info|**Manual close**: Yes|
|Zookeeper: Too many file descriptors used|
Number of file descriptors used more than {$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN}% of the available number of file descriptors.
|`min(/Zookeeper by HTTP/zookeeper.open_file_descriptor_count,5m) * 100 / last(/Zookeeper by HTTP/zookeeper.max_file_descriptor_count) > {$ZOOKEEPER.FILE_DESCRIPTORS.MAX.WARN}`|Warning||
|Zookeeper: Too many queued requests|
Number of queued requests in the server. This goes up when the server receives more requests than it can process.
|`min(/Zookeeper by HTTP/zookeeper.outstanding_requests,5m)>{$ZOOKEEPER.OUTSTANDING_REQ.MAX.WARN}`|Average|**Manual close**: Yes|

### LLD rule Leader metrics discovery

|Name|Description|Type|Key and additional info|
|----|-----------|----|-----------------------|
|Leader metrics discovery|
Additional metrics for leader node
|Dependent item|zookeeper.metrics.leader
**Preprocessing**

- 
JSON Path: `$.server_state`

- 
JavaScript: `The text is too long. Please see the template.`

|

### Item prototypes for Leader metrics discovery

|Name|Description|Type|Key and additional info|
|----|-----------|----|-----------------------|
|Pending syncs{#SINGLETON}|
Number of pending syncs to carry out to ZooKeeper ensemble followers.
|Dependent item|zookeeper.pending_syncs[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.pending_syncs`

|
|Quorum size{#SINGLETON}||Dependent item|zookeeper.quorum_size[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.quorum_size`

|
|Synced followers{#SINGLETON}|
Number of synced followers reported when a node server_state is leader.
|Dependent item|zookeeper.synced_followers[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.synced_followers`

|
|Synced non-voting follower{#SINGLETON}|
Number of synced voting followers reported when a node server_state is leader.
|Dependent item|zookeeper.synced_non_voting_followers[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.synced_non_voting_followers`

|
|Synced observers{#SINGLETON}|
Number of synced observers.
|Dependent item|zookeeper.synced_observers[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.synced_observers`

|
|Learners{#SINGLETON}|
Number of learners.
|Dependent item|zookeeper.learners[{#SINGLETON}]
**Preprocessing**

- 
JSON Path: `$.learners`

|

### Trigger prototypes for Leader metrics discovery

|Name|Description|Expression|Severity|Dependencies and additional info|
|----|-----------|----------|--------|--------------------------------|
|Zookeeper: Too many pending syncs||`min(/Zookeeper by HTTP/zookeeper.pending_syncs[{#SINGLETON}],5m)>{$ZOOKEEPER.PENDING_SYNCS.MAX.WARN}`|Average|**Manual close**: Yes|
|Zookeeper: Too few active followers|
The number of followers should equal the total size of your ZooKeeper ensemble, minus 1 (the leader is not included in the follower count). If the ensemble fails to maintain quorum, all automatic failover features are suspended.
|`last(/Zookeeper by HTTP/zookeeper.synced_followers[{#SINGLETON}]) < last(/Zookeeper by HTTP/zookeeper.quorum_size[{#SINGLETON}])-1`|Average||

### LLD rule Clients discovery

|Name|Description|Type|Key and additional info|
|----|-----------|----|-----------------------|
|Clients discovery|
Get list of client connections.

Note, depending on the number of client connections this operation may be expensive (i.e. impact server performance).
|HTTP agent|zookeeper.clients
**Preprocessing**

- 
JavaScript: `The text is too long. Please see the template.`

|

### Item prototypes for Clients discovery

|Name|Description|Type|Key and additional info|
|----|-----------|----|-----------------------|
|Zookeeper client {#TYPE} [{#CLIENT}]: Get client info|
The item gets information about "{#CLIENT}" client of "{#TYPE}" type.
|Dependent item|zookeeper.client_info[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `The text is too long. Please see the template.`

⛔️Custom on fail: Discard value

|
|Zookeeper client {#TYPE} [{#CLIENT}]: Latency, max|
The maximum amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.max_latency[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.max_latency`

|
|Zookeeper client {#TYPE} [{#CLIENT}]: Latency, min|
The minimum amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.min_latency[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.min_latency`

|
|Zookeeper client {#TYPE} [{#CLIENT}]: Latency, avg|
The average amount of time it takes for the server to respond to a client request.
|Dependent item|zookeeper.avg_latency[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.avg_latency`

|
|Zookeeper client {#TYPE} [{#CLIENT}]: Packets sent per sec|
The number of packets sent.
|Dependent item|zookeeper.packets_sent[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.packets_sent`

- Change per second
|
|Zookeeper client {#TYPE} [{#CLIENT}]: Packets received per sec|
The number of packets received.
|Dependent item|zookeeper.packets_received[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.packets_received`

- Change per second
|
|Zookeeper client {#TYPE} [{#CLIENT}]: Outstanding requests|
The number of queued requests when the server is under load and is receiving more sustained requests than it can process.
|Dependent item|zookeeper.outstanding_requests[{#TYPE},{#CLIENT}]
**Preprocessing**

- 
JSON Path: `$.outstanding_requests`

|

## Feedback

Please report any issues with the template at [`https://support.zabbix.com`](https://support.zabbix.com)

You can also provide feedback, discuss the template, or ask for help at [`ZABBIX forums`](https://www.zabbix.com/forum/zabbix-suggestions-and-feedback)

