edit

Plugins

Getting Started

The amonagent is a single golang binary. There is nothing to install and all dependencies are already bundled in. To enable a plugin, the only thing you have to do is edit the configuration file in /etc/opt/amonagent/plugins-enabled/{plugin}.conf and restart the agent

  • To list all available plugins run /opt/amonagent/amonagent -list-plugins
  • To test a plugin run /opt/amonagent/amonagent -test-plugin={name}
  • To display all available configuration option for a plugin run /opt/amonagent/amonagent -plugin-config={name}

StatsD

The Amon agent supports the StatsD protocol, enabling you to easily view custom metrics within Amon.

This eliminates the need to rollout an additional monitoring stack specifically for StatsD (statsd daemon, graphite, alerting, etc) as you can view those metrics on Amon dashboards and create triggers for alerting inside Amon.

Installation

The StatsD plugin is preinstalled. To enable it:

$ echo '{"address": ":8125", "delete_timings": true}' > /etc/opt/amonagent/plugins-enabled/statsd.conf
$ sudo service amonagent restart (or) sudo systemctl restart amonagent

That is it. You can start sending StatsD metrics from your language/platform of choice and you will see them in Amon.

Supported Metrics

Counters

Counters are used to measure the frequency of an event per minute, like page views or failed login attempts. You may specify the amount to increment the counter.

import statsd
sc = statsd.StatsClient(port=8125)

sc.incr('amon.counter')
sc.incr('amon.requests')

Timers

Timers are used to measure the duration of a task, like database calls or render times.

import statsd
sc = statsd.StatsClient(port=8125)

sc.timing('response_timer', 100)
sc.timing('unique_visitors', 300)

Health Checks

Amon checks allow you to monitor services or measure resources.

Checks are essentially commands (or scripts) that output data to STDOUT or STDERR and produce an exit status code to indicate a state. Common exit status codes used are 0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 or greater to indicate UNKNOWN or CUSTOM.

Amon checks use the same specification as Nagios and Sensu, therefore, Nagios and Sensu check plugins may be used with Amon.

Nagios Checks

Amon is fully compatible with any plugin from the Nagios Exchange or the nagios-plugins/monitoring-plugins already available for install in your distro. To get started on Ubuntu for example, you can do the following:

# Debian based distros
apt-get install nagios-plugins

# On Ubuntu 14.04+
apt-get install monitoring plugins

# All the plugins are installed in /usr/lib/nagios/plugins
/usr/lib/nagios/plugins/check_apt

#APT WARNING: 25 packages available for upgrade (0 critical updates). 
# |available_upgrades=25;;;0 critical_updates=0;;;0

# To add this check, in /etc/opt/amonagent/plugins-enabled/checks.conf
["/usr/lib/nagios/plugins/check_apt"]

service amonagent restart

# To test the output
amonagent -test-plugin=checks

That is it. You can see the results of your check(s) in Amon

Optionally you can setup alerts on any check and get notified on Slack/Hipchat/Email, etc.

Sensu Checks

Amon is fully compatible with Sensu checks. You can use any check from https://github.com/sensu-plugins


As an example we are going to use the https://github.com/sensu-plugins/sensu-plugins-filesystem-checks plugin. If you are not familiar with Sensu, the first step would be installing Ruby

# Debian based distros
apt-get install ruby ruby-dev

# Red Hat based distros
yum install ruby ruby-devel

# Install the Sensu plugin
gem install sensu-plugins-filesystem-checks

# All the plugins are installed in /usr/local/bin
check-dir-size.rb  -d /var/lib/mongodb/

# CheckDirSize CRITICAL: Directory /var/lib/mongodb/ is greater than 4,000,000 bytes [actual size: 3,389,005,918 bytes]

# To add this check, in /etc/opt/amonagent/plugins-enabled/checks.conf
["check-dir-size.rb  -d /var/lib/mongodb/"]

# To test the output
amonagent -test-plugin=checks

That is it. You can see the results of your Sensu check(s) in Amon

Optionally you can setup alerts on any check and get notified on Slack/Hipchat/Email, etc.

Custom check plugin

The following is an example Amon check plugin. This check uses the running process list to determine if the MySQL process is running. The check is written in Ruby, but you can write checks in Python and Bash.

#!/usr/bin/env ruby

# get the current list of processes
processes = `ps aux`

# determine if the chef-client process is running
running = processes.lines.detect do |process|
  process.include?('mysql')
end

# return appropriate check output and exit status code
if running
  puts 'OK - Mysql process is running'
  exit 0
else
  puts 'WARNING - Mysql process is NOT running'
  exit 1
end 

To enable your custom check

# In /etc/opt/amonagent/plugins-enabled/checks.conf
["ruby|python|bash|perl mysql_running.rb"]

# To test the output
amonagent -test-plugin=checks

Apache

Monitors Apache, reporting:

  • Idle, Busy workers
  • Requests per second
  • Total accesses/bytes

Installation

  1. Enable mod_status in your apache configuration. You can see a tutorial http://www.cyberciti.biz/faq/apache-server-status Replace with your apache status url and paste in the terminal
  2. Update your config file in /etc/opt/amonagent/plugins-enabled/apache.conf
$ echo '{"status_url": "http://localhost/server-status?auto"}' >
     /etc/opt/amonagent/plugins-enabled/apache.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=apache

Apache Plugin - Config file

Below you can see a complete configuration file (located at /etc/opt/amonagent/plugins-enabled/apache.conf) with all the options:

{
    "status_url": "http://localhost/server-status?auto",
}

Nginx

Monitors Nginx, sends to Amon the the following metrics:

  • Connections - reading, writing, waiting
  • Requests per second

Installation

  1. Enable the HttpStubStatusModule - http://wiki.nginx.org/HttpStubStatusModule in your Nginx configuration files
  2. Finally update your config file in /etc/opt/amonagent/plugins-enabled/nginx.conf
$ echo '{"status_url": "http://localhost/nginx_status"}' >
     /etc/opt/amonagent/plugins-enabled/nginx.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=nginx

Nginx Plugin - Config file

Below you can see a complete configuration file (located at /etc/opt/amonagent/plugins-enabled/nginx.conf) with all the options:

{"status_url": "http://localhost/nginx_status"}

MySQL

Monitors MySQL, reporting: parses the output from the SHOW STATUS command

Installation

  1. First, you have to create an Amon user with replication and select rights:
    $ sudo mysql -e "create user 'amon'@'localhost' identified by 'desired-password';"
    $ sudo mysql -e "grant SELECT, REPLICATION CLIENT on *.* to 'amon'@'localhost';"
    
  2. Enable slow queries logging in my.cnf
    # my.cnf within [mysqld] / [mysqld_safe]
    slow_query_log = 1
    long_query_time = 3
    log_output = TABLE
    general-log
    expire_logs_days = 1
    
  3. Update your config file in /etc/opt/amonagent/plugins-enabled/mysql.conf
    $ echo '{"host": "username:password@protocol(address)/dbname"}' >
         /etc/opt/amonagent/plugins-enabled/mysql.conf
    
    
    $ sudo service amonagent restart (or) sudo systemctl restart amonagent
    
    # Optionally you can test the data collected by the plugin with:
    $ /opt/amonagent/amonagent -test-plugin=mysql
    

MySQL Plugin - Config file

Below you can see a complete configuration file (located at /etc/opt/amonagent/plugins-enabled/mysql.conf) with all the options:

{"host": "username:password@protocol(address)/dbname"}

PostgreSQL

Monitors PostgreSQL, sends to Amon the the following metrics:

  • Database size
  • Table and Index sizes
  • Slow Queries
  • Missing Indexes
  • Index Hit Rate
  • pg_stats_database Metrics

Installation

  1. Create a read-only amon user with proper access to your PostgreSQL Server
    $ psql
    $ create user amon with password 'your-desired-password';
    $ grant SELECT ON pg_stat_database to amon;
    
  2. Install **postgresql-contrib** and enable the **pg_stat_statements** extension
    $ sudo yum/apt-get install postgresql-contrib
    $ sudo -u postgres psql
    $ CREATE EXTENSION pg_stat_statements;
    
  3. Finally update your config file in /etc/amonagent/plugins-enabled/postgres.conf
    $ echo '{"host": "postgres://user:password@localhost:port/dbname"}' >
         /etc/opt/amonagent/plugins-enabled/postgresql.conf
    
    
    $ sudo service amonagent restart (or) sudo systemctl restart amonagent
    
    # Optionally you can test the data collected by the plugin with:
    $ /opt/amonagent/amonagent -test-plugin=postgresql
    

PostgreSQL Plugin - Config file

Below you can see a complete configuration file (located at /etc/opt/amonagent/plugins-enabled/postgres.conf) with all the options:

{"host": "postgres://user:password@localhost:port/dbname"}

MongoDB

Sends to Amon the the following metrics:

Installation


$ echo '{"uri": "mongodb://username:password@host:port/database"}' >
    /etc/opt/amonagent/plugins-enabled/mongodb.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=mongodb

Configuration

If you have enabled authentication for your Mongo database, you have to create a read only admin user for Amon:

$ mongo
$ use admin
$ db.auth("admin", "admin-password")
$ db.addUser("amon", "your-desired-password", true)

MongoDB Plugin - Config file

Below you can see a complete configuration file (located at /etc/opt/amonagent/plugins-enabled/mongodb.conf) with all the options:

{"uri": "mongodb://username:password@host:port/database"}

Redis

Monitors Redis parsing the output from the INFO command - http://redis.io/commands/INFO.

Installation

Replace with your Redis connection parameters and paste in the terminal

$ echo '{"host": "tcp://localhost:6379"}' > /etc/opt/amonagent/plugins-enabled/redis.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=redis

Sensu

Amon is fully compatible with all Sensu Plugins (https://github.com/sensu-plugins)

Installation

  1. First install your desired Sensu Plugin. I am going to use the sensu-plugins-io-checks as an example. (https://github.com/sensu-plugins/sensu-plugins-io-checks)
apt-get install ruby ruby-dev
gem install sensu-plugins-io-checks

# Test the plugin
metrics-iostat-extended.rb
  1. Update your config file in /etc/opt/amonagent/plugins-enabled/sensu.conf
$ echo '["metrics-iostat-extended.rb"]' > /etc/opt/amonagent/plugins-enabled/sensu.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=sensu

Sensu Plugin - Config file

The configuration file is just a list with all the sensu plugins you want to execute(it is located at /etc/opt/amonagent/plugins-enabled/sensu.conf) with all the options:


[
    "metrics-iostat-extended.rb",
    "metrics-redis-graphite.rb"
]

Telegraf

Amon is fully compatible with all Telegraf Inputs (https://github.com/influxdata/telegraf)

Installation

  1. First install and configure the Telegraf agent and. Test if everything works as expected with telegraf -test -config /path/to/your/telegraf.conf
  2. Update your config file in /etc/opt/amonagent/plugins-enabled/telegraf.conf
$ echo '{"config": "/path/to/your/telegraf.conf"}' > /etc/opt/amonagent/plugins-enabled/telegraf.conf

$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by the plugin with:
$ /opt/amonagent/amonagent -test-plugin=telegraf

Telegraf Plugin - Config file

The configuration file is just the path to your Telegraf config (it is located at /etc/opt/amonagent/plugins-enabled/telegraf.conf)

{"config": "/etc/telegraf/telegraf.conf"}

You can see your telegraf plugins on the Servers screen in Amon

Custom Plugins

ou can create a custom plugin in any language with several lines of code. The simplest plugin for Amon can be 1 line of code. I am going to use Python as an example:

# your-metrics.py
print requests.per_second:10|gauge

Add the command that can execute this file in /etc/opt/amonagent/plugins-enabled/custom.conf

[{ "command":"python /path/to/your-metrics.py", "name":"requests"},]

That is it. The agent will execute the command, collect the output from STDOUT, parse it and send the result to Amon. You will see this plugin as custom.{name}

# Restart the agent
$ sudo service amonagent restart (or) sudo systemctl restart amonagent

# Optionally you can test the data collected by your plugin with:
$ /opt/amonagent/amonagent -test-plugin=custom

The agent recognizes the following syntax. Every value has to be on a new line:

# Custom Plugin format
chart.line:value|type

# If you want to group multiple values on a single chart:
print "connections.active:10|gauge"
print "connections.waiting:3|gauge"
print "connections.max:40|gauge"

# The agent currently supports two types: gauge and counter
print "connections.active:10|gauge"
print "logged_in_users:50|counter"

# In Python - use print
print "connections.active:10|gauge"

# In Ruby - use puts
puts "connections.active:10|gauge"

# In Bash - use echo or printf
echo "connections.active:10|gauge\n"
printf "connections.active:10|gauge\n"

The config file is just a list of commands. You can put as many as you need. All commands are executed in parallel.

[{
    "command":"python custom_metrics.py",
    "name":"requests"
},
{
    "command":"python connections.py",
    "name":"connections"
},
{
    "command":"bash response_time.sh",
    "name":"ping.amoncx"
}]

Examples

A fully functional ping plugin will generate the following chart in Amon

curl -s -w 'ping.amoncx.lookup_time:%{time_namelookup}|gauge
ping.amoncx.connect_time:%{time_connect}|gauge
ping.amoncx.total:%{time_total}|gauge\n' -o /dev/null https://www.amon.cx