Bell
Bell is a realtime anomalies detection system, designed only for periodic time series, built to be able to monitor thousands of metrics. It collects metrics from statsd, analyzes them with the 3-sigma rule , and visualizes analyzation results on the web. Once enough anomalies were found in a short time, it alerts you via alerters like hipchat etc.
Latest version: v0.2.8
Requirements
- node.js 0.11.x (bell is written in nodejs)
- ssdb 1.6.8.8+ (datastore)
- beanstalkd (job queue between listener and analyzers)
- statsd (metrics source)
Installation
npm install node-bell -g
then add node-bell to statsd's backends in statsd's config.js:
{
, backends: ["node-bell"]
}
Quick Start
- Start ssdb & beanstalkd & statsd
-
Generate sample configuration and edit it, default config/configs.toml:
$ bell -s $ mv sample.configs.toml configs.toml $ vi configs.toml
-
Start listener & analyzers (optional: webapp).
bell analyzer -c configs.toml bell listener -c configs.toml # default port: 8889 bell webapp -c configs.toml # default port: 8989 bell alerter -c configs.toml # default port: 8789
Services
- listener: receives incoming metrics from statsd, then put them to job queue.
- analyzer(s): get job from job queue, and then analyze if current metric an anomaly or not.
- webapp: visualizes analyzation result on web.
- alerter: alerts once enough anomalies were detected.
Write my own alerters
Bell comes with a built-in alerter: hipchat.js, but you can completely write one on your own, here are brief steps:
-
An alerter is a nodejs module which should export a function
init:init(configs, alerter, log)
-
Complete the module and add it to
alerter.modulesin configs.toml:[alerter] modules = ["my_module"]
Restart service alerter.
And events currently available for module alerter (also the second parameter in the init function):
-
Event 'anomaly detected'
Parameters:
datapoint, an array like[metric_name, [timestamp, metric_value, analyzation_result]]Emitted when an anomaly was detected.
Look Inside
Algorithm (How do you actually detect anomalies?)
See 3-sigma or called 68-95-99.7 rule, reference
Storage
Analyzers store metrics in ssdb, using zset, here is storage format for a single time series:
key | score
-----------------------------------------------
timestamp | value:anomalous multiples:timestamp
Data Flow
[statsd]
|
v send to queue
[listener] -----------------> [beanstalkd]
|
| reserve
history metrics v anomalies detected
---------------> [analyzers] ------------------
| | |
| | put to ssdb |
| v V
------------------- [ssdb] [alerter]
|
v
[webapp]
Questions?
See faq.md or open an issue.
License
MIT. Copyright (c) 2014 Eleme, Inc.
