rrdtool RD-BEGINERS(1)
NAME
rrd-beginners - RDtool Beginners' Guide
SYNOPSIS
Helping new RDtool users to understand the basics of
RDtool
DESCRIPTION
This manual is an attempt to assist beginners in
understanding the concepts of RDtool. It sheds a light on
differences between RDtool and other databases. With help
of an example, it explains the structure of RDtool
database. This is followed by an overview of the "graph"
feature of RDtool. At the end, it has sample scripts that
illustrate the usage/wrapping of RDtool within Shell or
Perl scripts.
What makes RDtool so special?
RDtool is GNU licensed software developed by Tobias
Oetiker, a system manager at the Swiss Federal Institute of
Technology. Though it is a database, there are distinct
differences between RDtool databases and other databases as
listed below:
]o RDtool stores data; that makes it a back-end tool. The
RDtool command set allows the creation of graphs; that
makes it a front-end tool as well. Other databases just
store data and can not create graphs.
]o In case of linear databases, new data gets appended at
the bottom of the database table. Thus its size keeps on
increasing, whereas the size of an RDtool database is
determined at creation time. Imagine an RDtool database
as the perimeter of a circle. Data is added along the
perimeter. When new data reaches the starting point, it
overwrites existing data. This way, the size of an
RDtool database always remains constant. The name
"Round Robin" stems from this behavior.
]o Other databases store the values as supplied. RDtool
can be configured to calculate the rate of change from
the previous to the current value and store this
information instead.
]o Other databases get updated when values are supplied.
The RDtool database is structured in such a way that it
needs data at predefined time intervals. If it does not
get a new value during the interval, it stores an
UNKNOWN value for that interval. So, when using the
RDtool database, it is imperative to use scripts that
run at regular intervals to ensure a constant data flow
1.3.5 Last change: 2008-03-15 1
rrdtool RD-BEGINERS(1)
to update the RDtool database.
RDtool is designed to store time series of data. With every
data update, an associated time stamp is stored. Time is
always expressed in seconds passed since epoch (01-01-1970).
RDtool can be installed on Unix as well as Windows. It
comes with a command set to carry out various operations on
RD databases. This command set can be accessed from the
command line, as well as from Shell or Perl scripts. The
scripts act as wrappers for accessing data stored in RDtool
databases.
Understanding by an example
The structure of an RD database is different than other
linear databases. Other databases define tables with
columns, and many other parameters. These definitions
sometimes are very complex, especially in large databases.
RDtool databases are primarily used for monitoring purposes
and hence are very simple in structure. The parameters that
need to be defined are variables that hold values and
archives of those values. Being time sensitive, a couple of
time related parameters are also defined. Because of its
structure, the definition of an RDtool database also
includes a provision to specify specific actions to take in
the absence of update values. Data Source (DS), heartbeat,
Date Source Type (DST), Round Robin Archive (RA), and
Consolidation Function (CF) are some of the terminologies
related to RDtool databases.
The structure of a database and the terminology associated
with it can be best explained with an example.
rrdtool create target.rrd \
--start 1023654125 \
--step 300 \
DS:mem:GAUGE:600:0:671744 \
RA:AVERAGE:0.5:12:24 \
RA:AVERAGE:0.5:288:31
This example creates a database named target.rrd. Start time
(1'023'654'125) is specified in total number of seconds
since epoch (time in seconds since 01-01-1970). While
updating the database, the update time is also specified.
This update time MUST be large (later) then start time and
MUST be in seconds since epoch.
The step of 300 seconds indicates that database expects new
values every 300 seconds. The wrapper script should be
scheduled to run every step seconds so that it updates the
database every step seconds.
1.3.5 Last change: 2008-03-15 2
rrdtool RD-BEGINERS(1)
DS (Data Source) is the actual variable which relates to the
parameter on the device that is monitored. Its syntax is
DS:variablename:DST:heartbeat:min:max
DS is a key word. "variablename" is a name under which the
parameter is saved in the database. There can be as many DSs
in a database as needed. After every step interval, a new
value of DS is supplied to update the database. This value
is also called Primary Data Point (PDP). In our example
mentioned above, a new PDP is generated every 300 seconds.
Note, that if you do NOT supply new datapoints exactly every
300 seconds, this is not a problem, RDtool will interpolate
the data accordingly.
DST (Data Source Type) defines the type of the DS. It can be
COUNTER, DERIVE, ABSOLUTE, GAUGE. A DS declared as COUNTER
will save the rate of change of the value over a step
period. This assumes that the value is always increasing
(the difference between the current and the previous value
is greater than 0). Traffic counters on a router are an
ideal candidate for using COUNTER as DST. DERIVE is the same
as COUNTER, but it allows negative values as well. If you
want to see the rate of change in free diskspace on your
server, then you might want to use the DERIVE data type.
ABSOLUTE also saves the rate of change, but it assumes that
the previous value is set to 0. The difference between the
current and the previous value is always equal to the
current value. Thus it just stores the current value divided
by the step interval (300 seconds in our example). GAUGE
does not save the rate of change. It saves the actual value
itself. There are no divisions or calculations. Memory
consumption in a server is a typical example of gauge. The
difference between the different types DSTs can be explained
better with the following example:
Values = 300, 600, 900, 1200
Step = 300 seconds
COUNTER DS = 1, 1, 1, 1
DERIVE DS = 1, 1, 1, 1
ABSOLUTE DS = 1, 2, 3, 4
GAUGE DS = 300, 600, 900, 1200
The next parameter is heartbeat. In our example, heartbeat
is 600 seconds. If the database does not get a new PDP
within 300 seconds, it will wait for another 300 seconds
(total 600 seconds). If it doesn't receive any PDP within
600 seconds, it will save an UNKNOWN value into the
database. This UNKNOWN value is a special feature of RDtool
- it is much better than to assume a missing value was 0
(zero) or any other number which might also be a valid data
1.3.5 Last change: 2008-03-15 3
rrdtool RD-BEGINERS(1)
value. For example, the traffic flow counter on a router
keeps increasing. Lets say, a value is missed for an
interval and 0 is stored instead of UNKNOWN. Now when the
next value becomes available, it will calculate the
difference between the current value and the previous value
(0) which is not correct. So, inserting the value UNKNOWN
makes much more sense here.
The next two parameters are the minimum and maximum value,
respectively. If the variable to be stored has predictable
maximum and minimum values, this should be specified here.
Any update value falling out of this range will be stored as
UNKNOWN.
The next line declares a round robin archive (RA). The
syntax for declaring an RA is
RA:CF:xff:step:rows
RA is the keyword to declare RAs. The consolidation
function (CF) can be AVERAGE, MINIMUM, MAXIMUM, and LAST.
The concept of the consolidated data point (CDP) comes into
the picture here. A CDP is CFed (averaged, maximum/minimum
value or last value) from step number of PDPs. This RA will
hold rows CDPs.
Lets have a look at the example above. For the first RA, 12
(steps) PDPs (DS variables) are AVERAGEed (CF) to form one
CDP. 24 (rows) of theses CDPs are archived. Each PDP occurs
at 300 seconds. 12 PDPs represent 12 times 300 seconds which
is 1 hour. It means 1 CDP (which is equal to 12 PDPs)
represents data worth 1 hour. 24 such CDPs represent 1 day
(1 hour times 24 CDPs). This means, this RA is an archive
for one day. After 24 CDPs, CDP number 25 will replace the
1st CDP. The second RA saves 31 CDPs; each CPD represents
an AVERAGE value for a day (288 PDPs, each covering 300
seconds = 24 hours). Therefore this RA is an archive for
one month. A single database can have many RAs. If there
are multiple DSs, each individual RA will save data for all
the DSs in the database. For example, if a database has 3
DSs and daily, weekly, monthly, and yearly RAs are
declared, then each RA will hold data from all 3 data
sources.
Graphical Magic
Another important feature of RDtool is its ability to
create graphs. The "graph" command uses the "fetch" command
internally to retrieve values from the database. With the
retrieved values it draws graphs as defined by the
parameters supplied on the command line. A single graph can
show different DS (Data Sources) from a database. It is also
1.3.5 Last change: 2008-03-15 4
rrdtool RD-BEGINERS(1)
possible to show the values from more than one database in a
single graph. Often, it is necessary to perform some math on
the values retrieved from the database before plotting them.
For example, in SNMP replies, memory consumption values are
usually specified in KBytes and traffic flow on interfaces
is specified in Bytes. Graphs for these values will be more
meaningful if values are represented in MBytes and mbps. The
RDtool graph command allows to define such conversions.
Apart from mathematical calculations, it is also possible to
perform logical operations such as greater than, less than,
and if/then/else. If a database contains more than one RA
archive, then a question may arise - how does RDtool decide
which RA archive to use for retrieving the values? RDtool
looks at several things when making its choice. First it
makes sure that the RA covers as much of the graphing time
frame as possible. Second it looks at the resolution of the
RA compared to the resolution of the graph. It tries to
find one which has the same or higher better resolution.
With the "-r" option you can force RDtool to assume a
different resolution than the one calculated from the pixel
width of the graph.
Values of different variables can be presented in 5
different shapes in a graph - AREA, LINE1, LINE2, LINE3, and
STACK. AREA is represented by a solid colored area with
values as the boundary of this area. LINE1/2/3 (increasing
width) are just plain lines representing the values. STACK
is also an area but it is "stack"ed on top AREA or
LINE1/2/3. Another important thing to note is that variables
are plotted in the order they are defined in the graph
command. Therefore care must be taken to define STACK only
after defining AREA/LINE. It is also possible to put
formatted comments within the graph. Detailed instructions
can be found in the graph manual.
Wrapping RDtool within Shell/Perl script
After understanding RDtool it is now a time to actually use
RDtool in scripts. Tasks involved in network management are
data collection, data storage, and data retrieval. In the
following example, the previously created target.rrd
database is used. Data collection and data storage is done
using Shell scripts. Data retrieval and report generation is
done using Perl scripts. These scripts are shown below:
Shell script (collects data, updates database)
1.3.5 Last change: 2008-03-15 5
rrdtool RD-BEGINERS(1)
#!/bin/sh
a=0
while [ "$a" == 0 ]; do
snmpwalk -c public 192.168.1.250 hrSWRunPerfMem > snmpreply
totalmem=`awk BEGIN {totmem=0}
{ if ($NF == "KBytes")
{totmem=totmem]$(NF-1)}
}
END {print totmem} snmpreply`
# I can use N as a replacement for the current time
rrdtool update target.rrd N:$totalmem
# sleep until the next 300 seconds are full
perl -e sleep 300 - time % 300
done # end of while loop
Perl script (retrieves data from database and generates
graphs and statistics)
#!/usr/bin/perl -w
# This script fetches data from target.rrd, creates a graph of memory
# consumption on the target (Dual P3 Processor 1 GHz, 656 MB RAM)
# call the RD perl module
use lib qw( /usr/local/rrdtool-1.0.41/lib/perl ../lib/perl );
use RDs;
my $curtime = time(); # set current time
my $endtime = $curtime - 86400; # set end time to 24 hours ago
my $starttime = $endtime - 2592000; # set start 30 days in the past
1.3.5 Last change: 2008-03-15 6
rrdtool RD-BEGINERS(1)
# fetch average values from the RD database between start and end time
my ($start,$step,$dsnames,$data) =
RDs::fetch("target.rrd", "AVERAGE",
"-r", "600", "-s", "$starttime", "-e", "$endtime");
# save fetched values in a 2-dimensional array
my $rows = 0;
my $columns = 0;
my $timevariable = $start;
foreach $line (@$data) {
$vals[$rows][$columns] = $timevariable;
$timevariable = $timevariable ] $step;
foreach $val (@$line) {
$vals[$rows][]$columns] = $val;}
$rows];
$columns = 0;
}
my $tottime = 0;
my $count = 0;
# save the values from the 2-dimensional into a 1-dimensional array
for $i ( 0 .. $#vals ) {
$totmem[$count] = $vals[$i][1];
$count];
}
my $totmemsum = 0;
# calculate the total of all values
for $i ( 0 .. ($count-1) ) {
$totmemsum = $totmemsum ] $totmem[$i];
}
# calculate the average of the array
my $totmemave = $totmemsum/($count);
# create the graph
RDs::graph ("/images/mem$count.png", \
"--title= Memory Usage", \
"--vertical-label=Memory Consumption (MB)", \
"--start=$starttime", \
"--end=$endtime", \
"--color=BACK#C", \
"--color=CANVAS#CF", \
"--color=SHADEB#9999C", \
"--height=125", \
"--upper-limit=656", \
"--lower-limit=0", \
"--rigid", \
"--base=1024", \
"DEF:totmem=target.rrd:mem:AVERAGE", \
"CDEF:totmemcor=totmem,0,671744,LIMIT,UN,0,totmem,IF,1024,/",\
"CDEF:machinemem=totmem,656,],totmem,-",\
"COMENT:Memory Consumption between $starttime",\
"COMENT: and $endtime ",\
"HRULE:656#000000:Maximum Available Memory - 656 MB",\
"AREA:machinemem#CF:Memory Unused", \
"AREA:totmemcor#6699C:Total memory consumed in MB");
1.3.5 Last change: 2008-03-15 7
rrdtool RD-BEGINERS(1)
my $err=RDs::error;
if ($err) {print "problem generating the graph: $err\n";}
# print the output
print "Average memory consumption is ";
printf "%5.2f",$totmemave/1024;
print " MB. Graphical representation can be found at /images/mem$count.png.";
AUTHOR
Ketan Patel
1.3.5 Last change: 2008-03-15 8
|
