Program name: mail2mon.pl Author: Martin Fuerstenau martin.fuerstenau_at_oce.com or Martin.fuerstenau_at_nagiossw.org Date: 10 May 2012 Purpose: ======== - Submitting alert emails to a monitor system based on N*gios (icinga,op5,opsview...) Features of the program: ======================== - Accepts Internet mail (text), MIME and Multipart MIME messages (nested MIME not supported) - Extracts sender, subject and content (text) - Converts HTML in message content to text - Converts base64 coded message content to text - Converts german umlauts to ASCII but there is no general conversion (ISO-8859-1 or UTF8 to 7-Bit Ascii or so) - Has a API to add filter scripts written in Perl or any other language to extract and reformat - host and/or application from subject and/or content - data from subject and/or content - reformat data and deliver the data back to the program - Writes the reformatted data (host, errorcode, content) to the command pipe - Includes a option to store mails in a given directory (because you need some stored mails to develop your filter) - Includes a testmode for testing filters while developing them. History and Changes: ==================== - 10 May 2012 Version 1 - First released version Prerequisites: ============== - Perl and some of its modules - A configured mail daemon (tested with postfix) willing to accept incoming mails. How does it works? ================== The program receives it data via pipe from postfix. (Ingo Lantschner described it in "SMTP-Handling für Nagios HowTo"). There are no temporary data to be stored for analytics purpose. As mentioned in features above text, MIME, Multipart-MIME and base64 encoding is supported. Why this? Normally system messages come as plain text. But it can happen, that a system is configured to send "wonderfull HTML messages" because the administrator is at present alarmed via email. So the only thing to change on the system is the mail address. Nothing else. Or you have some things like a jobscheduler on a AS400 which can only send emails. Or (as in my case) you use for some reasons things like MOM. But MOM can not contact your trouble ticket system. But your Nagios can. So you have to process messages from MOM.These are base64 coded. Or you have a application which can send alert mails but nothing else. After bases analysis of the message we have 3 things: - Sender - Subject - Content (Text ASCII) We have no sending host because a host refined from the mail header is not destinct. Therfore we have to ections in the config file (see later on) where the admin defines a relation mail to host and eventually mail to application. mail2mon uses this informations to build the of the filter file. Example: Mailaddress joe@something.org belongs to host harry then the name of the file is harry.fltr If joe@something.org belongs also to application main-erp-system the name of the filter would be: harry_main-erp-system.fltr The trick is that when the progam starts noch a real great bunch of filters is loaded. The name of the file is determined at runtime. mail2mon will handle over sender, subject and content to the filter script, will receive the output of the script and tranfer it to the eventdb. Some words about a filter script. Normally it is a little perl program which is included via "require" at runtime. A sample is included in the filter directory. For those of us who are not able to program in perl there are two little scripts. A perl programm as a sample which calls a shellscript/program. Via this interface you can use whatever you want as program language (but beware of runtime). Installation & Configuration: ============================= mail2mon comes as a zipped tar file. It includes 3 subdirectories: - conf - bin - filter Place the program it self which is located in the bin directory wherever you want. a good idea for the other ones is a subdirectory where you Nagios configuration is located. for example: /etc/nagios/eventdb/mail2mon/conf and /etc/nagios/eventdb/mail2mon/filter Please place the deliverd configuration files and the sample filter scripts in these directories. Base variables in the program: ------------------------------ $content_logsize = "250"; How much characters from the content will be written to the logfile in case of trouble BEWARE!! If empty the whole content will be written to logfile. my $logfile_default = "/var/log/mail2mon/mail2mon.log"; This is the default for the logfile. If you want to use a different logfile you can either change this variable or pass a different logfile via commandline (switch -l). This can be helpful if you are ready with some filtertests. The filtertests will not place anything into the eventdb. But perhaps you wanna have checked the whole chain i.e. pass a mail via pipe. $m2mon_dir= "/etc/nagios/eventdb/mail2mon/"; Directory where configuration and filter are stored. Change it to your needs. my $m2mon_cfg = $m2mon_dir."conf/mail2mon.cfg"; Central configuration file my $m2mon_cfg_t = $m2mon_dir."conf/mail2mon_t.cfg"; Central configuration file for testmode. These 2 files are necessary for testmode. If you use the same configuration file as for production it may happen that all emails for the newly added host/application will enter the eventdb even if the filter is not fully developed. my $filter_dir = $m2mon_dir."filter"; The name of the directory where the filters resides my $filter_default = "default.fltr"; Set the default filter. If empty no default filter will be used. my $eventdb = "eventdb"; Set your databasename for eventdb here. my $eventdb_host = "localhost"; Set your databasehost for eventdb here my $eventdb_user = "eventdb"; Set your satabaseuser for eventdb here my $eventdb_pass = "is-an-event"; Same for the database password my $eventdb_facility = "mail2mon"; Facility used for eventdb. Not necessary to follow syslog conventions because the database is addressed directly. my $eventdb_type = "mail"; Default type for eventdb. Can be changed to your needs. But if you changed it modify index.php. Configuration files: -------------------- Example file content: [LogMail] # 0 means mail will not be stored (normal processing) # 1 means mail will be stored store_mail = 1 # BEWARE!! The mailstore should always contain the the absolute path. # Otherwise it may be that you have to search for your mailstore (or do you # know where your program is located when it is started from Postfix?) mail_store = /tmp/testmail # Please use ALWAYS lowercase letters for mail addresses even in the case the # address used in your mail to be analyzed uses uppercase letters. Otherwise # you will received a error message [Mail2Host] duck@localhost.something.de = fileserver1 duck@ducktales.something.de = fileserver2 root@ocalhost.something.de = mailserver1 [Mail2Appl] duck@localhost.something.de = main-erp-system You have to add here a host (or maybe a distinct name). Thats mandatory. to add a application is optional. Filter: ------- Nothing has to be configured at this step. See later on under "Writing filters". Postfix: -------- As mentioned I followed Ingo Lantschners "SMTP-Handling für Nagios HowTo" which is unfortunately in german. So the steps here. YOU CAN DO THIS STEP AS THE LAST ONE!! You can test it, play around, develop some filters for testmails you have stored from other locations, mboxes Outlook - whatever. Okay - mainly you have to do 2 modifications to postfix. main.cf: alias_maps = hash:/etc/aliases, hash:/etc/postfix/nagios alias_database = hash:/etc/aliases, hash:/etc/postfix/nagios and /etc/postfix/nagios (if your user for these mails is nagios): ## /etc/postfix/nagios nagios: "|/etc/nagios/eventdb/mail2mon/bin/mail2mon.pl" Replace the paths with the path where your mail2mon.pl is located (I know /etc is not soooo correct for a program :-)) ). If your nagios is not running as root (normally user nagios) mail2mon.pl should have the following rights: -rw-rw-r-- 1 nagios root nagios You have to hash as user root with postalias -o nagios Change the owner of the generated nagios.db to nagios. It now should look like: /etc/postfix$ l -l nagios* -rw-rw-r-- 1 nagios root nagios -rw-r--r-- 1 nagios root nagios.db Now postfix will handle over every mail for user nagios to mail2mon.pl. Logging: ======== As mentioned mail2mon.pl will write logfile. Here we have a little problem. BE SURE THAT LOGFILES CAN BE WRITTEN TO THE GIVEN DIRECTORY!!! Postfix normally executes scripts as user nobody, group nogroup. Therefore the logfile will be owned user nobody and user nogroup. So be sure to set the rights for the logfile directory correctly. Usage ===== Now everything should be configured and we can use it. First you need to write some filters. It is good step for training to use some mails from some mailboxes etc.. It is best to take some different formats. Store the mails in files. Testing is easy: my_mail_file | mail2mon.pl -t You will receive the output on the screen. But first you need a entry in the configfile for the mailadress from the mailfile and a filterfile. Writing filters and testmode ---------------------------- If you wanna write a perl based filter copy sample.fltr to the name of you filter (host.fltr or host_application.fltr depending of the settings in your configuration file for testmode). In the filterfile there are 3 sections: - a head for the data handled over for mail2mon - a bottom for giving back data and a middle section with some dummy code. Here you have to place you code. If you wann write a shell script it is nearly the same. You have to copy sample_with_shellcall.fltr in the same manner as with sample.fltr. In this copied file replace in the line @filter_return=qx(/mybinarypath/sample_shell.sh "$filter_sender" "$filter_subject" "$filter_content" "$filter_host" "$filter_application"); /mybinarypath/sample_shell.sh with the path and the name of your shellscript/program and copy sample.sh (if you use it) to the given name. Read the documentation in the scripts. Do not change the order of the echo commands at the bottom of the script. It is important because the data is given back in a array (you know element 1, element 2, element 3...). To get "real" stuff for testing your filters set "mail_store" to a existing directory (best with right 777) and set "store_mail" to 1. Now every mail will be written as file in the given directory. If you have enough stuff for writing your filter set "store_mail" back 0. Test writing to database ------------------------ If you will test writing to database please copy your configuration to from test configuration file to production one. Ensure that "store_mail" is set to 0. Otherwise your saved mails for testing will be saved again. You test with my_mail_file | mail2mon.pl Using an alternative logfile will be good. Otherwise may be you have error message from the test in you production log. my_mail_file | mail2mon.pl -l testlogfile Some patches for index.php and check_eventdb.pl =============================================== All agents deliverd with eventdb are using the program from the database. Unfortunately it is not used in index.php and check_eventdb.pl. index.php --------- The history of eventdb is syslog-ng. Therefore out-of-the-box for coloured fields in the Type and Priority column are only colours reflecting items from syslog. But due to the fact that like some agents mail2mon does not use syslog to write to the database we use things like "critical" instead of "emerg" or so. Getting colors.Look for td.warning (around line 1800) and insert items for critical and (occasinally) down: td.alert { color: #ffffff; background-color: #ff0000; } td.critical { color: #ffffff; background-color: #ff0000; } td.down { color: #ffffff; background-color: #ff0000; } td.warning { color: #333333; background-color: #ffff00; } td.crit { color: #333333; background-color: #FFA0A0; } Getting field program to work. Insert line ('program' => 'Program',) in the visible fields section. // The visible fieldlist cset('page.fields', Array ( 'acknowledged' => 'rev', 'uid' => 'ID', 'type' => 'Type', 'host' => 'Host', 'facility' => 'Facility', 'priority' => 'Priority', 'datetime' => 'Datetime', 'program' => 'Program', 'message' => 'Message' )); check_eventdb.pl ---------------- First it is a good idea to write the database access data in the script instead of a lot of data in your Nagios command definitions: # default values $opt_db = "eventdb"; $opt_dbtable = "events"; $opt_dbhost = "localhost"; $opt_dbuser = "eventdb"; $opt_dbpasswd = "ereichniss"; $opt_host = ""; $opt_msg = ""; $opt_priority = ""; $opt_label = ""; $opt_type = ""; $opt_facility = ""; $opt_program = ""; Second to access to access the database field "program" you have to add the following. Define variable "$opt_program": use vars qw( $opt_msg $opt_host $opt_priority $opt_help $opt_db $opt_dbtable $opt_dbuser $opt_dbpasswd $opt_dbhost $opt_critical $opt_warning $opt_label $opt_type $opt_facility $opt_program $dbh $sth $result $update $matches @sql @params $PROGNAME $VERSION ); For a good style set it to nothing in "default values section" (see "access data" above) and add it to "GetOptions: GetOptions( "msg=s" => \$opt_msg, "m=s" => \$opt_msg, "l=s" => \$opt_label, "label=s" => \$opt_label, "host=s" => \$opt_host, "H=s" => \$opt_host, "t=s" => \$opt_type, "type=s" => \$opt_type, "f=s" => \$opt_facility, "facility=s" => \$opt_facility, "priority=s" => \$opt_priority, "p=s" => \$opt_priority, "program=s" => \$opt_program, "P=s" => \$opt_program, "help" => \$opt_help, "h" => \$opt_help, "db=s" => \$opt_db, "dbtable=s" => \$opt_dbtable, "dbuser=s" => \$opt_dbuser, "dbpassword=s" => \$opt_dbpasswd, "dbhost=s" => \$opt_dbhost, "critical=i" => \$opt_critical, "c=i" => \$opt_critical, "warning=i" => \$opt_warning, "w=i" => \$opt_warning, ); Now we must ensure that the select statement is complete. So add a "add_part": add_part('facility', $opt_facility) if ($opt_facility ne ''); add_part('program', $opt_program) if ($opt_program ne ''); add_part('acknowledged', '0'); And now we have to modify the help function print_usage: Old line: Usage: $PROGNAME [ -H host ] [ -p priority ] [ -t type ] [ -m msg ] [ --db db ] [ --dbtabe dbtable ] [ --dbuser dbuser ] [ --dbpassword dbpassword ] [ --dbhost dbhost ] [ -l label ] -w warn -c crit New line: Usage: $PROGNAME [ -H host ] [ -p priority ] [ -t type ] [ -m msg ] [ -P program ] [ --db db ] [ --dbtabe dbtable ] [ --dbuser dbuser ] [ --dbpassword dbpassword ] [ --dbhost dbhost ] [ -l label ] -w warn -c crit And later on: -t --type The logtype (e.g. syslog, snmptrap, mail). -P --program Program as logged by agent. --db STRING Database (default: eventdb) That's all. Enjoy and give me a feedback.