From a6399850de2aadc401998ca8c42aa568f232f9bb Mon Sep 17 00:00:00 2001 From: Asif Bacchus Date: Mon, 22 Oct 2018 00:46:12 -0600 Subject: [PATCH] Revert "Logwatch service script to process logfile" This reverts commit 2a15f81a2b82410050835b4a506254b389187ee9. Improper commit sequence --- etc/logwatch/readme.md | 290 --------------------------- etc/logwatch/scripts/services/backup | 112 ----------- 2 files changed, 402 deletions(-) delete mode 100644 etc/logwatch/readme.md delete mode 100644 etc/logwatch/scripts/services/backup diff --git a/etc/logwatch/readme.md b/etc/logwatch/readme.md deleted file mode 100644 index a93df5c..0000000 --- a/etc/logwatch/readme.md +++ /dev/null @@ -1,290 +0,0 @@ -# Using Logwatch to monitor backup script - -The backup script's log file has been set up so that utilities like Logwatch can -easily parse it. In order to make that happen, a LogFile Group file, Service -and Script have to be created for Logwatch to generate reports. The correct -(general) directory structure has been created in this git archive already. -Below are the details of each file. - -## Contents - -- [LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)](#logfile-group-file-etclogwatchconflogfilesbackupconf) - - [Log file location](#log-file-location) - - [Archive location and name format](#archive-location-and-name-format) - - [External script for timestamp processing](#external-script-for-timestamp-processing) -- [Service definition file (/etc/logwatch/conf/services/backup.conf)](#service-definition-file-etclogwatchconfservicesbackupconf) - - [LogFile Group file definition](#logfile-group-file-definition) - - [Report title](#report-title) - - [Detail level](#detail-level) -- [Service script (/etc/logwatch/scripts/services/backup)](#service-script-etclogwatchscriptsservicesbackup) - - [Detail levels](#detail-levels) -- [Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere) - - [The time format specification](#the-time-format-specification) - - [The search REGEX](#the-search-regex) -- [Testing](#testing) -- [Final thoughts](#final-thoughts) - -## LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf) - -### Log file location - -Update this as needed to point to the location and name of the log file -generated by the backup script. Remember, by default, the log file is created -in the same directory as the script itself. - -```Ini -LogFile = /path/to/your/backup.log -... -``` - -Best practices suggest you use the *-l* -flag to change this location to something like */var/log/backup.log*, for -example. In that case, the entry would look like: - -```Ini -LogFile = /var/log/backup.log -... -``` - -### Archive location and name format - -If you want Logwatch to process old (archived) log files generated by something -like *Logrotate*, then you have to specify the location and file name format of -those files. I've included the generalized compressed format of such rotated -files as the default in the script. Suppose you store your log files in the -recommended location (*/var/log/*) and are using *Logrotate* with compression -enabled, the archive line would look like: - -```Ini -... -Archive = /var/log/backup.log.?.gz -... -``` - -This would tell Logwatch, when the archive option is set to true, that your -*backup.log* files are archived as: *backup.log.1.gz*, *backup.log.2.gz*, etc. -and are all located in */var/log/*. - -**Note: This line is totally optional and only used if you set the archive -option in Logwatch to true (default). You can comment/delete this line if you -wish.** - -### External script for timestamp processing - -Since the log file uses a non-standard (according to Logwatch) method of -datestamping, a custom filter had to be created. See the -[relevant](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere) -section of this document for more information. - -The script file is called with an *\** before the filename. - -```Ini -... -*sqFullStampAnywhere -... -``` - -If you change the name of this file, you will have to change this line. -Remember that whatever you type here as a name is converted to all-lowercase -so your filename should be all lowercase also. - -## Service definition file (/etc/logwatch/conf/services/backup.conf) - -### LogFile Group file definition - -The service file needs to know what group of log files it is responsible for -processing. This MUST match the name of your *LogFile Group file*: - -```Ini -LogFile = backup -... -``` - -If you change your LogFile Group filename, then update it here too without the -*.conf* extension. - -### Report title - -The Logwatch output file (html or text) is divided into sections. You can -define the title to be anything that has meaning for you. I have arbitrarily -chosen *"System and NextCloud Backup"* but you can change it to anything you want by -modifying the line: - -```Ini -... -Title = "System and NextCloud Backup" -``` - -### Detail level - -If you want to set the *detail* level of this service differently from your -other services (which will use the *--detail* switch value or the value in your -*logwatch.conf*), then you can define that level here. By default, it appears -like this in the service configuration file: - -```Ini -... -# Override the detail level for this service -# Remember the levels are: 0, 1-4, 5, 6+ -# Detail = 0 -``` - -Simply change it to the value you want enforced. For example, here I'm setting -it to output level 5 regardless of whatever settings everything else is using. - -```Ini -# Override the detail level for this service -# Remember the levels are: 0, 1-4, 5, 6+ -Detail = 5 -``` - -## Service script (/etc/logwatch/scripts/services/backup) - -Logwatch calls any script with a name that **matches the service name**. You'll -notice that I just named everything *backup* to keep things simple. You can -change this to whatever you want, however. If you changed the service name to -*"NCbackup*.conf", for example, you would have to rename this script file to -"*NCbackup*" with no extension. Note: The script is a PERL file (note the -shebang) but it can be written in any language. - -In essence, Logwatch just spits out the log file(s) defined in the LogFile Group -file as standard input (STDIN) for the script and then takes whatever is output -(STDOUT) from the script to assemble into it's report. - -### Detail levels - -The script supports four (4) detail levels as follows: - -- **Level 0: Summary output only** - - This will display an aggregate total of certain logged elements. It will - display the total number successful script executions (SQL dump OK, borg - backup OK and borg prune OK), total generated warnings and total errors - encountered that stopped the normal execution of the script. All totals are - relative to the reporting period Logwatch is using (--range parameter). - - **This is the recommended reporting level.** It does not take up much space - and is quick to read. If you notice warnings and/or errors, you should - consult the full logs. -- **Levels 1-4: Critical messages** - - This uses the data which is summarized by Level 0 but outputs the actual - messages in the log file. For example, you will see the actual text of the - errors logged instead of just a total number of errors. This level of - reporting is useful when *initially* monitoring the script's operation since - you can see the actual text of any generated warnings/errors. -- **Level 5: Verbose (debugging) output** - - Like the previous level, this outputs the actual messages found in the log - file. However, it also includes *[INFO] tags* which contain logged - operational messages such as created temporary directories, entering/exiting - maintenance mode, what source files are being used to determine extra file - inclusions/exclusions, etc. This level of reporting is useful in diagnosing - why errors are occurring or if you just want more insight into how the - script works. - - **This level of output will make your Logwatch reports longer and consume - more of your time to review. You should not use this level day-to-day.** -- **Levels 6+: Complete log file dump** - - Any number greater than 5 passed as a detail level will trigger the script - to dump the entire log file out to Logwatch line-by-line. This is useful - only if you are debugging an issue and cannot get access to the actual raw - log file itself. The actual log file is colour-coded which makes it much - easier to read for debugging purposes. - - **Use this detail level only when you need to see the entire log file and - cannot otherwise access the log file.** - -## Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere) - -This is basically a modified version of the '*applyeurodate*' script that comes -with Logwatch. It had to be modified to search within [square brackets] and to -accept characters coming before the stamp (i.e. ANSI colour codes). If you -change the '**stamp**' variable in the backup script to update the timestamp to -your liking (which to totally fine!) then you'll probably have to update this -file. There are two lines you need to modify to suit your new '**stamp**' -variable. - -### The time format specification - -'*$SearchDate*' is the variable used in the PERL script to do exactly what it -says, search for the date stamp. I have it set up to look for the format -'*year-month-date hour:minute:second*'. Note, we don't care about brackets or -anything here, we're just defining the format of the date/time stamp. - -```Perl -... -$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S'); -... -``` - -If you changed the '**stamp**' variable so it was formatted as '*month/day/year -hour:minute*' (ex: '*[09/27/2018 18:38]*') then you'd update the **$SearchDate** -variable as follows (note: no mention of the square brackets!): - -```Perl -... -$SearchDate = TimeFilter('%m/%d/%Y %H:%M'); -... -``` - -### The search REGEX - -The PERL script uses a '*regular expression*' (REGEX) to search within the log file for -'*$SearchDate*'. For the default datestamp, this specification looks like: - -```Perl -... -if ($ThisLine =~ m/\[$SearchDate\] /o) { -... -``` - -The REGEX appears between '*m/*' and '*/o*'. In this case, it searches for -'*$SearchDate*' inside [square brackets] appearing anywhere on the line. This -is because ANSI colour-codes often appear before the datestamp in the default -log file. If you have modified this so that your datestamp appears at the -beginning of the line and in the example format in the section above (using -slashes instead of dashes) then you'd rewrite this REGEX as follows: - -```Perl -... -if ($ThisLine =~ m/^\[$SearchDate\] /o) { -... -``` - -or using regular brackets anywhere on the line: - -```Perl -... -if ($ThisLine =~ m/\($SearchDate\) /o) { -... -``` - -or without any brackets but appearing at the beginning of the line: - -```Perl -... -if ($ThisLine =~ m/^$SearchDate /o) { -... -``` - -## Testing - -Run *logwatch --help* and note the options. You can test just this service -locally on your screen with the following command (assuming you kept default -names for everything): - -```Bash -# Summary output, entire duration of log file -logwatch --service backup --output stdout --format text --range all --detail 0 - -# Minimal detail, yesterday only -logwatch --service backup --output stdout --format text --range yesterday --detail 3 - -# Verbose output, today only -logwatch --service backup --output stdout --format text --range today --detail 5 -``` - -## Final thoughts - -That's it! I'm a horrible PERL programmer so if anyone can optimize/improve the -script file used for Logwatch then please do it! Otherwise, I hope this made -sense and helped you integrate the backup script with Logwatch for easy -monitoring :-) \ No newline at end of file diff --git a/etc/logwatch/scripts/services/backup b/etc/logwatch/scripts/services/backup deleted file mode 100644 index f162ad7..0000000 --- a/etc/logwatch/scripts/services/backup +++ /dev/null @@ -1,112 +0,0 @@ -#!/usr/bin/perl - -############################################################################# -# $Id$ -############################################################################# -# Log: Backup script (backup) -# Revision 1.0 2018/10/16 -# Written by Asif Bacchus -############################################################################# - - -use strict; - -### Get Logwatch detail level (or default to 0) -my $detailLevel = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0; - - -### Declare variables -my $summaryErr; -my $summaryWarn; -my $summarySuccess; - -my %reportHash = (); -my $key; - - -### Minimal detail level: provide summary data only -if ($detailLevel == 0) { - ### process logfile and summarize message types - while (defined(my $ThisLine = )) { - if ($ThisLine =~ /\-- \[ERROR\] /) { - $summaryErr++; - } - elsif ($ThisLine =~ /\-- \[WARNING\] /) { - $summaryWarn++; - } - elsif ($ThisLine =~ /All processes completed successfully/) { - $summarySuccess++; - } - } - - ### fill hash table with headings and summary counts - if ($summarySuccess > 0) { - $reportHash{"All processes successfully completed"} = $summarySuccess; - } - if ($summaryWarn > 0) { - $reportHash{"Warnings issued"} = $summaryWarn; - } - if ($summaryErr > 0) { - $reportHash{"Errors encountered"} = $summaryErr; - } - - ### print hash table - foreach $key (sort keys %reportHash) { - print "$key: $reportHash{$key}\n"; - } -} -### Levels 1-4 provide the actual error, warning and success messages instead -### of a summary count -elsif ($detailLevel >= 1 && $detailLevel <= 4) { - while (defined(my $ThisLine = )) { - if ($ThisLine =~ /\-- \[ERROR\] /) { - print $ThisLine; - } - elsif ($ThisLine =~ /\-- \[WARNING\] /) { - print $ThisLine; - } - elsif ($ThisLine =~ /\-- \[SUCCESS\] /) { - print $ThisLine; - } - } -} -### Level 5 is similiar to levels 1-4 except it also reports informational -### messages such as the location of script created files, variable checks, -### etc. This is useful when verifying the script's operation. -elsif ($detailLevel == 5) { - while (defined(my $ThisLine = )) { - if ($ThisLine =~ /\-- \[ERROR\] /) { - print $ThisLine; - } - elsif ($ThisLine =~ /\-- \[WARNING\] /) { - print $ThisLine; - } - elsif ($ThisLine =~ /\-- \[SUCCESS\] /) { - print $ThisLine; - } - elsif ($ThisLine =~ /\-- \[INFO\] /) { - print $ThisLine; - } - } -} -### Any level above 5 will echo the entire log including the debugging notes -### within the script meant for troubleshooting. Using this level of detail -### should only be done if you cannot view the actual log file directly for -### whatever reason. The actual log file is colour-coded for easier debugging. -elsif ($detailLevel > 5) { - while (defined(my $ThisLine = )) { - print $ThisLine; - } -} - - - -### Exit gracefully -exit (0); - -# vi: shiftwidth=3 tabstop=3 et -# Local Variables: -# mode: perl -# perl-indent-level: 3 -# indent-tabs-mode: nil -# End: \ No newline at end of file