asif/MailcowBackup
1
0
Fork 0
Code Issues Releases 10 Wiki Activity

Revert "Logwatch service script to process logfile"

Browse Source 
This reverts commit 2a15f81a2b.

Improper commit sequence
This commit is contained in:
Asif Bacchus 2018-10-22 00:46:12 -06:00
parent 2a15f81a2b
commit a6399850de
2 changed files with 0 additions and 402 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

290
etc/logwatch/readme.md
View File

@ -1,290 +0,0 @@
# Using Logwatch to monitor backup script <!-- omit in toc -->
The backup script's log file has been set up so that utilities like Logwatch can
easily parse it. In order to make that happen, a LogFile Group file, Service
and Script have to be created for Logwatch to generate reports. The correct
(general) directory structure has been created in this git archive already.
Below are the details of each file.
## Contents <!-- omit in toc -->
- [LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)](#logfile-group-file-etclogwatchconflogfilesbackupconf)
- [Log file location](#log-file-location)
- [Archive location and name format](#archive-location-and-name-format)
- [External script for timestamp processing](#external-script-for-timestamp-processing)
- [Service definition file (/etc/logwatch/conf/services/backup.conf)](#service-definition-file-etclogwatchconfservicesbackupconf)
- [LogFile Group file definition](#logfile-group-file-definition)
- [Report title](#report-title)
- [Detail level](#detail-level)
- [Service script (/etc/logwatch/scripts/services/backup)](#service-script-etclogwatchscriptsservicesbackup)
- [Detail levels](#detail-levels)
- [Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
- [The time format specification](#the-time-format-specification)
- [The search REGEX](#the-search-regex)
- [Testing](#testing)
- [Final thoughts](#final-thoughts)
## LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)
### Log file location
Update this as needed to point to the location and name of the log file
generated by the backup script. Remember, by default, the log file is created
in the same directory as the script itself.
```Ini
LogFile = /path/to/your/backup.log
...
```
Best practices suggest you use the *-l*
flag to change this location to something like */var/log/backup.log*, for
example. In that case, the entry would look like:
```Ini
LogFile = /var/log/backup.log
...
```
### Archive location and name format
If you want Logwatch to process old (archived) log files generated by something
like *Logrotate*, then you have to specify the location and file name format of
those files. I've included the generalized compressed format of such rotated
files as the default in the script. Suppose you store your log files in the
recommended location (*/var/log/*) and are using *Logrotate* with compression
enabled, the archive line would look like:
```Ini
...
Archive = /var/log/backup.log.?.gz
...
```
This would tell Logwatch, when the archive option is set to true, that your
*backup.log* files are archived as: *backup.log.1.gz*, *backup.log.2.gz*, etc.
and are all located in */var/log/*.
**Note: This line is totally optional and only used if you set the archive
option in Logwatch to true (default). You can comment/delete this line if you
wish.**
### External script for timestamp processing
Since the log file uses a non-standard (according to Logwatch) method of
datestamping, a custom filter had to be created. See the
[relevant](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
section of this document for more information.
The script file is called with an *\** before the filename.
```Ini
...
*sqFullStampAnywhere
...
```
If you change the name of this file, you will have to change this line.
Remember that whatever you type here as a name is converted to all-lowercase
so your filename should be all lowercase also.
## Service definition file (/etc/logwatch/conf/services/backup.conf)
### LogFile Group file definition
The service file needs to know what group of log files it is responsible for
processing. This MUST match the name of your *LogFile Group file*:
```Ini
LogFile = backup
...
```
If you change your LogFile Group filename, then update it here too without the
*.conf* extension.
### Report title
The Logwatch output file (html or text) is divided into sections. You can
define the title to be anything that has meaning for you. I have arbitrarily
chosen *"System and NextCloud Backup"* but you can change it to anything you want by
modifying the line:
```Ini
...
Title = "System and NextCloud Backup"
```
### Detail level
If you want to set the *detail* level of this service differently from your
other services (which will use the *--detail* switch value or the value in your
*logwatch.conf*), then you can define that level here. By default, it appears
like this in the service configuration file:
```Ini
...
# Override the detail level for this service
# Remember the levels are: 0, 1-4, 5, 6+
# Detail = 0
```
Simply change it to the value you want enforced. For example, here I'm setting
it to output level 5 regardless of whatever settings everything else is using.
```Ini
# Override the detail level for this service
# Remember the levels are: 0, 1-4, 5, 6+
Detail = 5
```
## Service script (/etc/logwatch/scripts/services/backup)
Logwatch calls any script with a name that **matches the service name**. You'll
notice that I just named everything *backup* to keep things simple. You can
change this to whatever you want, however. If you changed the service name to
*"NCbackup*.conf", for example, you would have to rename this script file to
"*NCbackup*" with no extension. Note: The script is a PERL file (note the
shebang) but it can be written in any language.
In essence, Logwatch just spits out the log file(s) defined in the LogFile Group
file as standard input (STDIN) for the script and then takes whatever is output
(STDOUT) from the script to assemble into it's report.
### Detail levels
The script supports four (4) detail levels as follows:
- **Level 0: Summary output only**
- This will display an aggregate total of certain logged elements. It will
display the total number successful script executions (SQL dump OK, borg
backup OK and borg prune OK), total generated warnings and total errors
encountered that stopped the normal execution of the script. All totals are
relative to the reporting period Logwatch is using (--range parameter).
**This is the recommended reporting level.** It does not take up much space
and is quick to read. If you notice warnings and/or errors, you should
consult the full logs.
- **Levels 1-4: Critical messages**
- This uses the data which is summarized by Level 0 but outputs the actual
messages in the log file. For example, you will see the actual text of the
errors logged instead of just a total number of errors. This level of
reporting is useful when *initially* monitoring the script's operation since
you can see the actual text of any generated warnings/errors.
- **Level 5: Verbose (debugging) output**
- Like the previous level, this outputs the actual messages found in the log
file. However, it also includes *[INFO] tags* which contain logged
operational messages such as created temporary directories, entering/exiting
maintenance mode, what source files are being used to determine extra file
inclusions/exclusions, etc. This level of reporting is useful in diagnosing
why errors are occurring or if you just want more insight into how the
script works.
**This level of output will make your Logwatch reports longer and consume
more of your time to review. You should not use this level day-to-day.**
- **Levels 6+: Complete log file dump**
- Any number greater than 5 passed as a detail level will trigger the script
to dump the entire log file out to Logwatch line-by-line. This is useful
only if you are debugging an issue and cannot get access to the actual raw
log file itself. The actual log file is colour-coded which makes it much
easier to read for debugging purposes.
**Use this detail level only when you need to see the entire log file and
cannot otherwise access the log file.**
## Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)
This is basically a modified version of the '*applyeurodate*' script that comes
with Logwatch. It had to be modified to search within [square brackets] and to
accept characters coming before the stamp (i.e. ANSI colour codes). If you
change the '**stamp**' variable in the backup script to update the timestamp to
your liking (which to totally fine!) then you'll probably have to update this
file. There are two lines you need to modify to suit your new '**stamp**'
variable.
### The time format specification
'*$SearchDate*' is the variable used in the PERL script to do exactly what it
says, search for the date stamp. I have it set up to look for the format
'*year-month-date hour:minute:second*'. Note, we don't care about brackets or
anything here, we're just defining the format of the date/time stamp.
```Perl
...
$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S');
...
```
If you changed the '**stamp**' variable so it was formatted as '*month/day/year
hour:minute*' (ex: '*[09/27/2018 18:38]*') then you'd update the **$SearchDate**
variable as follows (note: no mention of the square brackets!):
```Perl
...
$SearchDate = TimeFilter('%m/%d/%Y %H:%M');
...
```
### The search REGEX
The PERL script uses a '*regular expression*' (REGEX) to search within the log file for
'*$SearchDate*'. For the default datestamp, this specification looks like:
```Perl
...
if ($ThisLine =~ m/\[$SearchDate\] /o) {
...
```
The REGEX appears between '*m/*' and '*/o*'. In this case, it searches for
'*$SearchDate*' inside [square brackets] appearing anywhere on the line. This
is because ANSI colour-codes often appear before the datestamp in the default
log file. If you have modified this so that your datestamp appears at the
beginning of the line and in the example format in the section above (using
slashes instead of dashes) then you'd rewrite this REGEX as follows:
```Perl
...
if ($ThisLine =~ m/^\[$SearchDate\] /o) {
...
```
or using regular brackets anywhere on the line:
```Perl
...
if ($ThisLine =~ m/\($SearchDate\) /o) {
...
```
or without any brackets but appearing at the beginning of the line:
```Perl
...
if ($ThisLine =~ m/^$SearchDate /o) {
...
```
## Testing
Run *logwatch --help* and note the options. You can test just this service
locally on your screen with the following command (assuming you kept default
names for everything):
```Bash
# Summary output, entire duration of log file
logwatch --service backup --output stdout --format text --range all --detail 0
# Minimal detail, yesterday only
logwatch --service backup --output stdout --format text --range yesterday --detail 3
# Verbose output, today only
logwatch --service backup --output stdout --format text --range today --detail 5
```
## Final thoughts
That's it! I'm a horrible PERL programmer so if anyone can optimize/improve the
script file used for Logwatch then please do it! Otherwise, I hope this made
sense and helped you integrate the backup script with Logwatch for easy
monitoring :-)

112
etc/logwatch/scripts/services/backup
View File

@ -1,112 +0,0 @@
#!/usr/bin/perl
#############################################################################
# $Id$
#############################################################################
# Log: Backup script (backup)
# Revision 1.0 2018/10/16
# Written by Asif Bacchus
#############################################################################
use strict;
### Get Logwatch detail level (or default to 0)
my $detailLevel = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;
### Declare variables
my $summaryErr;
my $summaryWarn;
my $summarySuccess;
my %reportHash = ();
my $key;
### Minimal detail level: provide summary data only
if ($detailLevel == 0) {
### process logfile and summarize message types
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
$summaryErr++;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
$summaryWarn++;
}
elsif ($ThisLine =~ /All processes completed successfully/) {
$summarySuccess++;
}
}
### fill hash table with headings and summary counts
if ($summarySuccess > 0) {
$reportHash{"All processes successfully completed"} = $summarySuccess;
}
if ($summaryWarn > 0) {
$reportHash{"Warnings issued"} = $summaryWarn;
}
if ($summaryErr > 0) {
$reportHash{"Errors encountered"} = $summaryErr;
}
### print hash table
foreach $key (sort keys %reportHash) {
print "$key: $reportHash{$key}\n";
}
}
### Levels 1-4 provide the actual error, warning and success messages instead
### of a summary count
elsif ($detailLevel >= 1 && $detailLevel <= 4) {
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[SUCCESS\] /) {
print $ThisLine;
}
}
}
### Level 5 is similiar to levels 1-4 except it also reports informational
### messages such as the location of script created files, variable checks,
### etc. This is useful when verifying the script's operation.
elsif ($detailLevel == 5) {
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[SUCCESS\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[INFO\] /) {
print $ThisLine;
}
}
}
### Any level above 5 will echo the entire log including the debugging notes
### within the script meant for troubleshooting. Using this level of detail
### should only be done if you cannot view the actual log file directly for
### whatever reason. The actual log file is colour-coded for easier debugging.
elsif ($detailLevel > 5) {
while (defined(my $ThisLine = <STDIN>)) {
print $ThisLine;
}
}
### Exit gracefully
exit (0);
# vi: shiftwidth=3 tabstop=3 et
# Local Variables:
# mode: perl
# perl-indent-level: 3
# indent-tabs-mode: nil
# End: