290 lines
11 KiB
Markdown
290 lines
11 KiB
Markdown
# Using Logwatch to monitor backup script <!-- omit in toc -->
|
|
|
|
The backup script's log file has been set up so that utilities like Logwatch can
|
|
easily parse it. In order to make that happen, a LogFile Group file, Service
|
|
and Script have to be created for Logwatch to generate reports. The correct
|
|
(general) directory structure has been created in this git archive already.
|
|
Below are the details of each file.
|
|
|
|
## Contents <!-- omit in toc -->
|
|
|
|
- [LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)](#logfile-group-file-etclogwatchconflogfilesbackupconf)
|
|
- [Log file location](#log-file-location)
|
|
- [Archive location and name format](#archive-location-and-name-format)
|
|
- [External script for timestamp processing](#external-script-for-timestamp-processing)
|
|
- [Service definition file (/etc/logwatch/conf/services/backup.conf)](#service-definition-file-etclogwatchconfservicesbackupconf)
|
|
- [LogFile Group file definition](#logfile-group-file-definition)
|
|
- [Report title](#report-title)
|
|
- [Detail level](#detail-level)
|
|
- [Service script (/etc/logwatch/scripts/services/backup)](#service-script-etclogwatchscriptsservicesbackup)
|
|
- [Detail levels](#detail-levels)
|
|
- [Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
|
|
- [The time format specification](#the-time-format-specification)
|
|
- [The search REGEX](#the-search-regex)
|
|
- [Testing](#testing)
|
|
- [Final thoughts](#final-thoughts)
|
|
|
|
## LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)
|
|
|
|
### Log file location
|
|
|
|
Update this as needed to point to the location and name of the log file
|
|
generated by the backup script. Remember, by default, the log file is created
|
|
in the same directory as the script itself.
|
|
|
|
```Ini
|
|
LogFile = /path/to/your/backup.log
|
|
...
|
|
```
|
|
|
|
Best practices suggest you use the *-l*
|
|
flag to change this location to something like */var/log/backup.log*, for
|
|
example. In that case, the entry would look like:
|
|
|
|
```Ini
|
|
LogFile = /var/log/backup.log
|
|
...
|
|
```
|
|
|
|
### Archive location and name format
|
|
|
|
If you want Logwatch to process old (archived) log files generated by something
|
|
like *Logrotate*, then you have to specify the location and file name format of
|
|
those files. I've included the generalized compressed format of such rotated
|
|
files as the default in the script. Suppose you store your log files in the
|
|
recommended location (*/var/log/*) and are using *Logrotate* with compression
|
|
enabled, the archive line would look like:
|
|
|
|
```Ini
|
|
...
|
|
Archive = /var/log/backup.log.?.gz
|
|
...
|
|
```
|
|
|
|
This would tell Logwatch, when the archive option is set to true, that your
|
|
*backup.log* files are archived as: *backup.log.1.gz*, *backup.log.2.gz*, etc.
|
|
and are all located in */var/log/*.
|
|
|
|
**Note: This line is totally optional and only used if you set the archive
|
|
option in Logwatch to true (default). You can comment/delete this line if you
|
|
wish.**
|
|
|
|
### External script for timestamp processing
|
|
|
|
Since the log file uses a non-standard (according to Logwatch) method of
|
|
datestamping, a custom filter had to be created. See the
|
|
[relevant](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
|
|
section of this document for more information.
|
|
|
|
The script file is called with an *\** before the filename.
|
|
|
|
```Ini
|
|
...
|
|
*sqFullStampAnywhere
|
|
...
|
|
```
|
|
|
|
If you change the name of this file, you will have to change this line.
|
|
Remember that whatever you type here as a name is converted to all-lowercase
|
|
so your filename should be all lowercase also.
|
|
|
|
## Service definition file (/etc/logwatch/conf/services/backup.conf)
|
|
|
|
### LogFile Group file definition
|
|
|
|
The service file needs to know what group of log files it is responsible for
|
|
processing. This MUST match the name of your *LogFile Group file*:
|
|
|
|
```Ini
|
|
LogFile = backup
|
|
...
|
|
```
|
|
|
|
If you change your LogFile Group filename, then update it here too without the
|
|
*.conf* extension.
|
|
|
|
### Report title
|
|
|
|
The Logwatch output file (html or text) is divided into sections. You can
|
|
define the title to be anything that has meaning for you. I have arbitrarily
|
|
chosen *"System and NextCloud Backup"* but you can change it to anything you want by
|
|
modifying the line:
|
|
|
|
```Ini
|
|
...
|
|
Title = "System and NextCloud Backup"
|
|
```
|
|
|
|
### Detail level
|
|
|
|
If you want to set the *detail* level of this service differently from your
|
|
other services (which will use the *--detail* switch value or the value in your
|
|
*logwatch.conf*), then you can define that level here. By default, it appears
|
|
like this in the service configuration file:
|
|
|
|
```Ini
|
|
...
|
|
# Override the detail level for this service
|
|
# Remember the levels are: 0, 1-4, 5, 6+
|
|
# Detail = 0
|
|
```
|
|
|
|
Simply change it to the value you want enforced. For example, here I'm setting
|
|
it to output level 5 regardless of whatever settings everything else is using.
|
|
|
|
```Ini
|
|
# Override the detail level for this service
|
|
# Remember the levels are: 0, 1-4, 5, 6+
|
|
Detail = 5
|
|
```
|
|
|
|
## Service script (/etc/logwatch/scripts/services/backup)
|
|
|
|
Logwatch calls any script with a name that **matches the service name**. You'll
|
|
notice that I just named everything *backup* to keep things simple. You can
|
|
change this to whatever you want, however. If you changed the service name to
|
|
*"NCbackup*.conf", for example, you would have to rename this script file to
|
|
"*NCbackup*" with no extension. Note: The script is a PERL file (note the
|
|
shebang) but it can be written in any language.
|
|
|
|
In essence, Logwatch just spits out the log file(s) defined in the LogFile Group
|
|
file as standard input (STDIN) for the script and then takes whatever is output
|
|
(STDOUT) from the script to assemble into it's report.
|
|
|
|
### Detail levels
|
|
|
|
The script supports four (4) detail levels as follows:
|
|
|
|
- **Level 0: Summary output only**
|
|
- This will display an aggregate total of certain logged elements. It will
|
|
display the total number successful script executions (SQL dump OK, borg
|
|
backup OK and borg prune OK), total generated warnings and total errors
|
|
encountered that stopped the normal execution of the script. All totals are
|
|
relative to the reporting period Logwatch is using (--range parameter).
|
|
|
|
**This is the recommended reporting level.** It does not take up much space
|
|
and is quick to read. If you notice warnings and/or errors, you should
|
|
consult the full logs.
|
|
- **Levels 1-4: Critical messages**
|
|
- This uses the data which is summarized by Level 0 but outputs the actual
|
|
messages in the log file. For example, you will see the actual text of the
|
|
errors logged instead of just a total number of errors. This level of
|
|
reporting is useful when *initially* monitoring the script's operation since
|
|
you can see the actual text of any generated warnings/errors.
|
|
- **Level 5: Verbose (debugging) output**
|
|
- Like the previous level, this outputs the actual messages found in the log
|
|
file. However, it also includes *[INFO] tags* which contain logged
|
|
operational messages such as created temporary directories, entering/exiting
|
|
maintenance mode, what source files are being used to determine extra file
|
|
inclusions/exclusions, etc. This level of reporting is useful in diagnosing
|
|
why errors are occurring or if you just want more insight into how the
|
|
script works.
|
|
|
|
**This level of output will make your Logwatch reports longer and consume
|
|
more of your time to review. You should not use this level day-to-day.**
|
|
- **Levels 6+: Complete log file dump**
|
|
- Any number greater than 5 passed as a detail level will trigger the script
|
|
to dump the entire log file out to Logwatch line-by-line. This is useful
|
|
only if you are debugging an issue and cannot get access to the actual raw
|
|
log file itself. The actual log file is colour-coded which makes it much
|
|
easier to read for debugging purposes.
|
|
|
|
**Use this detail level only when you need to see the entire log file and
|
|
cannot otherwise access the log file.**
|
|
|
|
## Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)
|
|
|
|
This is basically a modified version of the '*applyeurodate*' script that comes
|
|
with Logwatch. It had to be modified to search within [square brackets] and to
|
|
accept characters coming before the stamp (i.e. ANSI colour codes). If you
|
|
change the '**stamp**' variable in the backup script to update the timestamp to
|
|
your liking (which to totally fine!) then you'll probably have to update this
|
|
file. There are two lines you need to modify to suit your new '**stamp**'
|
|
variable.
|
|
|
|
### The time format specification
|
|
|
|
'*$SearchDate*' is the variable used in the PERL script to do exactly what it
|
|
says, search for the date stamp. I have it set up to look for the format
|
|
'*year-month-date hour:minute:second*'. Note, we don't care about brackets or
|
|
anything here, we're just defining the format of the date/time stamp.
|
|
|
|
```Perl
|
|
...
|
|
$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S');
|
|
...
|
|
```
|
|
|
|
If you changed the '**stamp**' variable so it was formatted as '*month/day/year
|
|
hour:minute*' (ex: '*[09/27/2018 18:38]*') then you'd update the **$SearchDate**
|
|
variable as follows (note: no mention of the square brackets!):
|
|
|
|
```Perl
|
|
...
|
|
$SearchDate = TimeFilter('%m/%d/%Y %H:%M');
|
|
...
|
|
```
|
|
|
|
### The search REGEX
|
|
|
|
The PERL script uses a '*regular expression*' (REGEX) to search within the log file for
|
|
'*$SearchDate*'. For the default datestamp, this specification looks like:
|
|
|
|
```Perl
|
|
...
|
|
if ($ThisLine =~ m/\[$SearchDate\] /o) {
|
|
...
|
|
```
|
|
|
|
The REGEX appears between '*m/*' and '*/o*'. In this case, it searches for
|
|
'*$SearchDate*' inside [square brackets] appearing anywhere on the line. This
|
|
is because ANSI colour-codes often appear before the datestamp in the default
|
|
log file. If you have modified this so that your datestamp appears at the
|
|
beginning of the line and in the example format in the section above (using
|
|
slashes instead of dashes) then you'd rewrite this REGEX as follows:
|
|
|
|
```Perl
|
|
...
|
|
if ($ThisLine =~ m/^\[$SearchDate\] /o) {
|
|
...
|
|
```
|
|
|
|
or using regular brackets anywhere on the line:
|
|
|
|
```Perl
|
|
...
|
|
if ($ThisLine =~ m/\($SearchDate\) /o) {
|
|
...
|
|
```
|
|
|
|
or without any brackets but appearing at the beginning of the line:
|
|
|
|
```Perl
|
|
...
|
|
if ($ThisLine =~ m/^$SearchDate /o) {
|
|
...
|
|
```
|
|
|
|
## Testing
|
|
|
|
Run *logwatch --help* and note the options. You can test just this service
|
|
locally on your screen with the following command (assuming you kept default
|
|
names for everything):
|
|
|
|
```Bash
|
|
# Summary output, entire duration of log file
|
|
logwatch --service backup --output stdout --format text --range all --detail 0
|
|
|
|
# Minimal detail, yesterday only
|
|
logwatch --service backup --output stdout --format text --range yesterday --detail 3
|
|
|
|
# Verbose output, today only
|
|
logwatch --service backup --output stdout --format text --range today --detail 5
|
|
```
|
|
|
|
## Final thoughts
|
|
|
|
That's it! I'm a horrible PERL programmer so if anyone can optimize/improve the
|
|
script file used for Logwatch then please do it! Otherwise, I hope this made
|
|
sense and helped you integrate the backup script with Logwatch for easy
|
|
monitoring :-) |