diff --git a/etc/logwatch/readme.md b/etc/logwatch/readme.md new file mode 100644 index 0000000..a93df5c --- /dev/null +++ b/etc/logwatch/readme.md @@ -0,0 +1,290 @@ +# Using Logwatch to monitor backup script + +The backup script's log file has been set up so that utilities like Logwatch can +easily parse it. In order to make that happen, a LogFile Group file, Service +and Script have to be created for Logwatch to generate reports. The correct +(general) directory structure has been created in this git archive already. +Below are the details of each file. + +## Contents + +- [LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf)](#logfile-group-file-etclogwatchconflogfilesbackupconf) + - [Log file location](#log-file-location) + - [Archive location and name format](#archive-location-and-name-format) + - [External script for timestamp processing](#external-script-for-timestamp-processing) +- [Service definition file (/etc/logwatch/conf/services/backup.conf)](#service-definition-file-etclogwatchconfservicesbackupconf) + - [LogFile Group file definition](#logfile-group-file-definition) + - [Report title](#report-title) + - [Detail level](#detail-level) +- [Service script (/etc/logwatch/scripts/services/backup)](#service-script-etclogwatchscriptsservicesbackup) + - [Detail levels](#detail-levels) +- [Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere) + - [The time format specification](#the-time-format-specification) + - [The search REGEX](#the-search-regex) +- [Testing](#testing) +- [Final thoughts](#final-thoughts) + +## LogFile Group file (/etc/logwatch/conf/logfiles/backup.conf) + +### Log file location + +Update this as needed to point to the location and name of the log file +generated by the backup script. Remember, by default, the log file is created +in the same directory as the script itself. + +```Ini +LogFile = /path/to/your/backup.log +... +``` + +Best practices suggest you use the *-l* +flag to change this location to something like */var/log/backup.log*, for +example. In that case, the entry would look like: + +```Ini +LogFile = /var/log/backup.log +... +``` + +### Archive location and name format + +If you want Logwatch to process old (archived) log files generated by something +like *Logrotate*, then you have to specify the location and file name format of +those files. I've included the generalized compressed format of such rotated +files as the default in the script. Suppose you store your log files in the +recommended location (*/var/log/*) and are using *Logrotate* with compression +enabled, the archive line would look like: + +```Ini +... +Archive = /var/log/backup.log.?.gz +... +``` + +This would tell Logwatch, when the archive option is set to true, that your +*backup.log* files are archived as: *backup.log.1.gz*, *backup.log.2.gz*, etc. +and are all located in */var/log/*. + +**Note: This line is totally optional and only used if you set the archive +option in Logwatch to true (default). You can comment/delete this line if you +wish.** + +### External script for timestamp processing + +Since the log file uses a non-standard (according to Logwatch) method of +datestamping, a custom filter had to be created. See the +[relevant](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere) +section of this document for more information. + +The script file is called with an *\** before the filename. + +```Ini +... +*sqFullStampAnywhere +... +``` + +If you change the name of this file, you will have to change this line. +Remember that whatever you type here as a name is converted to all-lowercase +so your filename should be all lowercase also. + +## Service definition file (/etc/logwatch/conf/services/backup.conf) + +### LogFile Group file definition + +The service file needs to know what group of log files it is responsible for +processing. This MUST match the name of your *LogFile Group file*: + +```Ini +LogFile = backup +... +``` + +If you change your LogFile Group filename, then update it here too without the +*.conf* extension. + +### Report title + +The Logwatch output file (html or text) is divided into sections. You can +define the title to be anything that has meaning for you. I have arbitrarily +chosen *"System and NextCloud Backup"* but you can change it to anything you want by +modifying the line: + +```Ini +... +Title = "System and NextCloud Backup" +``` + +### Detail level + +If you want to set the *detail* level of this service differently from your +other services (which will use the *--detail* switch value or the value in your +*logwatch.conf*), then you can define that level here. By default, it appears +like this in the service configuration file: + +```Ini +... +# Override the detail level for this service +# Remember the levels are: 0, 1-4, 5, 6+ +# Detail = 0 +``` + +Simply change it to the value you want enforced. For example, here I'm setting +it to output level 5 regardless of whatever settings everything else is using. + +```Ini +# Override the detail level for this service +# Remember the levels are: 0, 1-4, 5, 6+ +Detail = 5 +``` + +## Service script (/etc/logwatch/scripts/services/backup) + +Logwatch calls any script with a name that **matches the service name**. You'll +notice that I just named everything *backup* to keep things simple. You can +change this to whatever you want, however. If you changed the service name to +*"NCbackup*.conf", for example, you would have to rename this script file to +"*NCbackup*" with no extension. Note: The script is a PERL file (note the +shebang) but it can be written in any language. + +In essence, Logwatch just spits out the log file(s) defined in the LogFile Group +file as standard input (STDIN) for the script and then takes whatever is output +(STDOUT) from the script to assemble into it's report. + +### Detail levels + +The script supports four (4) detail levels as follows: + +- **Level 0: Summary output only** + - This will display an aggregate total of certain logged elements. It will + display the total number successful script executions (SQL dump OK, borg + backup OK and borg prune OK), total generated warnings and total errors + encountered that stopped the normal execution of the script. All totals are + relative to the reporting period Logwatch is using (--range parameter). + + **This is the recommended reporting level.** It does not take up much space + and is quick to read. If you notice warnings and/or errors, you should + consult the full logs. +- **Levels 1-4: Critical messages** + - This uses the data which is summarized by Level 0 but outputs the actual + messages in the log file. For example, you will see the actual text of the + errors logged instead of just a total number of errors. This level of + reporting is useful when *initially* monitoring the script's operation since + you can see the actual text of any generated warnings/errors. +- **Level 5: Verbose (debugging) output** + - Like the previous level, this outputs the actual messages found in the log + file. However, it also includes *[INFO] tags* which contain logged + operational messages such as created temporary directories, entering/exiting + maintenance mode, what source files are being used to determine extra file + inclusions/exclusions, etc. This level of reporting is useful in diagnosing + why errors are occurring or if you just want more insight into how the + script works. + + **This level of output will make your Logwatch reports longer and consume + more of your time to review. You should not use this level day-to-day.** +- **Levels 6+: Complete log file dump** + - Any number greater than 5 passed as a detail level will trigger the script + to dump the entire log file out to Logwatch line-by-line. This is useful + only if you are debugging an issue and cannot get access to the actual raw + log file itself. The actual log file is colour-coded which makes it much + easier to read for debugging purposes. + + **Use this detail level only when you need to see the entire log file and + cannot otherwise access the log file.** + +## Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere) + +This is basically a modified version of the '*applyeurodate*' script that comes +with Logwatch. It had to be modified to search within [square brackets] and to +accept characters coming before the stamp (i.e. ANSI colour codes). If you +change the '**stamp**' variable in the backup script to update the timestamp to +your liking (which to totally fine!) then you'll probably have to update this +file. There are two lines you need to modify to suit your new '**stamp**' +variable. + +### The time format specification + +'*$SearchDate*' is the variable used in the PERL script to do exactly what it +says, search for the date stamp. I have it set up to look for the format +'*year-month-date hour:minute:second*'. Note, we don't care about brackets or +anything here, we're just defining the format of the date/time stamp. + +```Perl +... +$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S'); +... +``` + +If you changed the '**stamp**' variable so it was formatted as '*month/day/year +hour:minute*' (ex: '*[09/27/2018 18:38]*') then you'd update the **$SearchDate** +variable as follows (note: no mention of the square brackets!): + +```Perl +... +$SearchDate = TimeFilter('%m/%d/%Y %H:%M'); +... +``` + +### The search REGEX + +The PERL script uses a '*regular expression*' (REGEX) to search within the log file for +'*$SearchDate*'. For the default datestamp, this specification looks like: + +```Perl +... +if ($ThisLine =~ m/\[$SearchDate\] /o) { +... +``` + +The REGEX appears between '*m/*' and '*/o*'. In this case, it searches for +'*$SearchDate*' inside [square brackets] appearing anywhere on the line. This +is because ANSI colour-codes often appear before the datestamp in the default +log file. If you have modified this so that your datestamp appears at the +beginning of the line and in the example format in the section above (using +slashes instead of dashes) then you'd rewrite this REGEX as follows: + +```Perl +... +if ($ThisLine =~ m/^\[$SearchDate\] /o) { +... +``` + +or using regular brackets anywhere on the line: + +```Perl +... +if ($ThisLine =~ m/\($SearchDate\) /o) { +... +``` + +or without any brackets but appearing at the beginning of the line: + +```Perl +... +if ($ThisLine =~ m/^$SearchDate /o) { +... +``` + +## Testing + +Run *logwatch --help* and note the options. You can test just this service +locally on your screen with the following command (assuming you kept default +names for everything): + +```Bash +# Summary output, entire duration of log file +logwatch --service backup --output stdout --format text --range all --detail 0 + +# Minimal detail, yesterday only +logwatch --service backup --output stdout --format text --range yesterday --detail 3 + +# Verbose output, today only +logwatch --service backup --output stdout --format text --range today --detail 5 +``` + +## Final thoughts + +That's it! I'm a horrible PERL programmer so if anyone can optimize/improve the +script file used for Logwatch then please do it! Otherwise, I hope this made +sense and helped you integrate the backup script with Logwatch for easy +monitoring :-) \ No newline at end of file