Cleaned up TOC

README.md

@@ -1,4 +1,4 @@
-# CloudflareDDNS
+# CloudflareDDNS <!-- omit in toc -->
 
 Update your CloudFlare DNS records with your current (dynamic) IP address via
 systemd timers and a bash script.
@@ -7,18 +7,18 @@ systemd timers and a bash script.
 auto-update itself.  However, you MUST update the systemd service file
 *(cfddns.service)* *ExecStart* line manually as explained below.**
 
-## Contents
+## Contents <!-- omit in toc -->
 
-* [cfddns&#46;sh](#cfddns46sh)
-  * [Installation:](#installation)
-  * [Usage:](#usage)
-* [cfddns.service](#cfddnsservice)
-  * [IP4 and/or IP6](#ip4-andor-ip6)
-  * [Examples](#examples)
-* [cfddns.timer](#cfddnstimer)
-  * [Activation](#activation)
-* [The log file](#the-log-file)
-* [Final thoughts](#final-thoughts)
+- [cfddns&#46;sh](#cfddns46sh)
+    - [Installation:](#installation)
+    - [Usage:](#usage)
+- [cfddns.service](#cfddnsservice)
+    - [IP4 and/or IP6](#ip4-andor-ip6)
+    - [Examples](#examples)
+- [cfddns.timer](#cfddnstimer)
+    - [Activation](#activation)
+- [The log file](#the-log-file)
+- [Final thoughts](#final-thoughts)
 
 ## cfddns&#46;sh
 
@@ -41,9 +41,10 @@ else in your path so it's easy to run.
    ```
 
 #### Usage:
+
 If you run the script with no parameters, it will display the help screen.  The
 script accepts several parameters with 2 being required.  The parameters are
-summarized here.  You can access the help screen and example usage screens by 
+summarized here.  You can access the help screen and example usage screens by
 running:
 
 ```Bash
@@ -52,17 +53,17 @@ cfddns.sh -x   # show script usage examples
 ```
 
 **-f: full path and filename containing account details (REQUIRED)**
-This is the full path to a plain-text file containing your CloudFlare account 
+This is the full path to a plain-text file containing your CloudFlare account
 details.  This file must contain 3 lines in the following order:
 
-* authorized email address
+- authorized email address
    This is an email address that is permitted to login to your CloudFlare account.
-* global api-key
+- global api-key
    You can get your Global API-key by going to your CloudFlare dashboard,
    clicking on your profile picture in the upper-right and opening your profile.
    Scroll down to to the API Keys section.  Click on the 'View' button next to
    Global API Key.
-* zone identifier
+- zone identifier
    You should be able to find this on the Overview page of your CloudFlare dashboard.
   
 Your completed file should look like (these are not real credentials):
@@ -140,8 +141,9 @@ This is the best way to learn how this script works.  Several examples are
 provided
 
 ## cfddns.service
+
 This file **must** be copied to your */etc/systemd/system* directory (or
-equivalent directory if you're not running debian/ubuntu).  If you change the 
+equivalent directory if you're not running debian/ubuntu).  If you change the
 name of the cfddns&#46;sh file, you must update the filename in the *ExecStart*
 line as shown below:
 
@@ -154,6 +156,7 @@ ExecStart=/full/path/to/your/renamed.file -parameter1 -parameter2 -parameter...
 ````
 
 #### IP4 and/or IP6
+
 The cfddns.service file includes two *ExecStart* lines, one without a specified
 IP-protocol parameter (default IP4) and the other with the -6 (IP6) parameter.
 The service will run the cfddns&#46;sh script in default (IP4) mode with specified
@@ -163,6 +166,7 @@ parameters.
 *Note: The parameters *can be different* in each case.*
 
 #### Examples
+
 1. **Only update A records**
   Update *mail<span>.example.com* A record with the current IP of this machine and log
   results to */var/log/cfddns.log*.
@@ -201,6 +205,7 @@ parameters.
    ```
 
 ## cfddns.timer
+
 This is the timer file that tells your system how often to call the
 cfddns.service file which runs the cfddns&#46;sh script.  By default, the timer is
 set for 5 minutes after the system boots up (to allow for other processes to
@@ -225,6 +230,7 @@ cfddns.service again.  I recommend setting OnUnitActiveSec to a low value (like
 minutes) after everything is working.
 
 #### Activation
+
 You can start the timer system immediately via systemctl
 
 ```Bash
@@ -247,6 +253,7 @@ It is NOT necessary to enable/start the *cfddns.service*, only the timer needs
 to be active.
 
 ## The log file
+
 The script logs every major action it takes and provides details on any errors
 it encounters in the log file (see the above section for details on log file
 location and name).  If errors are encountered, they are colour coded red and
@@ -258,16 +265,21 @@ have to review this log as part of your daily routine.  To make that easier, the
 following conventions are observed in the log file and can be used to program
 your log-watch system:
 
-* Errors always appear as **-- [ERROR] text and error code here --**
-* Errors are followed by an explanation of the specific error code on a new line
-* A clean exit appears as **-- [SUCCESS] some text here --**
-* The script always starts a new set of log entries with **-- Start CloudFlare
+- Errors always appear as **-- [ERROR] text and error code here --**
+- Errors are followed by an explanation of the specific error code on a new line
+- A clean exit appears as **-- [SUCCESS] some text here --**
+- The script always starts a new set of log entries with **-- Start CloudFlare
   DDNS script execution --**
-* All log file entries start with a time-stamp in **[square brackets]**
+- All log file entries start with a time-stamp in **[square brackets]**
+
+If you are using the Logwatch package to monitor your system, see the README in
+the /etc/logwatch folder for details about the pre-configured service files
+already done for you :-)
 
 ## Final thoughts
+
 I'm by no means an expert in BASH scripting and I only program/script as a hobby
-when I find stuff that irritates me and no other good solutions seem easily 
+when I find stuff that irritates me and no other good solutions seem easily
 available.  So, by all means, please comment, provide feedback and suggestions
 to make this script better!  Thanks, I hope this helps someone else out!
 

etc/logwatch/README.md

@@ -0,0 +1,265 @@
+# Using Logwatch to monitor Cloudflare DDNS updater script <!-- omit in toc -->
+
+The Cloudflare DDNS update script's log file has been set up so that utilities
+like Logwatch can easily parse it.  In order to make that happen, a LogFile
+Group file, Service and Script have to be created for Logwatch to generate
+reports.  The correct (general) directory structure has been created in this git
+archive already.  Below are the details of each file.
+
+## Contents <!-- omit in toc -->
+
+- [LogFile Group file (/etc/logwatch/conf/logfiles/cfddns.conf)](#logfile-group-file-etclogwatchconflogfilescfddnsconf)
+    - [Log file location](#log-file-location)
+    - [Archive location and name format](#archive-location-and-name-format)
+    - [External script for timestamp processing](#external-script-for-timestamp-processing)
+- [Service definition file (/etc/logwatch/conf/services/cfddns.conf)](#service-definition-file-etclogwatchconfservicescfddnsconf)
+    - [LogFile Group file definition](#logfile-group-file-definition)
+    - [Report title](#report-title)
+- [Service script (/etc/logwatch/scripts/services/cfddns)](#service-script-etclogwatchscriptsservicescfddns)
+    - [Detail levels](#detail-levels)
+- [Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
+    - [The time format specification](#the-time-format-specification)
+    - [The search REGEX](#the-search-regex)
+- [Testing](#testing)
+- [Final thoughts](#final-thoughts)
+
+## LogFile Group file (/etc/logwatch/conf/logfiles/cfddns.conf)
+
+### Log file location
+
+Update this as needed to point to the location and name of the log file
+generated by the updater script.  Remember, by default, the log file is created
+in the same directory as the script itself.
+
+```Ini
+LogFile = /path/to/your/cfddns.log
+...
+```
+
+Best practices suggest you use the *-l*
+flag to change this location to something like */var/log/cfddns.log*, for
+example.  In that case, the entry would look like:
+
+```Ini
+LogFile = /var/log/cfddns.log
+...
+```
+
+### Archive location and name format
+
+If you want Logwatch to process old (archived) log files generated by something
+like *Logrotate*, then you have to specify the location and file name format of
+those files.  I've included the generalized compressed format of such rotated
+files as the default in the script.  Suppose you store your log files in the
+recommended location (*/var/log/*) and are using *Logrotate* with compression
+enabled, the archive line would look like:
+
+```Ini
+...
+Archive = /var/log/cfddns.log.?.gz
+...
+```
+
+This would tell Logwatch, when the archive option is set to true, that your
+*cfddns.log* files are archived as: *cfddns.log.1.gz*, *cfddns.log.2.gz*, etc.
+and are all located in */var/log/*.
+
+**Note: This line is totally optional and only used if you set the archive
+option in Logwatch to true.  You can comment/delete this line if you wish.**
+
+### External script for timestamp processing
+
+Since the log file uses a non-standard (according to Logwatch) method of
+datestamping, a custom filter had to be created.  See the
+[relevant](#timestamp-processing-script-etclogwatchscriptssharedsqfullstampanywhere)
+section of this document for more information.
+
+The script file is called with an *\** before the filename.
+
+```Ini
+...
+*sqFullStampAnywhere
+...
+```
+
+If you change the name of this file, you will have to change this line.
+Remember that whatever you type here as a name is converted to all-lowercase
+so your filename should be all lowercase also.
+
+## Service definition file (/etc/logwatch/conf/services/cfddns.conf)
+
+### LogFile Group file definition
+
+The service file needs to know what group of log files it is responsible for
+processing.  This MUST match the name of your *LogFile Group file*:
+
+```Ini
+LogFile = cfddns
+...
+```
+
+If you change your LogFile Group filename, then update it here too without the
+*.conf* extension.
+
+### Report title
+
+The Logwatch output file (html or text) is divided into sections.  You can
+define the title to be anything that has meaning for you.  I have arbitrarily
+chosen *"CloudFlare DDNS update"* but you can change it to anything you want by
+modifying the line:
+
+```Ini
+...
+Title = "CloudFlare DDNS update"
+```
+
+## Service script (/etc/logwatch/scripts/services/cfddns)
+
+Logwatch calls any script with a name that **matches the service name**.  You'll
+notice that I just named everything *cfddns* to keep things simple.  You can
+change this to whatever you want, however.  If you changed the service name to
+*"cloudflare*.conf", for example, you would have to rename this script file to
+"*cloudflare*" with no extension.  Note: The script is a PERL file (note the
+shebang) but it can be written in any language.
+
+In essence, Logwatch just spits out the log file(s) defined in the LogFile Group
+file as standard input (STDIN) for the script and then takes whatever is output
+(STDOUT) from the script to assemble into it's report.
+
+### Detail levels
+
+The script supports four (4) detail levels as follows:
+
+- **Level 0: Summary output only**
+  - This will display an aggregate total of certain logged elements.  It will
+    display the total number of hostnames (A and AAAA) that are already
+    up-to-date, those that needed updating, those successfully updated and the
+    total number of errors (of any type) encountered by the script.  All totals
+    are relative to the reporting period Logwatch is using (--range parameter).
+
+    **This is the recommended reporting level.**  It does not take up much space
+    and is quick to read.  If you see successful updates match the number of
+    needed updates and no errors logged, then things are working properly.  If
+    you notice errors, you should consult the full logs.
+- **Levels 1-4: Critical messages**
+  - This uses the data which is summarized by Level 0 but outputs the actual
+    messages in the log file.  For example, you will see the actual text of the
+    errors logged instead of just a total number of errors.  This level of
+    reporting is useful when *initially* monitoring the script's operation since
+    you can see the actual text of any generated errors.
+- **Level 5: Verbose (debugging) output**
+  - Like the previous level, this outputs the actual messages found in the log
+    file.  However, it also includes *[INFO] tags* which contain logged messages
+    such as the detected IP address and the specific names of any hostnames not
+    found in your Cloudflare account, etc.  This level of reporting is useful in
+    diagnosing why errors are occurring or if you just want more insight into
+    how the script works.
+
+    **This level of output will make your Logwatch reports longer and consume
+    more of your time to review. You should not use this level day-to-day.**
+- **Levels 6+: Complete log file dump**
+  - Any number greater than 5 passed as a detail level will trigger the script
+    to dump the entire log file out to Logwatch line-by-line.  This is useful
+    only if you are debugging an issue and cannot get access to the actual raw
+    log file itself.  The actual log file is colour-coded which makes it much
+    easier to read for debugging purposes.
+
+    **Use this detail level only when you need to see the entire log file and
+    cannot otherwise access the log file.**
+
+## Timestamp processing script (/etc/logwatch/scripts/shared/sqfullstampanywhere)
+
+This is basically a modified version of the '*applyeurodate*' script that comes
+with Logwatch.  It had to be modified to search within [square brackets] and to
+accept characters coming before the stamp (i.e. ANSI colour codes).  If you
+change the '**stamp**' variable in the updater script to update the timestamp to
+your liking (which to totally fine!) then you'll probably have to update this
+file.  There are two lines you need to modify to suit your new '**stamp**'
+variable.
+
+### The time format specification
+
+'*$SearchDate*' is the variable used in the PERL script to do exactly what it
+says, search for the date stamp.  I have it set up to look for the format
+'*year-month-date hour:minute:second*'.  Note, we don't care about brackets or
+anything here, we're just defining the format of the date/time stamp.
+
+```Perl
+...
+$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S');
+...
+```
+
+If you changed the '**stamp**' variable so it was formatted as '*month/day/year
+hour:minute*' (ex: '*[09/27/2018 18:38]*') then you'd update the **$SearchDate**
+variable as follows (note: no mention of the square brackets!):
+
+```Perl
+...
+$SearchDate = TimeFilter('%m/%d/%Y %H:%M');
+...
+```
+
+### The search REGEX
+
+The PERL script uses a '*regular expression*' (REGEX) to search within the log file for
+'*$SearchDate*'.  For the default datestamp, this specification looks like:
+
+```Perl
+...
+if ($ThisLine =~ m/\[$SearchDate\] /o) {
+...
+```
+
+The REGEX appears between '*m/*' and '*/o*'.  In this case, it searches for
+'*$SearchDate*' inside [square brackets] appearing anywhere on the line.  This
+is because ANSI colour-codes often appear before the datestamp in the default
+log file.  If you have modified this so that your datestamp appears at the
+beginning of the line and in the example format in the section above (using
+slashes instead of dashes) then you'd rewrite this REGEX as follows:
+
+```Perl
+...
+if ($ThisLine =~ m/^\[$SearchDate\] /o) {
+...
+```
+
+or using regular brackets anywhere on the line:
+
+```Perl
+...
+if ($ThisLine =~ m/\($SearchDate\) /o) {
+...
+```
+
+or without any brackets but appearing at the beginning of the line:
+
+```Perl
+...
+if ($ThisLine =~ m/^$SearchDate /o) {
+...
+```
+
+## Testing
+
+Run *logwatch --help* and note the options.  You can test just this service
+locally on your screen with the following command (assuming you kept default
+names for everything):
+
+```Bash
+# Summary output, entire duration of log file
+logwatch --service cfddns --output stdout --format text --range all --detail 0
+
+# Minimal detail, yesterday only
+logwatch --service cfddns --output stdout --format text --range yesterday --detail 3
+
+# Verbose output, today only
+logwatch --service cfddns --output stdout --format text --range today --detail 5
+```
+
+## Final thoughts
+
+That's it!  I'm a horrible PERL programmer so if anyone can optimize/improve the
+script file used for Logwatch then please do it!  Otherwise, I hope this made
+sense and helped you integrate the updater script with Logwatch for easy
+monitoring :-)