fix(BACKUP): remove extraneous mailcow dir in xtraLocation

- description was misleading, this is not a required variable
- holdover from importing

README.md

@@ -1,4 +1,4 @@
-# Mailcow Backup Using borgbackup <!-- omit in toc -->
+# Mailcow Backup Using borgbackup
 
 This script automates backing up your Mailcow installation using borgbackup and a remote ssh-capable storage system.  I suggest using rsync.net since they
 have great speeds and a special pricing structure for borgbackup/attic users ([details here](https://www.rsync.net/products/attic.html)).
@@ -14,14 +14,19 @@ This script automates the following tasks:
 - Runs 'borg prune' to make sure you are trimming old backups on your schedule
 - Creates a clear, easy to parse log file so you can keep an eye on your backups and any errors/warnings
 
-## Contents <!-- omit in toc -->
+## contents
+
+<!-- toc -->
 
 - [quick start](#quick-start)
 - [configuration file](#configuration-file)
 - [running the script](#running-the-script)
 - [scheduling your backup via cron](#scheduling-your-backup-via-cron)
+- [restoring backups](#restoring-backups)
 - [final notes](#final-notes)
 
+<!-- tocstop -->
+
 ## quick start
 
 Clone this repo or download a release file into a directory of your choosing. For all examples in this document, I will assume you will run the script from */scripts/backup*. Make sure the script file is executable and you protect the *.details* file since it contains things like your repo password:
@@ -111,6 +116,10 @@ Edit your root user's crontab and add an entry like this which would run the scr
 7 1 * * * /scripts/backup/backup.sh -l /var/log/mailcow_backup.log > /dev/null 2>&1
 ```
 
+## restoring backups
+
+Starting with version 3.0, a *restore.sh* file has been included to semi-automate restoring your backups to a clean mailcow instance. There are a few steps required and they are better explained in the wiki than would be possible in a short write-up like this. Please check out the [restore process overview](https://git.asifbacchus.app/asif/MailcowBackup/wiki/8.0-Restore-overview) for more information.
+
 ## final notes
 
 I think that's everything. For detailed information, please review the [wiki](https://git.asifbacchus.app/asif/MailcowBackup/wiki/_pages). If I've forgotten to document something there, please let me know. I know the wiki is long but, I hate how much stuff for Linux and open-source programs/scripts in general are so poorly documented especially for newbies and I didn't want to make that same mistake.

backup/backup.details

@@ -36,12 +36,13 @@ borgRepoPassphrase="p@ssW0rd"
 # FULL PATH where the associated keyfile for your repo is located -- relevant
 # only if your repo requires a keyfile (i.e. 'keyfile' vs 'repokey') and if you
 # are not using the default keyfile location
-borgKeyfileLocation="/var/borgbackup/.config/borg/keys/server_address__repo_name"
+#borgKeyfileLocation="/var/borgbackup/.config/borg/keys/server_address__repo_name"
 
-# REQUIRED: path to text file containing a list (one per line) of files/
+# additional files to backup
-# directories to include in your backup. Since this is a generic backup script,
+# by default, the script will only backup your mailcow program directory
-# nothing is defined by default. Therefore, ONLY files specified in this file
+# (e.g. /opt/mailcow-dockerized) and your mailcow data volumes. If you would
-# will be backed up!
+# like additional files included (perhaps certificates, system configuration,
+# etc.) include those locations listed one item per line in the file below
 # see repo wiki for more details
 borgXtraListPath="/scripts/backup/xtraLocations.borg"
 

backup/backup.sh

@@ -164,6 +164,10 @@ scriptHelp() {
     textblock "Path to write log file"
     defaultsTextblock "(scriptPath/scriptName.log)"
     newline
+    switchTextblock "--compression"
+    textblock "Compression algorithm(s) that borg should use. Please run 'borg help compression' for details."
+    defaultsTextblock "(not specified, use borg default of lz4)"
+    newline
     switchTextblock "[SWITCH] -v | --verbose"
     textblock "Log borg output with increased verbosity (list all files). Careful! Your log file can get very large very quickly!"
     defaultsTextblock "(normal output, option is OFF)"
@@ -300,6 +304,15 @@ while [ $# -gt 0 ]; do
                 badParam empty "$@"
             fi
             ;;
+        --compression)
+            # set borg archive compression
+            if [ -n "$2" ]; then
+                borgCompression="$2"
+                shift
+            else
+                badParam empty "$@"
+            fi
+            ;;
         -v|--verbose)
             # set verbose logging from borg
             borgCreateParams='--list --stats'
@@ -367,6 +380,7 @@ while [ $# -gt 0 ]; do
                 badParam empty "$@"
             else
                 dockerStartTimeout="$2"
+                shift
             fi
             ;;
         -t2|--timeout-stop)
@@ -374,6 +388,7 @@ while [ $# -gt 0 ]; do
                 badParam empty "$@"
             else
                 dockerStopTimeout="$2"
+                shift
             fi
             ;;
         *)
@@ -417,7 +432,10 @@ fi
 if [ ! -f "$mcDockerCompose" ]; then
     badParam dne "(--docker-compose)" "$mcDockerCompose"
 fi
-
+# set compression level if specified
+if [ -n "$borgCompression" ]; then
+    borgCreateParams="${borgCreateParams} --compression ${borgCompression}"
+fi
 
 ### read mailcow.conf and set vars as needed
 # shellcheck source=./mailcow.conf.shellcheck
@@ -615,7 +633,7 @@ if [ "$use503" -eq 1 ]; then
 fi
 
 ### change to mailcow directory so docker commands execute properly
-cd "${mcConfig%/*}" || exitError 100 'Could not change to mailcow directory.'
+\cd "${mcConfig%/*}" || exitError 100 'Could not change to mailcow directory.'
 
 ### stop postfix and dovecot mail containers to prevent mailflow during backup
 doDocker stop postfix
@@ -672,7 +690,7 @@ printf "%s[%s] -- [INFO] Pre-backup tasks completed, calling borgbackup --%s\n"
 ## construct the proper borg commandline
 # base command
 if [ "$exclusions" -eq 0 ]; then
-    borgCMD="borg --show-rc create ${borgCreateParams} \
+    borgCMD="borg create --show-rc ${borgCreateParams} \
         ::$(date +%Y-%m-%d_%H%M%S) \
         ${mcConfig%/*} \
         ${sqlDumpDir} \
@@ -683,7 +701,7 @@ if [ "$exclusions" -eq 0 ]; then
         ${dockerVolumeCrypt} \
         ${xtraList}"
 elif [ "$exclusions" -eq 1 ]; then
-    borgCMD="borg --show-rc create ${borgCreateParams} \
+    borgCMD="borg create --show-rc ${borgCreateParams} \
         --exclude-from ${borgExcludeListPath} \
         ::$(date +%Y-%m-%d_%H%M%S) \
         ${mcConfig%/*} \

backup/restore.sh

@@ -47,11 +47,74 @@ exitError() {
     exit "$1"
 }
 
+doRestore() {
+    sourceFiles=$(find "${backupLocation}" -iname "${1}" -type d)
+    if [ -n "$sourceFiles" ]; then
+        if [ "$verbose" -eq 1 ]; then
+            if (! (cd "$sourceFiles/_data" && tar -cf - .) | (cd "${2}" && tar xvf -) >> "$logfile" ); then
+                return 1
+            else
+                return 0
+            fi
+        else
+            if (! (cd "$sourceFiles/_data" && tar -cf - .) | (cd "${2}" && tar xvf -) > /dev/null 2>&1 ); then
+                return 1
+            else
+                return 0
+            fi
+        fi
+    else
+        return 2
+    fi
+}
+
 scriptHelp() {
     textNewline
     printf "%sUsage: %s [parameters]%s\n\n" "$bold" "$scriptName" "$norm"
+    textblock "The only required parameter is -b | --backup-location."
+    textblock "If a parameter is not supplied, its default value will be used."
+    textblock "Switch parameters will only be activated if specified."
+    textblockHeader "script parameters"
+    textblockParam "-b | --backup-location"
+    textblock "Directory containing extracted backup files from borg repo. REQUIRED."
     textNewline
-    textblock "If a parameter is not supplied, its default value will be used. Switch parameters will remain DEactivated if NOT specified."
+    textblockParam "-l | --log" "scriptPath/scriptName.log"
+    textblock "Path to write log file. Best efforts will be made to create any specified paths."
+    textNewline
+    textblockParam "-v | --verbose" "false"
+    textblock "Enable verbose logging. This will list EVERY restored file possibly making your log file quite large! [SWITCH]"
+    textNewline
+    textblockParam "--skip-mail" "false"
+    textblock "Skip restoring mail and encryption key. [SWITCH]"
+    textNewline
+    textblockParam "--skip-sql" "false"
+    textblock "Skip restoring mailcow settings database. [SWITCH]"
+    textNewline
+    textblockParam "--skip-postfix" "false"
+    textblock "Skip restoring postfix settings. [SWITCH]"
+    textNewline
+    textblockParam "--skip-rspamd" "false"
+    textblock "Skip restoring Rspamd settings/configuration/history. [SWITCH]"
+    textNewline
+    textblockParam "--skip-redis" "false"
+    textblock "Skip restoring redis database. [SWITCH]"
+    textNewline
+    textblockParam "-? | -h | --help"
+    textblock "Display this help screen."
+    textblockHeader "mailcow parameters"
+    textblockParam "-d | --docker-compose" "/opt/mailcow-dockerized/docker-compose.yml"
+    textblock "FULL path to mailcow's 'docker-compose.yml' file."
+    textNewline
+    textblockParam "-m | --mailcow-config" "/opt/mailcow-dockerized/mailcow.conf"
+    textblock "FULL path to mailcow configuration file ('mailcow.conf'). The path of this file is also used to determine your mailcow directory."
+    textblockHeader "docker parameters"
+    textblockParam "-t1 | --timeout-start" "180"
+    textblock "Seconds to wait for docker containers to start."
+    textNewline
+    textblockParam "-t2 | --timeout-stop" "120"
+    textblock "Seconds to wait for docker containers to stop."
+    textNewline
+    textblock "More details and examples of script usage can be found in the repo wiki at ${yellow}https://git.asifbacchus.app/asif/MailcowBackup/wiki${norm}"
     textNewline
     exit 0
 }
@@ -65,7 +128,7 @@ textblock() {
 }
 
 textblockHeader() {
-    printf "\n%s%s***%s***%s\n" "$bold" "$magenta" "$1" "$norm"
+    printf "\n%s%s*** %s ***%s\n\n" "$bold" "$magenta" "$1" "$norm"
 }
 
 textblockParam() {
@@ -84,7 +147,6 @@ textNewline() {
 
 trapExit() {
     printf "%s[%s] -- [ERROR] 99: Caught signal --%s\n" "$err" "$(stamp)" "$norm" >> "$logfile"
-    cleanup
     printf "%s[%s] --- %s execution terminated via signal ---\n%s" "$err" "$(stamp)" "$scriptName" "$norm" >> "$logfile"
     exit 99
 }
@@ -202,7 +264,7 @@ while [ $# -gt 0 ]; do
     -b|--backup-location)
         if [ -n "$2" ]; then
             if [ -d "$2" ] && [ -n "$( ls -A "$2" )" ]; then
-                backupLocation="$2"
+                backupLocation="${2%/}"
                 shift
             else
                 consoleError '1' "$1: cannot find specified backup location directory or it is empty."
@@ -227,7 +289,7 @@ while [ $# -gt 0 ]; do
         restoreRspamd=0
         ;;
     *)
-        printf "\n%Unknown option: %s\n" "$err" "$1"
+        printf "\n%sUnknown option: %s\n" "$err" "$1"
         printf "Use '--help' for valid options.%s\n\n" "$norm"
         exit 1
         ;;
@@ -236,9 +298,13 @@ while [ $# -gt 0 ]; do
 done
 
 ### pre-flight checks
+# ensure there's something to do
+if [ "$restoreMail" -eq 0 ] && [ "$restoreSQL" -eq 0 ] && [ "$restorePostfix" -eq 0 ] && [ "$restoreRedis" -eq 0 ] && [ "$restoreRedis" -eq 0 ]; then
+    printf "\n%sAll restore operations skipped -- nothing to do!%s\n\n" "$yellow" "$norm"
+    exit 0
+fi
 # set path so checks are valid for this script environment
 export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
-
 # docker installed?
 if ! command -v docker >/dev/null; then
     consoleError '3' 'docker does not seem to be installed!'
@@ -251,8 +317,12 @@ fi
 if [ ! -f "$mcDockerCompose" ]; then
     consoleError '1' "docker-compose configuration ($mcDockerCompose) cannot be found."
 fi
+# forgot to set backup location?
+if [ -z "$backupLocation" ]; then
+    consoleError '1' "'--backup-location' cannot be unspecified or null/empty."
+fi
 # change to mailcow directory so commands execute properly
-\cd ${mcConfig%/*} || consoleError '4' 'Cannot change to mailcow directory as determined from mailcow.conf location.'
+\cd "${mcConfig%/*}" || consoleError '4' 'Cannot change to mailcow directory as determined from mailcow.conf location.'
 
 ### read mailcow.conf and import vars
 # shellcheck source=./mailcow.conf.shellcheck
@@ -314,7 +384,7 @@ if [ "$restoreSQL" -eq 1 ]; then
     if [ -n "$sqlBackup" ]; then
         # start mysql container if not already running
         if ! docker container inspect -f '{{ .State.Running }}' ${COMPOSE_PROJECT_NAME}_mysql-mailcow_1 > /dev/null 2>&1; then
-            docker-compose up -d mysql-mailcow
+            docker-compose up -d mysql-mailcow > /dev/null 2>&1
             if docker container inspect -f '{{ .State.Running }}' ${COMPOSE_PROJECT_NAME}_mysql-mailcow_1 > /dev/null 2>&1; then
                 sqlRunning=1
             else
@@ -338,27 +408,220 @@ if [ "$restoreSQL" -eq 1 ]; then
         else
             writeLog 'done' 'error'
             writeLog 'error' '13' "Something went wrong while trying to restore SQL database. Perhaps try again?"
+            errorCount=$((errorCount+1))
         fi
     fi
 fi
 
-#TODO: stop containers
+### stop containers (necessary for all restore operations except SQL)
-#TODO: copy backups to correct docker volumes
+writeLog 'task' "Stopping mailcow"
-#TODO: restart docker containers
+if ! docker-compose down --timeout "${dockerStopTimeout}" > /dev/null 2>&1; then
-#TODO: optionally reindex dovecot (parameter)
+    writeLog 'done' 'error'
+    writeLog 'error' '20' "Unable to bring mailcow containers down -- cannot reliably restore. Aborting."
+    exitError 20
+fi
+if [ "$( docker ps --filter "name=${COMPOSE_PROJECT_NAME}" -q | wc -l )" -gt 0 ]; then
+    writeLog 'done' 'error'
+    writeLog 'error' '20' "Unable to bring mailcow containers down -- cannot reliably restore. Aborting."
+    exitError 20
+fi
+writeLog 'done'
+
+### restore mail and encryption key
+if [ "$restoreMail" -eq 1 ]; then
+    if [ "$verbose" -eq 1 ]; then
+        writeLog 'info' "Restoring email"
+    else
+        writeLog 'task' "Restoring email"
+    fi
+
+    # restore email messages
+    doRestore "${COMPOSE_PROJECT_NAME}_vmail-vol-1" "$dockerVolumeMail"; ec="$?"
+    case "$ec" in
+        0)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'success' "Email messages restored"
+            else
+                writeLog 'done'
+            fi
+            ;;
+        1)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '52' "There was an error restoring one or more email messages."
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '52' "There was an error restoring one or more email messages."
+            fi
+            ;;
+        2)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '51' "Cannot locate email message backups!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '51' "Cannot locate email message backups!"
+            fi
+            ;;
+    esac
+
+    # restore encryption key
+    doRestore "${COMPOSE_PROJECT_NAME}_crypt-vol-1" "$dockerVolumeCrypt"; ec="$?"
+    case "$ec" in
+        0)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'success' "Encryption key restored"
+            else
+                writeLog 'done'
+            fi
+            ;;
+        1)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '52' "There was an error restoring the encryption key! Any restored messages are likely *not* readable!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '52' "There was an error restoring the encryption key! Any restored messages are likely *not* readable!"
+            fi
+            ;;
+        2)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '51' "Cannot locate encryption key backup!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '51' "Cannot locate encryption key backup!"
+            fi
+            ;;
+    esac
+fi
+
+### restore postfix
+if [ "$restorePostfix" -eq 1 ]; then
+    if [ "$verbose" -eq 1 ]; then
+        writeLog 'info' "Restoring postfix files"
+    else
+        writeLog 'task' "Restoring postfix files"
+    fi
+
+    doRestore "${COMPOSE_PROJECT_NAME}_postfix-vol-1" "$dockerVolumePostfix"; ec="$?"
+    case "$ec" in
+        0)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'success' "Postfix files restored"
+            else
+                writeLog 'done'
+            fi
+            ;;
+        1)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '52' "There was an error restoring one or more postfix files."
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '52' "There was an error restoring one or more postfix files."
+            fi
+            ;;
+        2)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '51' "Cannot locate postfix backups!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '51' "Cannot locate postfix backups!"
+            fi
+            ;;
+    esac
+fi
+
+### restore rspamd
+if [ "$restoreRspamd" -eq 1 ]; then
+    if [ "$verbose" -eq 1 ]; then
+        writeLog 'info' "Restoring Rspamd files"
+    else
+        writeLog 'task' "Restoring Rspamd files"
+    fi
+
+    doRestore "${COMPOSE_PROJECT_NAME}_rspamd-vol-1" "$dockerVolumeRspamd"; ec="$?"
+    case "$ec" in
+        0)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'success' "Rspamd files restored"
+            else
+                writeLog 'done'
+            fi
+            ;;
+        1)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '52' "There was an error restoring one or more Rspamd files."
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '52' "There was an error restoring one or more Rspamd files."
+            fi
+            ;;
+        2)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '51' "Cannot locate Rspamd backups!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '51' "Cannot locate Rspamd backups!"
+            fi
+            ;;
+    esac
+fi
+
+### restore redis
+if [ "$restoreRedis" -eq 1 ]; then
+    if [ "$verbose" -eq 1 ]; then
+        writeLog 'info' "Restoring redis database"
+    else
+        writeLog 'task' "Restoring redis database"
+    fi
+
+    doRestore "${COMPOSE_PROJECT_NAME}_redis-vol-1" "$dockerVolumeRedis"; ec="$?"
+    case "$ec" in
+        0)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'success' "Redis database restored"
+            else
+                writeLog 'done'
+            fi
+            ;;
+        1)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '52' "There was an error restoring the redis database. This is usually *not* a serious issue."
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '52' "There was an error restoring the redis database. This is usually *not* a serious issue."
+            fi
+            ;;
+        2)
+            if [ "$verbose" -eq 1 ]; then
+                writeLog 'error' '51' "Cannot locate redis database backups!"
+            else
+                writeLog 'done' 'error'
+                writeLog 'error' '51' "Cannot locate redis database backups!"
+            fi
+            ;;
+    esac
+fi
+
+### restart mailcow
+writeLog 'task' "Starting mailcow"
+if ! docker-compose up -d > /dev/null 2>&1; then
+    writeLog 'done' 'warn'
+    writeLog 'warn' '21' "Unable to automatically start mailcow containers. Please attempt a manual start and note any errors."
+    warnCount=$((warnCount+1))
+fi
+writeLog 'done'
 
 ### exit gracefully
+if [ "$errorCount" -gt 0 ]; then
+    # note non-terminating errors
+    printf "%s[%s] --- %s execution completed with %s error(s) ---\n%s" "$err" "$(stamp)" "$scriptName" "$errorCount" "$norm" >> "$logfile"
+    exit 98
+elif [ "$warnCount" -gt 0 ]; then
+    printf "%s[%s] --- %s execution completed with %s warning(s) ---\n%s" "$yellow" "$(stamp)" "$scriptName" "$warnCount" "$norm" >> "$logfile"
+    exit 97
+else
     writeLog 'success' "All processes completed"
     printf "%s[%s] --- %s execution completed ---\n%s" "$magenta" "$(stamp)" "$scriptName" "$norm" >> "$logfile"
-# note non-terminating errors
-if [ "$errorCount" -gt 0 ]; then
-    printf "%s%s errors encountered!%s\n" "$err" "$errorCount" "$norm" >>"$logfile"
-fi
-# note warnings
-if [ "$warnCount" -gt 0 ]; then
-    printf "%s%s warnings issued!%s\n" "$yellow" "$warnCount" "$norm" >>"$logfile"
-fi
     exit 0
+fi
 
 ### error codes:
 # 1: parameter error
@@ -370,9 +633,14 @@ exit 0
 #     11: cannot locate SQL dump in backup directory
 #     12: cannot start mysql-mailcow container
 #     13: restoring SQL dump was unsuccessful
+# 2x: Docker/Docker-Compose errors
+#     20: cannot bring docker container(s) down successfully
+#     21: cannot bring docker container(s) up successfully
+# 5x: File restore errors
+#     51: cannot locate source files in backup directory
+#     52: error restoring one or more files
+# 97: script completed with 1 or more warnings
+# 98: script completed with 1 or more non-terminating errors
 # 99: TERM signal trapped
-# 100: could not change to mailcow-dockerized directory
-# 101: could not stop container(s)
-# 102: could not start container(s)
 
 #EOF

backup/xtraLocations.borg

@@ -49,12 +49,12 @@
 # include the paths to important configuration files/directories and/or
 # data directories
 
-# mailcow configuration (example)
+# mailcow configuration
-/opt/mailcow-dockerized/
+# already backed-up by the script by default based on mailcow.conf location
 
-# NGINX (example)
+# NGINX (if this host is a reverse proxy, for example)
 /etc/nginx/
 /usr/share/nginx/html/
 
-# LetsEncrypt (example)
+# LetsEncrypt
 /etc/letsencrypt/

etc/logwatch/scripts/services/backup

@@ -3,8 +3,8 @@
 #############################################################################
 # $Id$
 #############################################################################
-# Log: Backup script (backup)
+# Log: mailcow backup
-# Revision 1.0  2018/10/16
+# Revision 1.1  2019/07/20
 # Written by Asif Bacchus
 #############################################################################
 
@@ -34,7 +34,7 @@ if ($detailLevel == 0) {
         elsif ($ThisLine =~ /\-- \[WARNING\] /) {
             $summaryWarn++;
         }
-        elsif ($ThisLine =~ /All processes completed successfully/) {
+        elsif ($ThisLine =~ /All processes completed/) {
             $summarySuccess++;
         }
     }