Compare commits

..

19 Commits

Author SHA1 Message Date
Asif Bacchus 0fcab85da9 fix incorrect filename 2019-07-30 01:11:15 -06:00
Asif Bacchus ed62bb1e38 update logwatch integration 2019-07-30 00:44:37 -06:00
Asif Bacchus 3a759f39ac reworded redundant success msgs 2019-07-30 00:41:02 -06:00
Asif Bacchus 5cd023010c update wiki link, match details filename to script 2019-07-30 00:39:17 -06:00
Asif Bacchus 8e30cb4048 fix timestamp double-bracket 2019-07-26 03:16:55 -06:00
Asif Bacchus 8edd833544 fix final success msg output spacing err 2019-07-26 03:15:26 -06:00
Asif Bacchus dd49378678 check running as root 2019-07-26 03:14:12 -06:00
Asif Bacchus efe202d9a3 new directory structure 2019-07-26 03:11:53 -06:00
Asif Bacchus 9936cf7de3 add link to wiki 2019-07-20 18:14:14 -06:00
Asif Bacchus 0b1d220b5b update readme and license 2019-07-20 18:09:19 -06:00
Asif Bacchus a299cca365 move logwatch under etc dir to mirror real sys 2019-07-20 17:17:33 -06:00
Asif Bacchus 271f08d0e4 sample logrotate config 2019-07-20 17:16:58 -06:00
Asif Bacchus 55162342c7 add logwatch example files 2019-07-20 17:14:58 -06:00
Asif Bacchus fdb139afc9 change success msg for logwatch parsing 2019-07-20 17:14:44 -06:00
Asif Bacchus 93bfdbfeb3 enclose timestamps in square brackets 2019-07-20 17:06:20 -06:00
Asif Bacchus 3f1861fc01 update example default locations 2019-07-20 16:57:37 -06:00
Asif Bacchus 4e5bfa49b4 use dbnames from details file for dump 2019-06-20 03:39:57 -06:00
Asif Bacchus 6fec178fae check for null database names in details file 2019-06-20 03:37:17 -06:00
Asif Bacchus ca190c89bb rewrote help section 2019-06-19 23:08:07 -06:00
12 changed files with 340 additions and 48 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License MIT License
Copyright (c) <year> <copyright holders> Copyright (c) 2019 Asif Bacchus
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
+61 -5
View File
@@ -1,7 +1,63 @@
# seafileBackup # Backup a bare-metal Seafile installation using borgbackup <!-- omit in toc -->
Backup Seafile DB and data using borgbackup to rsync.net Simple POSIX-compliant script to backup a bare-metal Seafile installation using borgbackup to a remote SSH-capable storage system. I strongly suggest using rsync.net since they are very reliable, fast and even have special pricing for people using borgbackup/attic [(details here)](https://www.rsync.net/products/attic.html).
This readme will be an overiew of the purpose of the repo and BASIC usability **THIS SCRIPT USES BORG [(borgbackup)](https://borgbackup.readthedocs.io/en/stable/) SO PLEASE ENSURE THAT IS INSTALLED CORRECTLY BEFORE TRYING TO USE THIS SCRIPT!**
information. Detailed usage information will all be presented in the repo's
wiki. - [Basic features of this script:](#Basic-features-of-this-script)
- [Check out the wiki](#Check-out-the-wiki)
- [Overview](#Overview)
- [Installation](#Installation)
- [Running the script](#Running-the-script)
- [*seafbackup.sh*](#seafbackupsh)
- [Final thoughts](#Final-thoughts)
## Basic features of this script:
- Backup your entire Seafile program and data directories.
- Dump Seafile SQL databases to a temporary directory, add them to the backup and then remove the temporary directory.
- Easily add additional directories/files to the backup by modifying a plain-text file.
- Easily exclude directories/files from being backed up (i.e. thumbnails) by modifying a plain-text file.
- Optionally stop Seafile services during the backup and restart them after.
- Optionally copy a 503 error page to your webserver so users know your server is unavailable due to a backup being performed. The 503 file is removed when the backup is completed.
- Run 'borg prune' to trim old backups according to your schedule so you don't pay storage fees for super-old or un-needed backups.
- Create a clear, easy to parse log file so you can keep an eye on warnings/errors, etc.
- The generated log file can be easily parsed by tools like *logwatch* so you can integrate backup-monitoring with your existing system-monitoring solutions.
## Check out the wiki
This readme is an overview of the script. For detailed instructions on how the script works, what it does and it's various options, please [read the wiki](https://git.asifbacchus.app/asif/seafileBackup/wiki)! I've tried to annotate the files as best I could with many many comments, but seriously, reading the [wiki](https://git.asifbacchus.app/asif/seafileBackup/wiki) if you're stuck is the best approach since I try to go over everything step-by-step.
## Overview
This script has NO mandatory parameters. If no parameters are specified, it will proceed using defaults which normally align with the recommended Seafile installation defaults. That being said, it is impossible to determine defaults for things like your backup-server settings, so please make sure you customize the *seafbackup.details* file with appropriate details!
By default, the script will dump Seafile SQL databases and back them up along with your entire Seafile program and data directories. You can add additional files to be backed up by adding paths to *xtraLocations.borg*.
## Installation
Simply clone this repo to the desired location. I recommend somewhere simple like *ROOT's home directory* so any sensitive information is restricted to the root user by default.
```bash
cd /root
sudo git clone https://git.asifbacchus.app/asif/seafileBackup.git
```
## Running the script
```bash
./seafbackup.sh [parameters]
# display abbreviated help
./seafbackup.sh --help
```
## *seafbackup.sh*
Please open this file in a text editor and follow the comments to fill in the parameters with the correct values for your environment and your backup server.
## Final thoughts
I hope this makes backing up your Seafile server easier and removes one more mundane administrative task from your schedule :-) If you have any comments, questions, concerns, suggestions for improvements or notice any bugs, please file an issue in this repo!
For more scripts and solutions like this, check out my blog at [My Techie-Thoughts](https://mytechiethoughts.com). Thanks!
+15
View File
@@ -0,0 +1,15 @@
### Rotate backup log file
# location of log file (-l parameter of script file)
/path/to/backup.log {
# rotate log file weekly -- you could also use 'daily', 'monthly' or
# specify a size using 'size 100k', for example
weekly
# keep 4 weeks of old logs (4 files) and delete older ones
rotate 4
# compress old log files using gzip (default)
compress
}
@@ -0,0 +1,13 @@
# Location of your script's log file, -l parameter
LogFile = /path/to/your/seafbackup.log
# Format of logrotate archives for your script. Example assumes compression and
# extension preservation
Archive = /path/to/your/seafbackup.log.?.gz
# Apply the correct date/time filtering to match the format of the script's log
# We are using a custom pl script in /etc/logwatch/scripts/shared/ You don't
# need to change this unless you have altered the 'stamp' function in the backup
# script in which case you will want to update the regex in the custom pl script
# below
*sqFullStampAnywhere
@@ -0,0 +1,9 @@
# Name of the logfile group without any extension
LogFile = seafbackup
# Heading displayed on Logwatch's report for this service
Title = "System and Seafile Backup"
# Override the detail level for this service
# Remember the levels are: 0, 1-4, 5, 6+
# Detail = 0
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/perl
#############################################################################
# $Id$
#############################################################################
# Log: Seafile backup script (seafbackup, Logwatch group 'backup')
# Revision 1.1 2019/07/20
# Written by Asif Bacchus
#############################################################################
use strict;
### Get Logwatch detail level (default to 0)
my $detailLevel = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;
### Declare variables
my $summaryErr;
my $summaryWarn;
my $summarySuccess;
my %reportHash = ();
my $key;
### Minimal detail level: provide summary data only
if ($detailLevel == 0) {
### process logfile and summarize message types
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
$summaryErr++;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
$summaryWarn++;
}
elsif ($ThisLine =~ /All processes completed/) {
$summarySuccess++;
}
}
### fill hash table with headings and summary counts
if ($summarySuccess > 0) {
$reportHash{"All processes successfully completed"} = $summarySuccess;
}
if ($summaryWarn > 0) {
$reportHash{"Warnings issued"} = $summaryWarn;
}
if ($summaryErr > 0) {
$reportHash{"Errors encountered"} = $summaryErr;
}
### print hash table
foreach $key (sort keys %reportHash) {
print "$key: $reportHash{$key}\n";
}
}
### Levels 1-4 provide the actual error, warning and success messages instead
### of a summary count
elsif ($detailLevel >= 1 && $detailLevel <= 4) {
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[SUCCESS\] /) {
print $ThisLine;
}
}
}
### Level 5 is similiar to levels 1-4 except it also reports informational
### messages such as the location of script created files, variable checks,
### etc. This is useful when verifying the script's operation.
elsif ($detailLevel == 5) {
while (defined(my $ThisLine = <STDIN>)) {
if ($ThisLine =~ /\-- \[ERROR\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[WARNING\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[SUCCESS\] /) {
print $ThisLine;
}
elsif ($ThisLine =~ /\-- \[INFO\] /) {
print $ThisLine;
}
}
}
### Any level above 5 will echo the entire log including the debugging notes
### within the script meant for troubleshooting. Using this level of detail
### should only be done if you cannot view the actual log file directly for
### whatever reason. The actual log file is colour-coded for easier debugging.
elsif ($detailLevel > 5) {
while (defined(my $ThisLine = <STDIN>)) {
print $ThisLine;
}
}
### Exit gracefully
exit (0);
# vi: shiftwidth=3 tabstop=3 et
# Local Variables:
# mode: perl
# perl-indent-level: 3
# indent-tabs-mode: nil
# End:
@@ -0,0 +1,37 @@
#!/usr/bin/perl
##########################################################################
# $Id$
##########################################################################
###############################################################################
## Filter dates in full-date-time international format, surrounded by square
## brackets located anywhere on a given line
## Format: '[%Y-%m-%d %H:%M:%S]'
###############################################################################
use Logwatch ':dates';
my $Debug = $ENV{'LOGWATCH_DEBUG'} || 0;
### Specify the format of the date/time stamp itself
$SearchDate = TimeFilter('%Y-%m-%d %H:%M:%S');
if ( $Debug > 5 ) {
print STDERR "DEBUG: Inside FullDateTime...\n";
print STDERR "DEBUG: Looking For: " . $SearchDate . "\n";
}
while (defined($ThisLine = <STDIN>)) {
### specify the regex that defines how to find 'SearchDate'
if ($ThisLine =~ m/\[$SearchDate\] /o) {
print $ThisLine;
}
}
# vi: shiftwidth=3 syntax=perl tabstop=3 et
# Local Variables:
# mode: perl
# perl-indent-level: 3
# indent-tabs-mode: nil
# End:
@@ -62,13 +62,13 @@ borgRepoPassphrase="p@ssW0rd"
# directories to include in your backup along with Seafile data # directories to include in your backup along with Seafile data
# see repo wiki for more details # see repo wiki for more details
# leave blank if you only want to backup Seafile related files/directories. # leave blank if you only want to backup Seafile related files/directories.
borgXtraListPath="/root/seafBackup/xtraLocations.borg" borgXtraListPath="/root/seafileBackup/xtraLocations.borg"
# OPTIONAL: path to file containing files/directories or 'patterns' to be # OPTIONAL: path to file containing files/directories or 'patterns' to be
# excluded in a BORG RECOGNIZED format # excluded in a BORG RECOGNIZED format
# see repo wiki for more details or consult borg documentation # see repo wiki for more details or consult borg documentation
# leave blank for no exclusions. # leave blank for no exclusions.
borgExcludeListPath="/root/seafBackup/excludeLocations.borg" borgExcludeListPath="/root/seafileBackup/excludeLocations.borg"
# parameters to determine how borg deletes aged backups # parameters to determine how borg deletes aged backups
# more details in the repo wiki and/or borg documentation # more details in the repo wiki and/or borg documentation
+89 -39
View File
@@ -96,40 +96,78 @@ exitError () {
# display script help information # display script help information
scriptHelp () { scriptHelp () {
printf "\n" printf "\n${cyan}%s\n" "--------------------------------------------------------------------------------"
printf "${magenta}%s usage instructions:${norm}\n\n" "$scriptName"
printf "This script performs a backup of your Seafile system assuming a fairly standard set up as per the documentation. Full details about this script can be found in the repo wiki at ${yellow}https://git.asifbacchus.app/asif/seafileBackup/wiki${norm}\n\n" printf "Usage:${norm} %s [parameters]\n\n" "$scriptName"
printf "${magenta}Usage:${norm} %s ${cyan}[parameters]${norm}\n\n" \ printf "${cyan}Parameters ${yellow}(default value):${norm}\n"
"$scriptName" printf "There are NO mandatory parameters. If a parameter is not "
printf "${magenta}Parameters (default in brackets):${norm}\n" printf "supplied, it's\n"
printf "There are NO mandatory parameters. If a parameter is not supplied, it's default value will be used. In the case of a switch parameter, it will be remain deactivated if not specified.\n\n" printf "default value will be used. In the case of a switch parameter, "
printf "${yellow}script related\n" printf "it will\n"
printf "${cyan}-c, --config, --details (scriptPath/seafbackup.details)\n" printf "remain deactivated if not specified.\n\n"
printf "${norm}Path to the '.details' configuration key/value pair file for this script.\n"
printf "${cyan}-h, -?, --help\n" printf "${magenta}script related\n"
printf "${norm}This help screen.\n" printf "${cyan}-c, --config, --details${norm}\n"
printf "${cyan}-l, --log (scriptPath/scriptName.log)\n" printf "Path to the configuration key/value pair file for this script.\n"
printf "${norm}Path where the log file for this script should be written.\n" printf "${yellow}(scriptPath/seafbackup.details)${norm}\n\n"
printf "${cyan}-v, --verbose (normal output, option is OFF)\n" printf "${cyan}-h, -?, --help${norm}\n"
printf "${norm}Log borg output with increased verbosity including listing EVERY file processed. Use this sparingly since your log file can get very large, very quickly! This is a switch value, specifying it will turn the option ON.\n\n" printf "This help screen.\n\n"
printf "${yellow}503 functionality\n" printf "${cyan}-l, --log${norm}\n"
printf "${cyan}-5, --use-503 (do NOT copy, option is OFF)\n" printf "Path to write log file.\n"
printf "${norm}Copy an 'error 503' page to your webroot. Your webserver can then scan for that file and display it (details in wiki for how to set that up). This is a switch value, specifying it will instruct the script to copy the error page.\n" printf "${yellow}(scriptPath/scriptName.log)${norm}\n\n"
printf "${cyan}--503-path (scriptPath/503_backup.html)\n" printf "${cyan}-v, --verbose${norm}\n"
printf "${norm}Path to the file you want copied to your webroot as the 'error 503' page during backup operations.\n" printf "Log borg output with increased verbosity (list all files). "
printf "${cyan}-w, --webroot (/usr/share/nginx/html)\n" printf "Careful! Your\n"
printf "${norm}Path to your webroot or wherever you want the 'error 503' file copied during backups.\n\n" printf "log file can get large very quickly! This is a switch value, "
printf "${yellow}seafile related\n" printf "specifying it\n"
printf "${cyan}-d, --data, --datadir, --seafdata (/var/seafile)\n" printf "will turn the option ON.\n"
printf "${norm}Path to your seafile DATA base/root directory. All your seafile data exists under this path.\n" printf "${yellow}(normal output, option is OFF)${norm}\n\n"
printf "${cyan}-o, --offline (leave seafile running, OFF)\n"
printf "${norm}If you want this script to stop your seafile systemd services and start them up again after the backup, then specify this option. This is a switch value.\n" printf "${magenta}503 functionality\n"
printf "${cyan}-p, --seaf, --seafdir (/opt/seafile)\n" printf "${cyan}-5, --use-503${norm}\n"
printf "${norm}Path to your seafile PROGRAM base/root directory. All your seafile configuration and program files exist under this path.\n" printf "Copy an 'error 503' page to your webroot for your webserver to "
printf "${cyan}--seafile-service (seafile.service)\n" printf "find. This is a\n"
printf "${norm}The name of your seafile systemd service. Only needed if you want the script to perform an offline backup and you are not using the default service name.\n" printf "switch value, specifying it will instruct the script to copy the "
printf "${cyan}--seahub-service (seahub.service)\n" printf "error page.\n"
printf "${norm}The name of your seahub (web) systemd service. Only needed if you want the script to perform an offline backup and you are not using the default service name.\n\n" printf "${yellow}(do NOT copy, option is OFF)${norm}\n\n"
printf "${cyan}--503-path${norm}\n"
printf "Path to the file you want copied to your webroot as the 'error "
printf "503' page during\n"
printf "backup operations.\n"
printf "${yellow}(scriptPath/503_backup.html)${norm}\n\n"
printf "${cyan}-w, --webroot${norm}\n"
printf "Path to where the 'error 503' file should be copied during "
printf "backups.\n"
printf "${yellow}(/usr/share/nginx/html)${norm}\n\n"
printf "${magenta}seafile related\n"
printf "${cyan}-d, --data, --datadir, --seafdata${norm}\n"
printf "Path to your seafile DATA base/root directory."
printf "${yellow}(/var/seafile)${norm}\n\n"
printf "${cyan}-o, --offline${norm}\n"
printf "Stop your seafile systemd services and start them up again after "
printf "the backup.\n"
printf "This is a switch value.\n"
printf "${yellow}(leave seafile running, OFF)${norm}\n\n"
printf "${cyan}-p, --seaf, --seafdir${norm}\n"
printf "${norm}Path to your seafile PROGRAM base/root directory.\n"
printf "${yellow}(/opt/seafile)${norm}\n\n"
printf "${cyan}--seafile-service${norm}\n"
printf "The name of your seafile systemd service. Use with '--offline' "
printf "when using a\n"
printf "non-default service name.\n"
printf "${yellow}(seafile.service)${norm}\n\n"
printf "${cyan}--seahub-service${norm}\n"
printf "The name of your seahub (web) systemd service. Use with "
printf "'--offline' when using\n"
printf "a non-default service name.\n"
printf "${yellow}(seahub.service)${norm}\n\n"
printf "More details and examples of script usage can be found in the "
printf "repo wiki at\n"
printf "${yellow}https://git.asifbacchus.app/BacchusGroup/seafileBackup"
printf "/wiki${norm}.\n"
printf "${cyan}%s${norm}\n\n" "--------------------------------------------------------------------------------"
} }
# control seafile services (systemd) # control seafile services (systemd)
@@ -191,7 +229,7 @@ scriptPath="$( CDPATH='' cd -- "$( dirname -- "$0" )" && pwd -P )"
scriptName="$( basename "$0" )" scriptName="$( basename "$0" )"
logFile="$scriptPath/${scriptName%.*}.log" logFile="$scriptPath/${scriptName%.*}.log"
warnCount=0 warnCount=0
configDetails="$scriptPath/seafbackup.details" configDetails="$scriptPath/${scriptName%.*}.details"
err503Copied=0 err503Copied=0
sqlCopied=0 sqlCopied=0
includeXtra=0 includeXtra=0
@@ -349,6 +387,11 @@ done
### check pre-requisites and default values ### check pre-requisites and default values
# check if running as root, otherwise exit
if [ $( id -u ) -ne 0 ]; then
printf "\n${err}ERROR: script MUST be run as ROOT${norm}\n\n"
exit 2
fi
# does the details file exist? # does the details file exist?
if [ ! -f "$configDetails" ]; then if [ ! -f "$configDetails" ]; then
badParam dne "(--details default)" "$configDetails" badParam dne "(--details default)" "$configDetails"
@@ -452,6 +495,11 @@ fi
if [ -z "${sqlPass}" ]; then if [ -z "${sqlPass}" ]; then
badDetails empty 'sqlPass' badDetails empty 'sqlPass'
fi fi
# database names
if [ -z "${ccnetDB_name}" ] || [ -z "${seafileDB_name}" ] || \
[ -z "${seahubDB_name}" ]; then
badDetails empty 'one or more mysql database names'
fi
## create tmp directory and generate dumpfile names ## create tmp directory and generate dumpfile names
printf "${cyan}[%s] -- [INFO] Dumping SQL databases --${norm}\n" \ printf "${cyan}[%s] -- [INFO] Dumping SQL databases --${norm}\n" \
@@ -473,20 +521,22 @@ sqlDump_seahub="backup-$(date +%Y%m%d_%H%M%S)_${seahubDB_name}.sql"
## dump databases ## dump databases
# dump CCNET-DB # dump CCNET-DB
if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \ if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \
--opt ccnet-db > "${sqlDumpDir}/${sqlDump_ccnet}" 2>> "$logFile"; then --opt "${ccnetDB_name}" > "${sqlDumpDir}/${sqlDump_ccnet}" \
2>> "$logFile"; then
exitError 115 "Could not dump ${ccnetDB_name} database" exitError 115 "Could not dump ${ccnetDB_name} database"
fi fi
# dump SEAFILE-DB # dump SEAFILE-DB
if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \ if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \
--opt ccnet-db > "${sqlDumpDir}/${sqlDump_seafile}" 2>> "$logFile"; then --opt "${seafileDB_name}" > "${sqlDumpDir}/${sqlDump_seafile}" \
2>> "$logFile"; then
exitError 116 "Could not dump ${seafileDB_name} database" exitError 116 "Could not dump ${seafileDB_name} database"
fi fi
# dump CCNET-DB # dump CCNET-DB
if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \ if ! mysqldump -h"${sqlServer}" -u"${sqlUser}" -p"${sqlPass}" \
--opt ccnet-db > "${sqlDumpDir}/${sqlDump_seahub}" 2>> "$logFile"; then --opt "${seahubDB_name}" > "${sqlDumpDir}/${sqlDump_seahub}" 2>> "$logFile"; then
exitError 117 "Could not dump ${seahubDB_name} database" exitError 117 "Could not dump ${seahubDB_name} database"
fi fi
printf "${ok}[%s] -- [SUCCESS] SQL databases dumped successfully --${norm}\n" \ printf "${ok}[%s] -- [SUCCESS] Dumped SQL databases --${norm}\n" \
"$(stamp)" >> "$logFile" "$(stamp)" >> "$logFile"
@@ -33,7 +33,7 @@
# root user files # root user files
/root/.bashrc /root/.bashrc
/root/.ssh/ /root/.ssh/
/root/SeafBackup/ /root/seafileBackup/
### configuration files for programs related to Seafile ### configuration files for programs related to Seafile