Compare commits
8 Commits
34525daedc
...
88a77066a9
Author | SHA1 | Date |
---|---|---|
Asif Bacchus | 88a77066a9 | |
Asif Bacchus | 3926699007 | |
Asif Bacchus | e166086f91 | |
Asif Bacchus | 4211b2e29c | |
Asif Bacchus | 2c3aa9db64 | |
Asif Bacchus | d1aee2049b | |
Asif Bacchus | 305d571d46 | |
Asif Bacchus | 2e35e2ff1f |
208
README.md
208
README.md
|
@ -20,17 +20,35 @@ This script automates the following tasks:
|
|||
- Creates an clear, easy to parse log file so you can easily keep an eye on your
|
||||
backups and any errors/warnings
|
||||
|
||||
## Installation/copying
|
||||
|
||||
Once you've either cloned this git or downloaded the release file, simply copy
|
||||
the *'NCscripts'* folder to wherever you like. I suggest putting it in your
|
||||
*'/root'* directory since the root user must execute the script. If you edit
|
||||
the 503.html, nc_borg.details and nc_sql.details files in place, then you don't
|
||||
have to specify their locations when running the script.
|
||||
|
||||
Remember to make the script executable!
|
||||
|
||||
```Bash
|
||||
chmod +x backup.sh
|
||||
```
|
||||
|
||||
In addition, you can rename this script file to anything you like. The log file
|
||||
will use that same name by default when naming itself and any mention of this
|
||||
file in the logs will automatically use whatever name you choose to give it.
|
||||
|
||||
## Environment notes
|
||||
|
||||
The script is designed to be easy to use but still be flexible enough to
|
||||
accommodate a wide range of common NextCloud setups. I have tested it with
|
||||
accommodate a wide range of cn NextCloud setups. I have tested it with
|
||||
NextCloud 13 and 14 using a standard LEMP setup (Debian Stretch, NGINX, mariaDB
|
||||
& PHP7). The script accepts several parameters to provide it with the settings
|
||||
it requires to function. In addition, it reads external files for SQL and borg
|
||||
settings, so you don't have to weed through the script code to supply things
|
||||
like passwords.
|
||||
|
||||
## Why root?
|
||||
## Why this script must be run as root
|
||||
|
||||
This script must be run by the root user and will exit with an error if you try
|
||||
running it otherwise. This is because NextCloud's OCC command (needed to put
|
||||
|
@ -87,7 +105,7 @@ unavailable' while being backed up. A sample 503 page is included for you.
|
|||
|
||||
If you remove the default file or the one you specify is missing, a warning will
|
||||
be issued by the script but, it will continue executing. More details on the
|
||||
503 notification can be found later in the [503 functionality]() section of this
|
||||
503 notification can be found later in the [503 functionality](#503-functionality) section of this
|
||||
document. **Default: _scriptpath/503.html_**
|
||||
|
||||
#### -b _/path/to/filename.file_, path to borg details file
|
||||
|
@ -111,7 +129,7 @@ in the same directory as the script itself.
|
|||
|
||||
This is text file containing the details needed to connect to NextCloud's SQL
|
||||
database. For more information about the *required order* of entries can be
|
||||
found later in this document in the [sql details file]() section.
|
||||
found later in this document in the [sql details file](#sql-details-file) section.
|
||||
**Default: _scriptpath/nc_sql.details_**
|
||||
|
||||
#### -v, verbose output from borg
|
||||
|
@ -134,10 +152,10 @@ as your NextCloud webroot.
|
|||
This is used exclusively for 503 functionality since the script has to know
|
||||
where to copy the 503 file. If you don't want to use this functionality, you
|
||||
can omit this parameter and the script will issue a warning and move on. More
|
||||
details can be found in the [503 functionality]() section later in this
|
||||
details can be found in the [503 functionality](#503-functionality) section later in this
|
||||
document.
|
||||
|
||||
### Borg details file
|
||||
## Borg details file
|
||||
|
||||
This file contains all the data needed to access your borg remote data repo.
|
||||
Each line must contain specific information in a specific order or needs to be
|
||||
|
@ -154,7 +172,7 @@ order:
|
|||
7. purge timeframe options
|
||||
8. location of borg remote instance
|
||||
|
||||
#### Protect this file!
|
||||
### Protect your borg details file
|
||||
|
||||
This file contains information on how to access and decrypt your borg repo,
|
||||
therefore, you **must** protect it. You should lock it out to your root user.
|
||||
|
@ -168,13 +186,13 @@ chown root:root nc_borg.details
|
|||
chmod 600 nc_borg.details
|
||||
```
|
||||
|
||||
#### borg specific entries (lines 1-4)
|
||||
### borg specific entries (lines 1-4)
|
||||
|
||||
If you need help with these options, then you should consult the borg
|
||||
documentation or search my blog at
|
||||
[https://mytechiethoughts.com](https://mytechiethoughts.com) for borg.
|
||||
|
||||
#### additional files/directories to backup
|
||||
### additional files/directories to backup
|
||||
|
||||
This points to a plain-text file listing additional files and directories you'd
|
||||
like borg to include in the backup. The sample file, *'xtraLocations.borg'*
|
||||
|
@ -195,7 +213,7 @@ directory and the SQL dump. However, this is pretty unusual since you would not
|
|||
be including any configuration files, webserver configurations, etc. If you
|
||||
omit this line, the script will log a warning in your log.
|
||||
|
||||
#### exclusion patterns
|
||||
### exclusion patterns
|
||||
|
||||
This points to a plain-text file containing borg-specific patterns describing
|
||||
what files you'd like borg to ignore during the backup. The sample file,
|
||||
|
@ -204,7 +222,7 @@ standard NextCloud install -- the previews directory and the cache directory.
|
|||
You need to run *'borg help patterns'* for help on how to specify any additional
|
||||
exclusion patterns.
|
||||
|
||||
#### purge timeframe options
|
||||
### purge timeframe options
|
||||
|
||||
Here you can let borg purge know how you want to manage your backup history.
|
||||
Consult the borg documentation and then copy the relevant options directly into
|
||||
|
@ -219,14 +237,14 @@ This would tell borg prune to keep ALL backups made for any reason within the
|
|||
last 7 days, keep 30 days worth of daily backups, 12 weeks of end-of-week
|
||||
backups and then an infinite amount of end-of-month backups.
|
||||
|
||||
#### borg remote location
|
||||
### borg remote location
|
||||
|
||||
If you're using rsync, then just have this say *'borg1'*. If you are using
|
||||
another provider, you'll have to reference their locally installed copy of borg
|
||||
relative to your repo path. You can also leave this blank if your provider does
|
||||
not run borg locally but your backups/restores will be slower.
|
||||
|
||||
#### Examples:
|
||||
### Examples
|
||||
|
||||
All fields including pointers to additional files to backup, exclusion patterns
|
||||
and a remote borg path. Prune: keep all backups made in the last 14 days.
|
||||
|
@ -269,7 +287,7 @@ pAsSw0rD
|
|||
|
||||
```
|
||||
|
||||
### SQL details file
|
||||
## SQL details file
|
||||
|
||||
This file contains all the information needed to access your NextCloud SQL
|
||||
database in order to dump it's contents into a file that can be easily
|
||||
|
@ -291,7 +309,7 @@ pAsSwOrD
|
|||
nextcloudDB
|
||||
```
|
||||
|
||||
#### Protect this file!
|
||||
### Protect your sql details file
|
||||
|
||||
This file contains information on how to access your SQL installation therefore,
|
||||
you **must** protect it. You should lock it out to your root user. Putting it
|
||||
|
@ -304,3 +322,163 @@ chown root:root nc_sql.details
|
|||
# restrict access to root only
|
||||
chmod 600 nc_sql.details
|
||||
```
|
||||
|
||||
## 503 functionality
|
||||
|
||||
This script includes an entire section dedicated to copying an html file to act
|
||||
as an error 503 notification page. Error 503 is by definition "service
|
||||
temporarily unavailable" which is exactly the case for your NextCloud server
|
||||
during a backup since it is in maintenance mode and no logins are permitted.
|
||||
|
||||
The script copies whatever file is defined by the *'-5'* parameter (or the
|
||||
default located at *'scriptpath/503.html'*) to whatever path is defined as the
|
||||
'webroot' by the *'-w'* parameter. This means that if you omit the *'-w'*
|
||||
parameter, the script will necessarily skip this entire process and just issue a
|
||||
warning to let you know about it.
|
||||
|
||||
### Conditional forwarding by your webserver
|
||||
|
||||
The script copying the file to the webroot is the easy part. Your webserver has
|
||||
to look for the presence of that file and generate a 503 error in order for the
|
||||
magic to happen. To do that, you have to include an instruction to that effect
|
||||
in your default server definition and/or your NextCloud virtual server
|
||||
definition file depending on your setup.
|
||||
|
||||
#### NGINX
|
||||
|
||||
You can copy the following code into the relevant server definition(s) on an
|
||||
NGINX server:
|
||||
|
||||
```Perl
|
||||
server {
|
||||
...
|
||||
if (-f /usr/share/nginx/html/503.html) {
|
||||
return 503;
|
||||
}
|
||||
...
|
||||
error_page 530 @backup
|
||||
location @backup {
|
||||
root /usr/share/nginx/html;
|
||||
rewrite ^(.*)$ /503.html break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This tells NGINX that if it finds the file *'503.html'* at the path
|
||||
*'/usr/share/nginx/html'* (webroot) then return an error code 503. Next,
|
||||
rewrite any url to *'domain.tld/503.html'* and thus, display the custom 503
|
||||
error page. On the other hand, if it can't find 503.html at the path specified
|
||||
(i.e. the script has deleted it because the backup is completed), then go about
|
||||
business as usual.
|
||||
|
||||
#### Apache
|
||||
|
||||
I don't use apache for anything, ever... so I'm not sure how exactly you'd do
|
||||
this but I think you'd have to use something like:
|
||||
|
||||
```Perl
|
||||
RewriteEngine On
|
||||
RewriteCond %{ENV:REDIRECT_STATUS} !=503
|
||||
RewriteCond "/var/www/503.html" -f
|
||||
RewriteRule ^ - [R=503,L]
|
||||
...
|
||||
ErrorDocument 503 /503.html
|
||||
...
|
||||
```
|
||||
|
||||
Let me know if that works and I'll update this document accordingly. Like I
|
||||
said, I don't use Apache so I can't really test it very easily.
|
||||
|
||||
#### Disabling 503 functionality altogether
|
||||
|
||||
If you don't want to use the 503 functionality for whatever reason and don't
|
||||
want your log file junked up with warnings about it, then find the section of
|
||||
the script file that starts with *'--- Begin 503 section ---'* and either
|
||||
comment all the lines (put a *'#'* at the beginning of each line) or delete all
|
||||
the lines until you get to *'--- End 503 section ---'*.
|
||||
|
||||
## Scheduling: Cron
|
||||
|
||||
After running this script at least once manually to test your settings, you
|
||||
should schedule it to run automatically so things stay backed up. This is
|
||||
easiest with a simple cron job.
|
||||
|
||||
1. Open root's crontab:
|
||||
|
||||
```Bash
|
||||
sudo crontab -e
|
||||
```
|
||||
|
||||
2. Add your script command line and set the time. I'm assuming your script is
|
||||
located at *'/root/NCscripts'*, all files are at their default locations and
|
||||
you want to run your backup at 1:07am daily.
|
||||
|
||||
```Bash
|
||||
7 1 * * * /root/NCscripts/backup.sh -d /var/nc_data -n /usr/share/nginx/html/nextcloud -u www-data -l /var/log/backup.log -w /usr/share/nginx/html > /dev/null 2>&1
|
||||
```
|
||||
|
||||
The last part redirects all output to 'null' and forwards any errors to
|
||||
'null' also. You don't need output because the script creates a wonderfully
|
||||
detailed log file that you can review :-)
|
||||
3. Save the file and exit.
|
||||
4. Confirm by listing the root user's crontab:
|
||||
|
||||
```Bash
|
||||
sudo crontab -l
|
||||
```
|
||||
|
||||
## The log file
|
||||
|
||||
The script creates a very detailed log file of all major operations along with
|
||||
any errors and warnings. Everything is timestamped so you can see how long
|
||||
things take and when any errors tooks place. The script includes debugging
|
||||
notes such as where temp files are located, where it's looking for data, whether
|
||||
it created/moved/copied files, etc. All major operations are tagged *'-- [INFO]
|
||||
message here --'*. Similarily, warnings are tagged *'-- [WARNING] message here
|
||||
(code: xxxx) --'* and errors are tagged *'-- [ERROR] message here (code: xxx)
|
||||
--'*. Successful operations generate a *'-- [SUCCESS] message here --'* stamp.
|
||||
|
||||
This tagging makes it easy for you to set up a log screening program to make
|
||||
keeping an eye on your backup results easier. If you plan on using Logwatch
|
||||
(highly recommended, great program!) then I've done the work for you...
|
||||
|
||||
### Using Logwatch
|
||||
|
||||
Log-group, conf and service files are included so that you can easily setup
|
||||
Logwatch to monitor the script's log file and report at your desired detail
|
||||
level as follows:
|
||||
|
||||
1. 0: Summary of total success, warnings & errors only
|
||||
2. 1-4: Actual success, error & warning messages
|
||||
3. 5: Same as above, but includes info messages
|
||||
4. 6+: Dumps entire raw log file including debugging messages
|
||||
|
||||
A detailed breakdown of the files and all options are included in a separate
|
||||
readme in the *'/etc/logwatch'* folder of this git archive.
|
||||
|
||||
### Remember to rotate your logs
|
||||
|
||||
The log file generated by this script is fairly detailed so it can grow quite
|
||||
large over time. This is especially true if you are using verbose output from
|
||||
borg for any troubleshoot or for compliance/auditing. I've included a sample
|
||||
commented logrotate config file in this git archive at *'/etc/logrotate.d'*
|
||||
which you can modify and drop into that same directory on your Debian/Ubuntu
|
||||
system. If you are using another log rotating solution, then please remember to
|
||||
configure it so that your log files don't get overwhelmingly large should you
|
||||
need to parse them if something goes wrong with your backups.
|
||||
|
||||
## Final notes
|
||||
|
||||
I think that's everything. If I've forgotten to document something, please let
|
||||
me know. I know this readme is long but, I hate how much stuff for linux and
|
||||
open-source programs/scripts in general are so poorly documented especially for
|
||||
newbies and I didn't want to make that same mistake.
|
||||
|
||||
I don't script too often and I'm a horrible programmer, so if you see anything
|
||||
that can be/should be improved, please let me know or submit your changes! I
|
||||
love learning new ways of doing things and getting feedback, so suggestions and
|
||||
comments are more than welcome.
|
||||
|
||||
If this has helped you out, then please visit my blog at
|
||||
[https://mytechiethoughts.com](https://mytechiethoughts.com) where I solve
|
||||
problems like this all the time on a shoe-string or zero budget. Thanks!
|
Loading…
Reference in New Issue