add restore documentation
Asif Bacchus
parent
b035a2f045
commit
8e0cbe307c
38
8.0-Restore-overview.md
Normal file
38
8.0-Restore-overview.md
Normal file
@ -0,0 +1,38 @@
|
||||
# Restoring Backups
|
||||
|
||||
Restoring mailcow backups made by the backup script in this git repo is very much like doing a migration between servers. The notable exception is that the SQL database is restored from a dump file instead of straight-copied. The majority of these instructions will discuss a semi-automated restore using the new `restore.sh` script. If you want to do a fully manual restore, please refer to [page 8.3.1](https://git.asifbacchus.app/asif/MailcowBackup/wiki/8.3.1-Restore-manual-restore).
|
||||
|
||||
## Overview
|
||||
|
||||
First off, **please remember to run everything as root!** This will allow borg to restore files with proper permissions, ensure that docker doesn’t run into any problems and generally make your life easier.
|
||||
|
||||
The basic restore process is as follows:
|
||||
|
||||
1. Download (‘extract’) the borg backup archive you want to restore from your remote repository to a local directory.
|
||||
|
||||
2. Basic first-install of mailcow:
|
||||
<ol type="a">
|
||||
<li>Download mailcow git repo</li>
|
||||
<li>Run `generate.sh` to create configuration, CA list and snake-oil certificates</li>
|
||||
</ol>
|
||||
|
||||
3. Copy configuration files from restored backup to mailcow directory (i.e. */opt/mailcow-dockerized*):
|
||||
|
||||
- mailcow.conf
|
||||
- customization files (usually in *data/conf/…*)
|
||||
- certificates
|
||||
- etc.
|
||||
|
||||
4. Start mailcow normally once to generate default configurations and volumes.
|
||||
|
||||
5. Browse to mailcow admin panel and/or SOGo (front page) to verify everything is working with the defaults and the install is working before data restore.
|
||||
|
||||
6. Run `restore.sh` script to restore data.
|
||||
|
||||
If this overview is enough for you to figure things out, then you can get started. Remember, you can run the restore script with the `--help` parameter for a quick overview on how to use it:
|
||||
|
||||
```bash
|
||||
./restore.sh --help
|
||||
```
|
||||
|
||||
If you need a little more guidance, the following pages will go through each step.
|
||||
64
8.1-Restore-extract.md
Normal file
64
8.1-Restore-extract.md
Normal file
@ -0,0 +1,64 @@
|
||||
# Restoring Backups - extract borg backup
|
||||
|
||||
The first step in restoring your data is extracting it from your borg archive that the backup script created for you. Let’s go through that process.
|
||||
|
||||
> Reminder: **Run your entire restore process as ROOT!**
|
||||
|
||||
First, why are we extracting the data locally instead of pulling it directly from our remote server? Well, a few reasons:
|
||||
|
||||
- Downloading it locally does not require any additional programs to be installed. The restore script uses *tar* which is installed on pretty much every Linux distro by default. Restoring directly from a remote repo would require something like *rsync* which is an excellent tool but I wanted to focus on staying as POSIX-compliant as possible. If you’d like to see *rsync* support, please file an issue requesting it and I’ll add it based on interest.
|
||||
- Restoring data doesn’t always go smoothly the first time around. By having a local copy, you do not have to re-run lengthy and potentially costly downloads multiple times. Remember, many backup storage services charge you for downloading your data!
|
||||
- Of course, the downside is that you require twice the storage space available - space for the downloaded backup and space to copy it to mailcow.
|
||||
|
||||
## Set up borg
|
||||
|
||||
Obviously, you need to have borg installed and functional before going any further. To access your remote borg repository, you’ll probably also need your SSH key and likely your borg repository key (unless you set up a *repokey* type repository in which case your key is stored in the repo itself and you only need your password). Once you have those things, you need to set some environment variables to get borg to work.
|
||||
|
||||
If you have access to the *backup.details* file used by *backup.sh* from your previous set up then this step is really easy. Just ‘source’ the file to import the environment variables into your working environment. I am assuming your borg set up is the same as your old system and **you are already running as root**.
|
||||
|
||||
```bash
|
||||
# assuming the old backup.details file is in your home directory
|
||||
source /root/backup.details
|
||||
```
|
||||
|
||||
If you don’t have that file, we can still do things manually.
|
||||
|
||||
```bash
|
||||
# assuming your borg base directory is /var/borgbackup
|
||||
export BORG_BASE_DIR='/var/borgbackup'
|
||||
export BORG_REMOTE_PATH=borg1
|
||||
export BORG_RSH='ssh -i /path/to/your/ssh.key'
|
||||
export BORG_REPO='user@remote.server.tld:repoName'
|
||||
export BORG_PASSPHRASE='repository_password'
|
||||
# if you are using a keyfile style repo, export this variable also
|
||||
export BORG_KEY_FILE='/path/to/borg/key.file'
|
||||
```
|
||||
|
||||
## Extract backup archive
|
||||
|
||||
Let’s see what our options are first – get a list of available archives in our borg repository:
|
||||
|
||||
```bash
|
||||
borg list
|
||||
```
|
||||
|
||||
If you get any errors running this command, double-check your environment variables in the previous step and reset them as necessary.
|
||||
|
||||
Each of your archives should be listed in the first column with their names being numerical date stamps. Review your options and choose the one you want to restore. Remember that borg extracts archives to the current directory, so we’ll make a directory first and switch to it.
|
||||
|
||||
```bash
|
||||
# create directory for our files and switch to it
|
||||
mkdir /var/mcRestore
|
||||
cd /var/mcRestore
|
||||
|
||||
# extract borg archive from repository
|
||||
borg extract --list ::archiveName
|
||||
```
|
||||
|
||||
I’m using the `--list` parameter to output each file name to the console. This is optional but I like seeing what’s going on.
|
||||
|
||||
> N.B. If you do *not* run as root, borg **cannot** restore file permissions and ownership! If you run as a regular user, you’ll notice that all your files are restored with that user as the owner. This will **not** work since mailcow requires particular files to have particular owners within its various containers.
|
||||
>
|
||||
> Bottom line: **RUN BORG EXTRACT AS ROOT!**
|
||||
|
||||
Assuming the extraction is successful, you’re ready for the next step.
|
||||
56
8.2-Restore-install-mailcow.md
Normal file
56
8.2-Restore-install-mailcow.md
Normal file
@ -0,0 +1,56 @@
|
||||
# Restoring Backups - install mailcow
|
||||
|
||||
We need to do a basic vanilla installation of mailcow so that we have something to restore data into. Follow the [official mailcow documentation](https://mailcow.github.io/mailcow-dockerized-docs/i_u_m_install/) but **do not** start mailcow yet!
|
||||
|
||||
> N.B. DO NOT start mailcow via `docker-compose up -d` just yet! We need to do some stuff before. You can, however, feel free to pull the containers.
|
||||
|
||||
> Remember to run *generate_config.sh* so that snake-oil certificates and the CA file is created. This is important regardless of whether or not you are using your own certificates. Since we will be using our old mailcow.conf, it doesn’t really matter what options you choose when running the generate script.
|
||||
|
||||
## Copy old configuration
|
||||
|
||||
Before we start mailcow, we need to copy our old configuration files so that everything is created using the correct *project name*, *hostname*, etc. This is simple since we have our backup already extracted.
|
||||
|
||||
### mailcow.conf
|
||||
|
||||
When you ran *generate_config.sh* a mailcow.conf file was created. Since we don’t need that file, you can delete it. Next, we’ll copy our old configuration from our backup:
|
||||
|
||||
```bash
|
||||
cd /opt/mailcow-dockerized
|
||||
rm mailcow.conf
|
||||
cp /var/mcRestore/opt/mailcow-dockerized/mailcow.conf ./
|
||||
```
|
||||
|
||||
## Customizations and certificates
|
||||
|
||||
If you have any custom postfix/unbound settings, certificates and docker-compose overrides, we need to copy those so that the environment is consistent with our backup. I’ll provide a few examples here, but ultimately only you know your own particular setup. Alternatively, if you have no idea what I’m talking about, you can likely skip this section entirely :stuck_out_tongue_winking_eye:
|
||||
|
||||
```bash
|
||||
# assuming you are in the mailcow home directory (e.g. /opt/mailcow-dockerized)
|
||||
|
||||
# copy certificates
|
||||
mkdir /certs
|
||||
chown root:root /certs
|
||||
chmod 700 /certs
|
||||
cp /var/mcRestore/certs/* /certs/
|
||||
|
||||
# copy postfix extra.cf
|
||||
cp /var/mcRestore/opt/mailcow-dockerized/data/conf/postfix/extra.cf ./data/conf/postfix/
|
||||
|
||||
# copy docker-compose override
|
||||
cp /var/mcRestore/opt/mailcow-dockerized/docker-compose.override.yml ./
|
||||
```
|
||||
|
||||
## Start mailcow
|
||||
|
||||
Now that our configuration is copied, we can start mailcow. We need to do this for two very important reasons:
|
||||
|
||||
1. Ensure that mailcow runs properly with our restored settings but without any data; and
|
||||
2. Create empty default volumes for us to copy our data into!
|
||||
|
||||
```bash
|
||||
docker-compose up -d && docker-compose logs -f
|
||||
```
|
||||
|
||||
Quickly check the logs for errors, there shouldn’t be any. When you are done reviewing your real-time logs, press `ctrl-c` to exit. Now, open a browser and verify that you can get to the mailcow admin page and, optionally, the SOGo homepage. If you copied certificates, it should be using those assuming you set them up properly. We only need to test front pages, it is **not** necessary to login or anything.
|
||||
|
||||
Ok, we’re ready to restore our data!
|
||||
103
8.3-Restore-restoring-data.md
Normal file
103
8.3-Restore-restoring-data.md
Normal file
@ -0,0 +1,103 @@
|
||||
# Restoring Backups - restore data
|
||||
|
||||
Before restoring data, please ensure you have completed the previous steps. Specifically, ensure that mailcow has been started at least once to create volumes otherwise the restore script will exit with an error. Assuming we’re all good with the previous steps, let’s finish the restore.
|
||||
|
||||
## The restore script
|
||||
|
||||
The restore script has sane defaults and only requires one argument, `--backup-location`, which is the location of your extracted borg archive on the local system. You can run the script with the `--help` parameter for a quick overview of all options but, let’s go over them in slightly more detail here:
|
||||
|
||||
| parameter | description | default | required? |
|
||||
| :------------------------ | ------------------------------------------------------------ | ------------------------------------------ | :-------: |
|
||||
| -b<br />--backup-location | Directory containing your extracted borg backup archive. | none | YES |
|
||||
| -l<br />--log | Path to write log file.<br />If you specify a filename, it will be created in the current directory.<br />If you specify a path ending with a ‘/’, the log file will be created using the default filename in that directory.<br />If you specify a full path, the log file will be saved as such.<br />In any case, the script will attempt to create any paths as needed. | scriptPath/scriptName.log | NO |
|
||||
| -v<br />--verbose | Enable verbose logging. This will output the paths of all processed (restored) files in the log file. *Warning:* This may lead to a large log file. | false | NO |
|
||||
| --skip-mail | Skip restoring email and encryption key. | false | NO |
|
||||
| --skip-sql | Skip restoring mailcow settings database (mySQL). | false | NO |
|
||||
| --skip-postfix | Skip restoring postfix settings/configuration files. | false | NO |
|
||||
| --skip-rspamd | Skip restoring rspamd settings, configuration and history. | false | NO |
|
||||
| --skip-redis | Skip restoring the redis database. | false | NO |
|
||||
| -d<br />--docker-compose | FULL path to the *docker-compose.yml* file used by mailcow. | /opt/mailcow-dockerized/docker-compose.yml | NO |
|
||||
| -m<br />--mailcow-config | FULL path to mailcow’s configuration file (*mailcow.conf*). This path is also assumed to be the main mailcow directory. | /opt/mailcow-dockerized/mailcow.conf | NO |
|
||||
| -t1<br />--timeout-start | Number of seconds to wait for docker-compose to bring the mailcow project up before throwing an error. | 180 | NO |
|
||||
| -t2<br />--timeout-stop | Number of seconds to wait for docker-compose to bring the mailcow project down before throwing an error. | 120 | NO |
|
||||
| -?<br />-h<br />--help | Show in-script help (basically this table). | n/a | n/a |
|
||||
|
||||
### Restore data
|
||||
|
||||
Based on the information above, we can restore our data easily as follows:
|
||||
|
||||
```bash
|
||||
# run with verbose logging
|
||||
./restore.sh --backup-location /var/mcRestore --log /var/log/mailcow-restore.log --verbose
|
||||
|
||||
# run with standard logging
|
||||
./restore.sh --backup-location /var/mcRestore --log /var/log/mailcow-restore.log
|
||||
```
|
||||
|
||||
In this case, I have specified our example backup location and have told the script where to place the log file. In the first example, I’ve requested verbose output so my log file will show each file that was restored and whether or not there is a problem. The verbose option is very useful when testing restore operations but, for a production server you could end up listing hundreds of thousands of mail files in your log which could be overwhelming and not very useful.
|
||||
|
||||
That’s it. The script will restore the SQL database then shut down mailcow and copy all the data to the proper volumes before restarting mailcow.
|
||||
|
||||
## Post-restore steps
|
||||
|
||||
As with any restore operation, you should review the log to verify no errors or warnings were issued and you should verify that mailcow restarted properly. Assuming everything worked, your mailcow is fully restored!
|
||||
|
||||
> N.B. For some reason, the Rspamd console *password* is *not* restored. All Rspamd data and history is restored, however, you must manually reset the password for Rspamd via your mailcow admin console.
|
||||
>
|
||||
> If someone figures out why this is, I’d love to know!
|
||||
|
||||
### Clean-up
|
||||
|
||||
After you’ve verified your backup was successfully restored, you can remove your extracted borg archive:
|
||||
|
||||
```bash
|
||||
rm -rf /var/mcRestore
|
||||
```
|
||||
|
||||
## More script examples
|
||||
|
||||
- Default log filename and location (same directory as *restore.sh* and named *restore.log*).
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore
|
||||
```
|
||||
|
||||
Note that if you rename *restore.sh*, the default name for the logfile will change accordingly. For example, if the restore script is renamed to *mailcow-restore.sh*, the default logfile will be named *mailcow-restore.log*.
|
||||
|
||||
- Only restore email, encryption key and mailcow database.
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore -l /var/log/mailcow-restore.log --skip-postfix --skip-rspamd --skip-redis
|
||||
```
|
||||
|
||||
- Restore settings database and settings for all containers, skip email and encryption key. This might be useful if you want a clean-start on a new server with the same email addresses, configuration etc. but no old email.
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore -l /logs/mailcow-restore.log --skip-mail
|
||||
```
|
||||
|
||||
- A very common situation is restoring email, encryption key, mailcow settings and spam database.
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore -l /root/mc_restore.log --skip-postfix --skip-redis
|
||||
```
|
||||
|
||||
- Customized docker-compose file (this does *not* affect *override* file processing).
|
||||
|
||||
- ```bash
|
||||
./restore.sh -b /var/mcRestore -l /var/log/mc_restore.log --docker-compose /opt/mailcow-dockerized/myMailcow.yml
|
||||
```
|
||||
|
||||
- Non-standard mailcow location.
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore -l /var/log/mc_restore.log --docker-compose /usr/local/mailcow/docker-compose.yml --mailcow-config /usr/local/mailcow/mailcow.conf
|
||||
```
|
||||
|
||||
- On some systems, docker-compose may need more time to start and/or stop. I’ll show setting both timeouts at the same time, but you can specify only one or the other as needed.
|
||||
|
||||
```bash
|
||||
./restore.sh -b /var/mcRestore -l /logs/mcRestore.log --timeout-start 90 --timeout-stop 300
|
||||
```
|
||||
|
||||
|
||||
73
8.3.1-Restore-manual-restore.md
Normal file
73
8.3.1-Restore-manual-restore.md
Normal file
@ -0,0 +1,73 @@
|
||||
# Restoring Backups - manual restore
|
||||
|
||||
If you don’t want to use the restore script or cannot use it due to a non-standard set up, here is how you can manually restore your mailcow from the extracted borg archive.
|
||||
|
||||
First off, you still have to do all the steps as outlined on pages [8.1](https://git.asifbacchus.app/asif/MailcowBackup/wiki/8.1-Restore-extract) and [8.2](https://git.asifbacchus.app/asif/MailcowBackup/wiki/8.2-Restore-install-mailcow).
|
||||
|
||||
1. Switch to your mailcow directory (I will assume the default location for this example) and START MAILCOW. Then ‘source’ your *mailcow.conf* file so you can use environment variables in the subsequent commands. Wait a minute or two before proceeding to the next step so that mailcow has time to start properly.
|
||||
|
||||
```bash
|
||||
cd /opt/mailcow-dockerized
|
||||
docker-compose up -d
|
||||
source mailcow.conf
|
||||
```
|
||||
|
||||
2. Let’s restore the mailcow database first. Remember to change the location of your extracted backup as necessary. You can skip this step if you are not restoring the mailcow settings database.
|
||||
|
||||
```bash
|
||||
SQLBACKUP=$(find /var/mcRestore/tmp -iname "*.sql")
|
||||
docker exec -i "$(docker-compose ps -q mysql-mailcow)" mysql -u${DBUSER} -p${DBPASS} ${DBNAME} < "${SQLBACKUP}"
|
||||
```
|
||||
|
||||
3. Stop all containers.
|
||||
|
||||
```bash
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
4. Copy our extracted data to the proper volumes. You can skip any of these as per your requirements. Also, remember to change the location of your extracted backups as necessary.
|
||||
|
||||
```bash
|
||||
# copy email
|
||||
SRC=$(find /var/mcRestore -iname "*vmail*" -type d)
|
||||
TGT=$(docker volume inspect -f '{{.Mountpoint}}' ${COMPOSE_PROJECT_NAME}_vmail-vol-1)
|
||||
(cd "$SRC/_data" && tar -cf - .) | (cd "$TGT" && tar xvf -)
|
||||
|
||||
# copy encryption key
|
||||
SRC=$(find /var/mcRestore -iname "*crypt*" -type d)
|
||||
TGT=$(docker volume inspect -f '{{.Mountpoint}}' ${COMPOSE_PROJECT_NAME}_crypt-vol-1)
|
||||
(cd "$SRC/_data" && tar -cf - .) | (cd "$TGT" && tar xvf -)
|
||||
|
||||
# copy postfix data
|
||||
SRC=$(find /var/mcRestore -iname "*postfix*" -type d)
|
||||
TGT=$(docker volume inspect -f '{{.Mountpoint}}' ${COMPOSE_PROJECT_NAME}_postfix-vol-1)
|
||||
(cd "$SRC/_data" && tar -cf - .) | (cd "$TGT" && tar xvf -)
|
||||
|
||||
# copy rspamd data
|
||||
SRC=$(find /var/mcRestore -iname "*rspamd*" -type d)
|
||||
TGT=$(docker volume inspect -f '{{.Mountpoint}}' ${COMPOSE_PROJECT_NAME}_rspamd-vol-1)
|
||||
(cd "$SRC/_data" && tar -cf - .) | (cd "$TGT" && tar xvf -)
|
||||
|
||||
# copy redis
|
||||
SRC=$(find /var/mcRestore -iname "*redis*" -type d)
|
||||
TGT=$(docker volume inspect -f '{{.Mountpoint}}' ${COMPOSE_PROJECT_NAME}_redis-vol-1)
|
||||
(cd "$SRC/_data" && tar -cf - .) | (cd "$TGT" && tar xvf -)
|
||||
```
|
||||
|
||||
5. Restart mailcow and monitor the logs. When you’re done monitoring the real-time log file, press `ctrl-c`
|
||||
|
||||
```bash
|
||||
docker-compose up -d && docker-compose logs -f
|
||||
```
|
||||
|
||||
6. Login to the mailcow admin console. Ensure your admin login works properly. Reset Rspamd console password (not restored for some reason).
|
||||
|
||||
7. Test out mailcow, ensure data is restored.
|
||||
|
||||
8. Clean-up extracted borg archive (optional but recommended to save space)
|
||||
|
||||
```bash
|
||||
rm -rf /var/mcRestore
|
||||
```
|
||||
|
||||
That’s it!
|
||||
@ -1,7 +1,7 @@
|
||||
# final notes
|
||||
|
||||
I think that's everything. If I've forgotten to document something, please let me know. I know this wiki is long but, I hate how much stuff for linux and open-source programs/scripts in general are so poorly documented especially for newbies and I didn't want to make that same mistake.
|
||||
|
||||
I don't script too often and I'm a horrible programmer, so if you see anything that can be/should be improved, please file an issue to let me know or a pull request to submit your changes! I love learning new ways of doing things and getting feedback, so suggestions and comments are more than welcome. If you are submitting changes, please *ensure they are commented*. I've tried to keep the script and all supporting files as commented as possible so others can look at the source and understand why I've done certian things in a certain way. I think this helps out as a learning tool too, so please respect that and offer your knowledge along with your suggestions.
|
||||
|
||||
# final notes
|
||||
|
||||
I think that's everything. If I've forgotten to document something, please let me know. I know this wiki is long but, I hate how much stuff for linux and open-source programs/scripts in general are so poorly documented especially for newbies and I didn't want to make that same mistake.
|
||||
|
||||
I don't script too often and I'm a horrible programmer, so if you see anything that can be/should be improved, please file an issue to let me know or a pull request to submit your changes! I love learning new ways of doing things and getting feedback, so suggestions and comments are more than welcome. If you are submitting changes, please *ensure they are commented*. I've tried to keep the script and all supporting files as commented as possible so others can look at the source and understand why I've done certian things in a certain way. I think this helps out as a learning tool too, so please respect that and offer your knowledge along with your suggestions.
|
||||
|
||||
If this has helped you out, then please visit my blog at https://mytechiethoughts.com where I solve problems like this all the time on a shoe-string or zero budget. Thanks!
|
||||
Loading…
Reference in New Issue
Block a user