MongoDB - Percona Backup for MongoDB

Percona Backup for MongoDB (PBM) is an open source and distributed solution for consistent backups and restores of MongoDB sharded clusters and replica sets to a specific point in time.

Architecture

Percona Backup for MongoDB consists of the following components:

• pbm-agent is a process running on every mongod node within the cluster or within a replica set that performs backup and restore operations.
• pbm CLI is a command-line utility that instructs pbm-agents to perform an operation.
• PBM Control collections are special collections in MongoDB that store the configuration data and backup states. Both pbm CLI and pbm-agent use PBM Control collections to check backup status in MongoDB and communicate with each other.
• Remote backup storage is where Percona Backup for MongoDB saves backups. It can be either an S3 compatible storage or a filesystem-type storage.

The following diagram illustrates how Percona Backup for MongoDB components communicate with MongoDB.

1. Installation

• Install percona-release
• $ sudo yum install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm
• Enable the repository
• $ sudo percona-release enable pbm release
• Install Percona Backup for MongoDB packages
• $ sudo yum install percona-backup-mongodb

[root@mongodb-01 ~]# ls -l /etc | grep pbm

-rw-r-----   1 root         root           3626 May 15 17:10 pbm-conf-reference.yml

-rw-r-----   1 mongod       mongod           91 Aug 14 10:56 pbm-storage.conf

2. Configuration

👀Create user for backup on Primary Node

use admin

db.getSiblingDB("admin").createRole({ "role": "pbmAnyAction",

      "privileges": [

         { "resource": { "anyResource": true },

           "actions": [ "anyAction" ]

         }

      ],

      "roles": []

   });

   

db.getSiblingDB("admin").createUser({user: "p4backup",

       "pwd": "Password789",

       "roles" : [

          { "db" : "admin", "role" : "readWrite", "collection": "" },

          { "db" : "admin", "role" : "backup" },

          { "db" : "admin", "role" : "clusterMonitor" },

          { "db" : "admin", "role" : "restore" },

          { "db" : "admin", "role" : "pbmAnyAction" }

       ]

    });  

💀Create configuration file

[mongod@mongodb-01 ~]$ vi pbm_config.yaml 

storage:

        type: filesystem

        filesystem:

                path: /percona/backup   //this device is shared for All Nodes in Replica Set

pitr:

        enabled: true

        oplogSpanMin: 10    // with every "10" minutes, PBM will automatically backup Oplog. Check "pbm logs" for more information.

backup:

        oplogSpanMin: 10

        priority:

                "mongodb-02:27017": 2.5

                "mongodb-slave:27017": 2.5

                "mongodb-01:27017": 2.0

Note: pitr.oplogSpanMin and backup.oplogSpanMin are different.

// Load this configuration into database

[mongod@mongodb-01 ~]$ pbm config --file pbm_config.yaml

pitr:

  enabled: true

  oplogSpanMin: 15

storage:

  type: filesystem

  filesystem:

    path: /percona/backup

backup:

  oplogSpanMin: 15

  priority:

    mongodb-01:27017: 2

    mongodb-02:27017: 2.5

    mongodb-slave:27017: 2.5

[mongod@mongodb-01 ~]$ pbm config

Note:

If using NFS for sharing backup storage (/percona/backup) for all nodes in Replica set, take note "The ID of mongod user on all nodes in Replica set must be the same" in order to all nodes can Read Write on this storage device.

👻Configure pbm-agent on ALL nodes

[root@mongodb-01 ~]# vi /etc/sysconfig/pbm-agent

PBM_MONGODB_URI="mongodb://p4backup:Password789@mongodb-01:27017/?authSource=admin&replicaSet=rs0"

[root@mongodb-01 ~]# systemctl start pbm-agent.service 

[root@mongodb-01 ~]# systemctl enable pbm-agent.service 

[root@mongodb-01 ~]# systemctl status pbm-agent.service 

//  View log of pbm agent

[root@mongodb-01 ~]# journalctl -u pbm-agent.service

👿Configure env for pbm cli

[root@mongodb-01 ~]# su - mongod 

[mongod@mongodb-01 ~]$ vi .bash_profile 

...

export PBM_MONGODB_URI="mongodb://p4backup:Password789@mongodb-01:27017,mongodb-02:27017,mongodb-slave:27017/?authSource=admin&replicaSet=rs0"

3. Logical Backup

Note: 

Percona Backup for MongoDB is compatible with the following MongoDB versions:

• For logical backups – Percona Server for MongoDB and MongoDB Community v4.0 and higher with MongoDB Replication enabled. // Demo for this feature
• For physical backups – Percona Server for MongoDB starting from versions 4.2.15-16, 4.4.6-8, 5.0 and higher with MongoDB Replication enabled and WiredTiger configured as the storage engine.
• Incremental physical backups are supported for Percona Server for MongoDB starting with the following versions: 4.2.24-24, 4.4.18, 5.0.2-1, 6.0.2-1 and higher.
• Selective backup and restore
• Only logical backups and restores are supported
• Backup a single database or a specific collection and restore all data from it.
• Restore a specific collection from a single database backup
• Restore certain databases and / or collections from a full backup
• Make a point-in time recovery for the specified databases / collections. Available for replica sets only.

[mongod@mongodb-01 ~]$ pbm backup --type=logical --num-parallel-collections=4

Starting backup '2024-08-19T02:10:15Z'....Backup '2024-08-19T02:10:15Z' to remote store '/percona/backup' has started

[mongod@mongodb-01 ~]$ pbm list

Backup snapshots:

  2024-08-19T02:10:15Z <logical> [restore_to_time: 2024-08-19T02:10:30Z]

PITR <on>:

[mongod@mongodb-01 ~]$ pbm status

Cluster:

========

rs0:

  - rs0/mongodb-01:27017 [P]: pbm-agent v2.5.0 OK

  - rs0/mongodb-02:27017 [S]: pbm-agent v2.5.0 OK

  - rs0/mongodb-slave:27017 [S]: pbm-agent v2.5.0 OK

  - rs0/mongodb-arbiter:27017 [!Arbiter]: arbiter node is not supported

PITR incremental backup:

========================

Status [ON]

Currently running:

==================

(none)

Backups:

========

FS  /percona/backup

  Snapshots:

    2024-08-19T02:10:15Z 284.15MB <logical> [restore_to_time: 2024-08-19T02:10:30Z]

  PITR chunks [314.73KB]:

    2024-08-16T08:49:41Z - 2024-08-19T02:10:30Z (no base snapshot) // Do not use this pitr backup

After "oplogSpanMin" minutes, check again:

[mongod@mongodb-01 ~]$ pbm list

Backup snapshots:

  2024-08-19T02:10:15Z <logical> [restore_to_time: 2024-08-19T02:10:30Z]

PITR <on>:

  2024-08-19T02:10:31Z - 2024-08-19T02:23:28Z

[mongod@mongodb-01 ~]$ pbm status

Cluster:

========

rs0:

  - rs0/mongodb-01:27017 [P]: pbm-agent v2.5.0 OK

  - rs0/mongodb-02:27017 [S]: pbm-agent v2.5.0 OK

  - rs0/mongodb-slave:27017 [S]: pbm-agent v2.5.0 OK

  - rs0/mongodb-arbiter:27017 [!Arbiter]: arbiter node is not supported

PITR incremental backup:

========================

Status [ON]

Currently running:

==================

(none)

Backups:

========

FS  /percona/backup

  Snapshots:

    2024-08-19T02:10:15Z 284.15MB <logical> [restore_to_time: 2024-08-19T02:10:30Z]

  PITR chunks [431.99KB]:

    2024-08-19T02:10:31Z - 2024-08-19T02:14:28Z

    2024-08-16T08:49:41Z - 2024-08-19T02:10:30Z (no base snapshot)

[mongod@mongodb-01 ~]$ ls -l /percona/backup/pbmPitr/rs0/20240819/

total 60

-rw-r--r-- 1 mongod mongod 4576 Aug 19 09:10 20240819021017-6.20240819021030-4.oplog.s2

-rw-r--r-- 1 mongod mongod 9897 Aug 19 09:13 20240819021030-4.20240819021328-2.oplog.s2

[mongod@mongodb-01 ~]$ pbm logs

2024-08-19T02:10:35Z I [rs0/mongodb-02:27017] [pitr] copied chunks 2024-08-19T02:10:17 - 2024-08-19T02:10:30

2024-08-19T02:10:35Z I [rs0/mongodb-02:27017] [pitr] streaming started from 2024-08-19 02:10:30 +0000 UTC / 1724033430

2024-08-19T02:12:20Z I [rs0/mongodb-02:27017] [pitr] got wake_up signal

2024-08-19T02:12:20Z E [rs0/mongodb-02:27017] [pitr] streaming oplog: epoch mismatch. Got sleep in {1723798042 2}, woke up in {1724033535 3}. Too old for that stuff.

2024-08-19T02:12:28Z I [rs0/mongodb-01:27017] [pitr] streaming started from 2024-08-19 02:10:30 +0000 UTC / 1724033430

2024-08-19T02:13:30Z I [rs0/mongodb-01:27017] [pitr] created chunk 2024-08-19T02:10:30 - 2024-08-19T02:13:28. Next chunk creation scheduled to begin at ~2024-08-19 09:14:30.379413155 +0700 +07 m=+526.943743997

2024-08-19T02:14:28Z I [rs0/mongodb-01:27017] [pitr] created chunk 2024-08-19T02:13:28 - 2024-08-19T02:14:28. Next chunk creation scheduled to begin at ~2024-08-19 09:15:28.990051464 +0700 +07 m=+585.554382288

[mongod@mongodb-01 ~]$ pbm describe-backup "2024-08-19T02:10:15Z"

name: "2024-08-19T02:10:15Z"

opid: 66c2a98798fc4ba51e50a5c4

type: logical

last_write_time: "2024-08-19T02:10:30Z"

last_transition_time: "2024-08-19T02:10:32Z"

mongodb_version: 7.0.12

fcv: "7.0"

pbm_version: 2.5.0

status: done

size_h: 284.1 MiB

replsets:

- name: rs0

  status: done

  node: mongodb-slave:27017

  last_write_time: "2024-08-19T02:10:30Z"

  last_transition_time: "2024-08-19T02:10:31Z" 

[mongod@mongodb-01 ~]$ pbm describe-backup "2024-08-19T02:10:15Z" --with-collections

name: "2024-08-19T02:10:15Z"

opid: 66c2a98798fc4ba51e50a5c4

type: logical

last_write_time: "2024-08-19T02:10:30Z"

last_transition_time: "2024-08-19T02:10:32Z"

mongodb_version: 7.0.12

fcv: "7.0"

pbm_version: 2.5.0

status: done

size_h: 284.1 MiB

replsets:

- name: rs0

  status: done

  node: mongodb-slave:27017

  last_write_time: "2024-08-19T02:10:30Z"

  last_transition_time: "2024-08-19T02:10:31Z"

  collections:

  - admin.pbmAgents

  - admin.pbmBackups

  - admin.pbmCmd

  - admin.pbmConfig

  - admin.pbmLock

  - admin.pbmLockOp

  - admin.pbmLog

  - admin.pbmOpLog

  - admin.pbmPITRChunks

  - admin.pbmRRoles

  - admin.pbmRUsers

  - admin.system.roles

  - admin.system.users

  - admin.system.version

  - config.settings

  - sample_airbnb.helloWorld

  - sample_airbnb.listingsAndReviews

  - sample_airbnb.listingsAndReviewsNEW

  - sample_airbnb.listingsAndReviewsNEW01

  - sample_airbnb.listingsAndReviewsNEW02

  - sample_airbnb.listingsAndReviewsNEW03

  - sample_analytics.accounts

  - sample_analytics.customers

  - sample_analytics.transactions

  - sample_geospatial.shipwrecks

  - sample_mflix.comments

  - sample_mflix.movies

  - sample_mflix.sessions

  - sample_mflix.theaters

  - sample_mflix.users

  - sample_restaurants.neighborhoods

  - sample_restaurants.restaurants

  - sample_supplies.sales

  - sample_training.companies

  - sample_training.grades

  - sample_training.inspections

  - sample_training.posts

  - sample_training.routes

  - sample_training.trips

  - sample_training.zips

  - sample_weatherdata.data

💓How to check which Point-in-Time Recovery (PITR) Oplog interval backup belongs to a specific base backup ? // Need to check again

• List backup
• pbm list
• pbm status
• Get detail information of a specific base backup
• pbm describle-backup "<base backup name>"
• Get  last_write_time & last_transition_time 
• Go to /percona/backup/pbmPitr and find all relevant Oplog interval backups base on timestamp

4. Restoring a backup into a NEW environment

To restore a backup from one environment to another, ensure the following:

• Percona Backup for MongoDB configuration in the new environment must point to the same remote storage that is defined for the original environment, including the authentication credentials if it is an object store. Once you run pbm list and see the backups made from the original environment, then you can run the pbm restore command.
• Don’t run pbm backup from the new environment while Percona Backup for MongoDB configuration is pointing to the remote storage location of the original environment.

NEW environment:

Host: "192.168.56.91    mongodb-restore.taolaoxibup.com    mongodb-restore"

[root@mongodb-restore ~]# vi /etc/mongod.conf 

...

storage:

  dbPath: /u01/mongoDB/omni

replication:

  replSetName: "rs0"

Create replicat set

use admin

admin> rs.initiate()

Create PBM user for Restoration

Create the same user on Production server

Configure pbm-agent & pbm cli

[root@mongodb-restore ~]# vi /etc/sysconfig/pbm-agent

PBM_MONGODB_URI="mongodb://p4backup:Password789@mongodb-restore:27017/?authSource=admin"

[root@mongodb-restore ~]# systemctl restart pbm-agent.service 

[root@mongodb-restore ~]# journalctl -u pbm-agent.service

    

Load config setting into database

[mongod@mongodb-restore ~]$ export PBM_MONGODB_URI="mongodb://p4backup:Password789@mongodb-restore:27017/?authSource=admin&replicaSet=rs0"

[mongod@mongodb-restore ~]$ pbm config --file=pbm_config.yaml

pitr:

  enabled: false

  oplogSpanMin: 2

storage:

  type: filesystem

  filesystem:

    path: /percona/backup

[mongod@mongodb-restore ~]$ pbm list

Backup snapshots:

  2024-08-19T02:10:15Z <logical> [restore_to_time: 2024-08-19T02:10:30Z]

  2024-08-19T03:27:37Z <logical> [restore_to_time: 2024-08-19T03:27:55Z]

PITR <off>:

  2024-08-19T02:10:31Z - 2024-08-19T06:53:52Z

👉 If there is no backups listed, executing:

[mongod@mongodb-restore ~]$ pbm config --force-resync

Restore database

[mongod@mongodb-restore ~]$ pbm restore "2024-08-19T02:10:15Z"

Starting restore 2024-08-19T09:58:12.073412426Z from '2024-08-19T02:10:15Z'....Restore of the snapshot from '2024-08-19T02:10:15Z' has started

OR

[mongod@mongodb-restore ~]$ pbm restore --time "2024-08-19T02:10:31"    //PITR

Ref:

• Percona Backup for MongoDB - Open Source MongoDB Backup
• Percona Backup for MongoDB
• MongoDB Database Backup: Best Practices & Expert Tips (percona.com)
• Percona Backup for MongoDB in Action