Tuesday, August 06, 2024

MongoDB - Replication

Preparation
1. Install mongodb on all nodes
2. Configuring /etc/hosts on nodes
[root@mongodb-xxx ~]# cat /etc/hosts 
192.168.56.87  mongodb-01.taolaoxibup.com               mongodb-01
192.168.56.88  mongodb-02.taolaoxibup.com               mongodb-02
192.168.56.89  mongodb-slave.taolaoxibup.com           mongodb-slave
192.168.56.90  mongodb-arbiter.taolaoxibup.com         mongodb-arbiter    // an arbiter
 
3. Configuring /etc/mongod.conf on all nodes
[root@mongodb-xxx ~]# vi /etc/mongod.conf 
...

# network interfaces
net:
  port: 27017
  bindIp: mongodb-01  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.            // Update right value for each mongos instances

replication:
  replSetName: "rs0"
...

[root@mongodb-xxx ~]# systemctl restart mongod.service 

Configuration

1. On Primary Node
- Initiate the replica set
  • rs.initiate()
  • rs.status()
  • rs.conf()
- Add Members to a Replica Set
  • rs.add({host: "mongodb-02:27017"})
  • rs.add({host: "mongodb-slave:27017"})
- Verify configuration
  • rs.status()
  • rs.conf()
  • rs.hello()
  • db.isMaster()
2. On Secondary node - Verify configuration
  • rs.status()
  • rs.conf()
  • rs.hello()
  • db.isMaster()
3. Add Members to a Replica Set
  • Start new mongod instance
    • mongod --port 27018 --dbpath /u01/mongoDB/saigon --replSet rs0 --bind_ip 0.0.0.0
    • OR nohup mongod --port 27018 --dbpath /u01/mongoDB/saigon --replSet replOMNI --bind_ip_all > 27018_$HOSTNAME.log 2>&1 &
  • On Primary node, add new member to replica set
    • rs.add("mongodb-01:27018")
  • Verify configuration
    • rs.status()
    • rs.conf()
    • db.isMaster()
4. Check Data Synchronization
  • rs.status() Command
    • stateStr: Indicates the state of each member (e.g., PRIMARY, SECONDARY).
    • optimeDate: Shows the last applied operation time on each member.
    • syncingTo: Indicates which member a secondary node is syncing from.
    • lag: Any significant difference in optimeDate values may indicate synchronization lag.
  • rs.printSlaveReplicationInfo() Command
    • source: The host from which the secondary is replicating.
    • syncedTo: The timestamp of the last operation applied by the secondary.
    • replLag: The difference in time between the syncedTo timestamp and the current time, indicating replication lag
  • rs.printReplicationInfo() Command
    • actual oplog size
    • configured oplog size: The size of the oplog.
    • log length start to end: The time range covered by the oplog.
    • oplog first event time: The timestamp of the first operation in the oplog.
    • oplog last event time: The timestamp of the last operation in the oplog.
    • now
  • MongoDB Logs
    • grep 'replication' /var/log/mongodb/mongod.log
  • mongostat Tool
  • Monitoring Tools
    • MongoDB Atlas: If you are using MongoDB Atlas, it provides a comprehensive dashboard for monitoring replication and other metrics.
    • Ops Manager: MongoDB Ops Manager offers advanced monitoring and management capabilities for on-premises deployments.
    • Third-Party Tools: Tools like Prometheus, Grafana, and Datadog can be configured to monitor MongoDB replication metrics.
Example:

rs0 [direct: primary] admin> db.getReplicationInfo()
{
  configuredLogSizeMB: 2723.10107421875,
  logSizeMB: 2723.10107421875,
  usedMB: 0.46,
  timeDiff: 174872,
  timeDiffHours: 48.58,
  tFirst: 'Wed Aug 07 2024 09:33:21 GMT+0700 (Indochina Time)',
  tLast: 'Fri Aug 09 2024 10:07:53 GMT+0700 (Indochina Time)',
  now: 'Fri Aug 09 2024 10:07:54 GMT+0700 (Indochina Time)'
}

rs0 [direct: primary] admin> rs.printReplicationInfo()
actual oplog size
'2723.10107421875 MB'
---
configured oplog size
'2723.10107421875 MB'
---
log length start to end
'174902 secs (48.58 hrs)'
---
oplog first event time
'Wed Aug 07 2024 09:33:21 GMT+0700 (Indochina Time)'
---
oplog last event time
'Fri Aug 09 2024 10:08:23 GMT+0700 (Indochina Time)'
---
now
'Fri Aug 09 2024 10:08:25 GMT+0700 (Indochina Time)'

rs0 [direct: primary] admin> rs.printSecondaryReplicationInfo()
source: mongodb-02:27017
{
  syncedTo: 'Fri Aug 09 2024 10:08:38 GMT+0700 (Indochina Time)',
  replLag: '0 secs (0 hrs) behind the primary '
}
---
source: mongodb-slave:27017
{
  syncedTo: 'Fri Aug 09 2024 10:08:38 GMT+0700 (Indochina Time)',
  replLag: '0 secs (0 hrs) behind the primary '
}
---
source: mongodb-01:27018
{
  syncedTo: 'Fri Aug 09 2024 10:08:38 GMT+0700 (Indochina Time)',
  replLag: '0 secs (0 hrs) behind the primary '
}

5. Add an Arbiter to Replica Set - on Primary Node

rs0 [direct: primary] admin> rs.addArb("mongodb-arbiter:27017")
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1723175124, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1723175124, i: 1 })
}

rs0 [direct: primary] admin> rs.conf()
...
    {
      _id: 4,
      host: 'mongodb-arbiter:27017',
      arbiterOnly: true,
      buildIndexes: true,
      hidden: false,
      priority: 0,
      tags: {},
      secondaryDelaySecs: Long('0'),
      votes: 1
    }
...

rs0 [direct: primary] admin> rs.status()
...
    {
      _id: 4,
      name: 'mongodb-arbiter:27017',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 240,
      lastHeartbeat: ISODate('2024-08-09T03:56:31.954Z'),
      lastHeartbeatRecv: ISODate('2024-08-09T03:56:31.979Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 22,
      configTerm: 22
    }
...

Note:
  • Secondaries capture data from the primary member to maintain an up to date copy of the sets' data. However, by default secondaries may automatically change their sync targets to secondary members based on changes in the ping time between members and the state of other members’ replication. Check syncSourceHost in rs.status() for more information.
  • The oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. For Unix and Windows systems, the minimum oplog size is 990 MB and The maximum default oplog size is 50 GB.
  • The priority settings of replica set members affect both the timing and the outcome of elections for primary. Higher-priority members are more likely to call elections, and are more likely to win. Use this setting to ensure that some members are more likely to become primary and that others can never become primary. Adjust on Primary Node (use cls comamd in mongosh to refresh role of replication cluster)
    • cfg = rs.conf()
    • cfg.members[0].priority = 3   
    • cfg.members[1].priority = 2
    • cfg.members[2].priority = 1
    • rs.reconfig(cfg)
          ==> member[1] becomes PRIMARY if member[0] is DOWN. If member[0] is UP again, member[0] will automatically become PRIMARY.


at
Labels: