Ongoing replication from MongoDB to RDS PostgresSQL

0

Created an AWS DMS pipeline: Source endpoint - **MongoDB ** Target endpoint - RDS Postgres SQL

Successfully did all the security configuration, and both **endpoints **returned **successful **while testing it. For the MongoDB source, I am using one of the three **replicas **sets with a username and a password that is not the admin username.

I also added the privilege "changeStream" in the replica set user.

But when starting the DMS migration task getting this error in cloud watch.

Encountered an error while initializing change stream: 'not authorized on admin to execute command 
{ aggregate: 1, pipeline: [ { $changeStream: { fullDocument: "updateLookup", startAtOperationTime: Timestamp(1656005815, 0), 
allChangesForCluster: true } }, "ok" : { "$numberDouble" : "0.0" },
 "errmsg" : "not authorized on admin to execute command { aggregate: 1, pipeline: [ { $changeStream: { fullDocument:
 \"updateLookup\", startAtOperationTime: Timestamp(1656005815, 0), allChangesForCluster: true } },
 74f1-4aab-9ca1-f964ab655777\ (change_streams_capture.c:356)

Assuming this is due to some missing privileges in mongo replica sets USER.

asked 2 years ago371 views
1 Answer
0
Accepted Answer

Hi it is likely that it has to do with your MONGO DB source server . so you need to set up mongo DB according to the steps below.

Modify mongod configuration file /etc/mongod.conf using vi editor

    sudo vi /etc/mongod.conf

Change below lines from

    # network interfaces
    net:
      port: 27017
      bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
    #security:


T0

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.
security:
  authorization: enabled

Restart mongod service

   [ec2-user@ip-172-31-0-47 ~]$ sudo service mongod restart
    Stopping mongod:                                           [  OK  ]
    Starting mongod:                                           [  OK  ]

add dmsuser with read access to testdb. This time you need to login with root access

 [ec2-user@ip-172-31-0-47 ~]$ mongo localhost/admin -u root -p
    MongoDB shell version v3.6.0-rc8
    Enter password: 
    connecting to: mongodb://localhost:27017/admin
    MongoDB server version: 3.6.0-rc8
    Server has startup warnings: 
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] 
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
    > use admin
    switched to db admin
    > db.createUser( { user:"dms", pwd:"dms123", roles: [ { role: "read", db: "testdb" } ] } )
    Successfully added user: {
        "user" : "dms",
        "roles" : [
            {
                "role" : "read",
                "db" : "testdb"
            }
        ]
    }
    > show users
    {
        "_id" : "admin.dms",
        "user" : "dms",
        "db" : "admin",
        "roles" : [
            {
                "role" : "read",
                "db" : "testdb"
            }
        ]
    }
    {
        "_id" : "admin.root",
        "user" : "root",
        "db" : "admin",
        "roles" : [
            {
                "role" : "root",
                "db" : "admin"
            }
        ]
    }
    > 

Above settings are for FULL LOAD task

For CDC replication needs to be setup and permissions need to be modified as below

Modify mongod.conf using vi editor

[ec2-user@ip-172-31-0-47 ~]$ sudo vi /etc/mongod.conf
replication:
  replSetName: rs0

restart mongod service

[ec2-user@ip-172-31-0-47 ~]$ sudo service mongod restart
Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]

[ec2-user@ip-172-31-0-47 ~]$ mongo localhost/admin -u root -p
MongoDB shell version v3.6.0-rc8
Enter password: 
connecting to: mongodb://localhost:27017/admin
MongoDB server version: 3.6.0-rc8
Server has startup warnings: 
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] 
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
> rs.status()
{
    "info" : "run rs.initiate(...) if not yet done for the set",
    "ok" : 0,
    "errmsg" : "no replset config has been received",
    "code" : 94,
    "codeName" : "NotYetInitialized",
    "$clusterTime" : {
        "clusterTime" : Timestamp(0, 0),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "ip-172-31-0-47:27017",
    "ok" : 1,
    "operationTime" : Timestamp(1512306101, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1512306101, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
> rs.status()
{
    "operationTime" : Timestamp(1512306102, 5),
    "ok" : 0,
    "errmsg" : "Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1512306101, 1) } with id: 0",
    "code" : 211,
    "codeName" : "KeyNotFound",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1512306102, 5),
        "signature" : {
            "hash" : BinData(0,"U10mDrsgP7b6tcG2ShZEKpDQrzA="),
            "keyId" : NumberLong("6495305249631240193")
        }
    }
}
> 

Grant dmsuser permissions read to local so that it can read logs from local.oplog.rs.

rs0:PRIMARY> db.dropUser("dms")
true
rs0:PRIMARY> db.createUser( { user:"dms", pwd:"dms123", roles: [ { role: "read", db: "testdb" },{ role: "read", db: "local" } ] } )
Successfully added user: {
    "user" : "dms",
    "roles" : [
        {
            "role" : "read",
            "db" : "testdb"
        },
        {
            "role" : "read",
            "db" : "local"
        }
    ]
}
rs0:PRIMARY> 

Make sure security group is open for dms replication group on running port (default 27017)

references

Using MongoDB as a source for AWS DMS - Permissions needed when using MongoDB as a source for AWS DMS -[1] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html#CHAP_Source.MongoDB.PrerequisitesCDC

answered 2 years ago
profile picture
EXPERT
reviewed 17 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions