Ongoing replication from MongoDB to RDS PostgresSQL

0

Created an AWS DMS pipeline: Source endpoint - **MongoDB ** Target endpoint - RDS Postgres SQL

Successfully did all the security configuration, and both **endpoints **returned **successful **while testing it. For the MongoDB source, I am using one of the three **replicas **sets with a username and a password that is not the admin username.

I also added the privilege "changeStream" in the replica set user.

But when starting the DMS migration task getting this error in cloud watch.

Encountered an error while initializing change stream: 'not authorized on admin to execute command 
{ aggregate: 1, pipeline: [ { $changeStream: { fullDocument: "updateLookup", startAtOperationTime: Timestamp(1656005815, 0), 
allChangesForCluster: true } }, "ok" : { "$numberDouble" : "0.0" },
 "errmsg" : "not authorized on admin to execute command { aggregate: 1, pipeline: [ { $changeStream: { fullDocument:
 \"updateLookup\", startAtOperationTime: Timestamp(1656005815, 0), allChangesForCluster: true } },
 74f1-4aab-9ca1-f964ab655777\ (change_streams_capture.c:356)

Assuming this is due to some missing privileges in mongo replica sets USER.

gefragt vor 2 Jahren382 Aufrufe
1 Antwort
0
Akzeptierte Antwort

Hi it is likely that it has to do with your MONGO DB source server . so you need to set up mongo DB according to the steps below.

Modify mongod configuration file /etc/mongod.conf using vi editor

    sudo vi /etc/mongod.conf

Change below lines from

    # network interfaces
    net:
      port: 27017
      bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
    #security:


T0

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.
security:
  authorization: enabled

Restart mongod service

   [ec2-user@ip-172-31-0-47 ~]$ sudo service mongod restart
    Stopping mongod:                                           [  OK  ]
    Starting mongod:                                           [  OK  ]

add dmsuser with read access to testdb. This time you need to login with root access

 [ec2-user@ip-172-31-0-47 ~]$ mongo localhost/admin -u root -p
    MongoDB shell version v3.6.0-rc8
    Enter password: 
    connecting to: mongodb://localhost:27017/admin
    MongoDB server version: 3.6.0-rc8
    Server has startup warnings: 
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] 
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
    2017-12-03T12:15:40.584+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
    > use admin
    switched to db admin
    > db.createUser( { user:"dms", pwd:"dms123", roles: [ { role: "read", db: "testdb" } ] } )
    Successfully added user: {
        "user" : "dms",
        "roles" : [
            {
                "role" : "read",
                "db" : "testdb"
            }
        ]
    }
    > show users
    {
        "_id" : "admin.dms",
        "user" : "dms",
        "db" : "admin",
        "roles" : [
            {
                "role" : "read",
                "db" : "testdb"
            }
        ]
    }
    {
        "_id" : "admin.root",
        "user" : "root",
        "db" : "admin",
        "roles" : [
            {
                "role" : "root",
                "db" : "admin"
            }
        ]
    }
    > 

Above settings are for FULL LOAD task

For CDC replication needs to be setup and permissions need to be modified as below

Modify mongod.conf using vi editor

[ec2-user@ip-172-31-0-47 ~]$ sudo vi /etc/mongod.conf
replication:
  replSetName: rs0

restart mongod service

[ec2-user@ip-172-31-0-47 ~]$ sudo service mongod restart
Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]

[ec2-user@ip-172-31-0-47 ~]$ mongo localhost/admin -u root -p
MongoDB shell version v3.6.0-rc8
Enter password: 
connecting to: mongodb://localhost:27017/admin
MongoDB server version: 3.6.0-rc8
Server has startup warnings: 
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] 
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-12-03T12:59:31.476+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
> rs.status()
{
    "info" : "run rs.initiate(...) if not yet done for the set",
    "ok" : 0,
    "errmsg" : "no replset config has been received",
    "code" : 94,
    "codeName" : "NotYetInitialized",
    "$clusterTime" : {
        "clusterTime" : Timestamp(0, 0),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "ip-172-31-0-47:27017",
    "ok" : 1,
    "operationTime" : Timestamp(1512306101, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1512306101, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
> rs.status()
{
    "operationTime" : Timestamp(1512306102, 5),
    "ok" : 0,
    "errmsg" : "Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1512306101, 1) } with id: 0",
    "code" : 211,
    "codeName" : "KeyNotFound",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1512306102, 5),
        "signature" : {
            "hash" : BinData(0,"U10mDrsgP7b6tcG2ShZEKpDQrzA="),
            "keyId" : NumberLong("6495305249631240193")
        }
    }
}
> 

Grant dmsuser permissions read to local so that it can read logs from local.oplog.rs.

rs0:PRIMARY> db.dropUser("dms")
true
rs0:PRIMARY> db.createUser( { user:"dms", pwd:"dms123", roles: [ { role: "read", db: "testdb" },{ role: "read", db: "local" } ] } )
Successfully added user: {
    "user" : "dms",
    "roles" : [
        {
            "role" : "read",
            "db" : "testdb"
        },
        {
            "role" : "read",
            "db" : "local"
        }
    ]
}
rs0:PRIMARY> 

Make sure security group is open for dms replication group on running port (default 27017)

references

Using MongoDB as a source for AWS DMS - Permissions needed when using MongoDB as a source for AWS DMS -[1] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html#CHAP_Source.MongoDB.PrerequisitesCDC

beantwortet vor 2 Jahren
profile picture
EXPERTE
überprüft vor einem Monat

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen