DynamoDB scan with multiple filters

Question

I'm trying to do a DynamoDB Scan from a NodeJS lambda, and the results I'm getting are hugely different than when I enter the same filters explicitly in DynamoDB directly.
I'm using the following code:

```
var params = {
        TableName : tbl,
        FilterExpression: "#uploadDate BETWEEN :date1 and :date2 AND #imageSeen = :notSeen",
        ExpressionAttributeValues : {
            ":date1" : StartDate,
            ":date2" : EndDate,
            ":notSeen"  : false
        },
        ExpressionAttributeNames: {
            "#uploadDate": "UploadDate",
            "#imageSeen" : "ImageSeen"
          }
      };
```
And then the scan:

```
scanWithParams: async(params) => {
        let res ={};
        try {
             res = await documentClient.scan(params).promise();
        }
        catch (err) {
            console.log('[scanWithParams] : error scanning' , err);
        }
        return res.Items;
    }   
```
Both dates are strings, and ImageSeen is a boolean. When I run this directly, I get 1800 results back, while via the above code I only get 36.

Any help is greatly appreciated.

Update: - Here's a picture of doing it directly -- Does my params match this, the way I wrote it?![Enter image description here](/media/postImages/original/IMzOF4hBrXQz24C5htCbM_qA)

Accepted Answer

The DynamoDB Web Console auto-paginates on your behalf, and sets a limit of 300, so it will always return 300 items if they exist:

```
{
    "TableName":"YourTable",
    "ReturnConsumedCapacity":"TOTAL",
    "Limit":300,
    "Select":"ALL_ATTRIBUTES",
    "FilterExpression":"YourFilterExpression"
}
```

In order for you to achieve the same results, you will need to include pagination in your code. You can read more [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Pagination)

```
myItems = []
scanWithParams: async(params) => {
        let res ={};
        try {
             res = await documentClient.scan(params).promise();
             myItems  = [...myItems, ...res['Items']];
             if (data.LastEvaluatedKey) {
                     params.ExclusiveStartKey = res.LastEvaluatedKey;
                     return await getAllData(params);
            }
        }
        catch (err) {
            console.log('error scanning' , err);
        }
        return myItems;
    }
```

Answer

Is it possible that the scan operation is returning more than 1MB of data? [That is the limit](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) per scan operation.

If by "run this directly" you mean in the console, what you're seeing is the console automatically calling scan multiple times to get additional data. This is indicated in the return data if the `LastEvaluatedKey` parameter is present. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.Pagination.html

DynamoDB scan with multiple filters

相關內容