My list queries are returning the wrong number of items when I use Amazon DynamoDB with AWS AppSync. How do I resolve this?
Short description
When you use a list query with AWS AppSync and Amazon DynamoDB, DynamoDB performs a Scan operation on the table and returns a certain number of evaluated items. If you provide a FilterExpression operation on the request, then DynamoDB applies the FilterExpression to only these evaluated items.
The number of items that the Scan checks depends on the limit variable that you applied to the query. If no limit is applied, then DynamoDB uses the default limit that's configured in the request mapping template. The Scan returns only the items that are evaluated and match the filter expression. To return the expected number of items, you must adjust the limit variable.
You can increase the default limit on your mapping template, or add a limit variable to the list GraphQL query. However, the maximum number of items that you can evaluate is 1 MB. If your Scan operation exceeds 1 MB of data, then you can use pagination to get the rest of the data. To help you query a database that has thousands of items, you can index your data to filter results by a common field.
Resolution
Increase the default limit
In the following example mapping template for an AWS Amplify generated API, the default limit is 100:
#set( $limit = $util.defaultIfNull($context.args.limit, 100) )
#set( $ListRequest = {
"version": "2018-05-29",
"limit": $limit
} )
To evaluate more items, change the default limit in the mapping template. Or, add a limit variable to the list query. For example, in the following list query, the limit is set to 1000:
query MyQuery {
listEmployees(limit: 1000) {
items {
id
name
company
}
}
}
Paginate with AWS AppSync
To paginate with AWS AppSync, add a nextToken argument to your GraphQL query.
Example query:
query MyQuery {
listEmployees{
items {
id
name
company
}
nextToken
}
}
Note: The nextToken value that the query returns is either null or a long string that starts with ey and ends with =. This value represents the last evaluated key. If the value is null, then there are no more items in the table to evaluate. If it's a long string, then there are more items to evaluate.
For example:
"nextToken": "eyJ2ZXJzaW9uIjoyLCJ0b2tlbiI6IkFRSUNBSGg5OUIvN3BjWU41eE96NDZJMW5GeGM4WUNGeG1acmFOMUpqajZLWkFDQ25BRkQxUjREaVVxMkd1aDZNZ2ZTMmhPMUFBQUNIVENDQWhrR0NTcUdTSWIzRFFFSEJxQ0NBZ293Z2dJR0FnRUFNSUlCL3dZSktvWklodmNOQVFjQk1CNEdDV0NHU0FGbEF3UUJMakFSQkF4c0RFY1ZZUGpaRDFxbDcxNENBUkNBZ2dIUWkwaGtJYytoU04vRFMzL3VQT2ZDMnpIV1dBVkg4LzU3TWFwMGZWSHVackt1VmV4emRvTVQrNml6VC9RdDRsSVNGN1pmc3NWTHVvcnJpRE1RZVE5ckNyd3J4dmNOY3ZZUzhTc21PRFFkaTUreVhQcDJ1OENaK29Sd2wrV3VoOGJ0YlBpQXQydjRNdmw2L09jRzRHV2RxRmFjeTFDRjRyK2FPd29velRTV3NqMTd4OUpwVi93cHVYc2tTN2R5TmYxa3JJS3hSL3BMWlY5M3JPSlVocEpDV2xEL3Y1UU5JdGJoOWpyaTI3N09LbUZsVlh3bDRna3hDa1kzeGZMNjZrM2dZY0hpbHlUOE1GUnBLU0VCTFE3RGEzSlJtUG8xUkJvQ051K3dBTkwwd09Vckp0N1BFb0QvTVRrenZDV1lCTXhVaUhaRDdrM3Y5a2FJS2NScHY0ajhuTDJ2Nk9EZ3plTjgwN1RtRFViM21rRUsrakRRcjUvd3ZQVW56cGFaN1RnR21yT21FaTlGQklOUnl6dk9rdDRWazZEaVU3RCtNYUJSdm5iNnR0VklPa2lDdFlhODRqenhlOFlFRUZGOElyTksrQm9yL28vdktxMVczSUxsU1VWWFd0N0hPWjV4TDBudHVTeGlBdW9ZK1Y0NEkzMXlPQkJ1T1AwMVpUek1TdGUvZCtIT1RRUEt2SGVGanF5Y0tpNGNTQUdZN3BobGs5eWJJem9hOTM0YldJOUFyRmF0WDY4UnkzTkF4cWNCbzh4ZklxZGZNN3Rlam02NldMT0Z6T3F6MDRrK1B0K0lXdWhOeS9CWEN2YXh2dk09In0="
To get the rest of the items in the table, run another query with the nextToken as a query variable. Continue to include the nextToken variable until all the items are evaluated.
Example query:
query MyQuery {
listEmployees(nextToken:"eyJ......="){
items {
id
name
company
}
nextToken
}
}
Index your data
In the following example query, the filter expression is set to return a number of items that are based on employees that equal AnyCompany:
query MyQuery {
listEmployees(filter: {company: {eq: "AnyCompany"}}) {
items {
id
name
company
}
nextToken
}
}
To list all employees based on their company, create a global secondary index for the field to query against. For example, in Amplify, you can use the @index directive to define secondary indexes:
type Employee @model{
id:ID
name : String
email : AWSEmail
company: String @index(name:"employeeByCompany", queryField: "listEmployeesByCompany")
}
In the following example query, a GSI employeeByCompany is set for the company field, and the query is defined as listEmployeesByCompany:
query MyQuery {
listEmployeesByCompany(company: "AnyCompany") {
items {
id
name
company
}
nextToken
}
}
You can specify a limit variable to return an intended number of items in the database. For example, in the following query, the limit variable is set to 5:
query MyQuery {
listEmployeesByCompany(company: "AnyCompany",limit:5) {
items {
id
name
company
}
nextToken
}
}
Note: Setting a limit variable returns only the intended number of items, even if you applied additional filterExpressions. For example, in the following query, the limit variable is set to 5 and the filter equals Mary Major:
query MyQuery {
listEmployeesByCompany(company: "AnyCompany",limit:5) {
name:{
eq:"Mary Major"
}
}) {
items {
id
name
company
}
nextToken
}
}
The preceding query returns only 5 items that equal Mary Major. However, there might be more than 5 Mary Majors.
Note: Indexed queries also return a maximum of 1 MB of data at a time. For larger indexed queries, you can use pagination.