How do I resolve error in consolidating answers for custom labeling job in SageMaker?

I have created a custom labelling job using API. Job creation is successful. Also I am able to get responses. But in the console- it says that the job is complete with labelling errors and in the output manifest file I am getting this error and not the labels. Getting this for all images.

{"source-ref":"s3://Bucket_name/Unlabelled Images/image.png","result-metadata":{"retry-count":1,"failure-reason":"ServerError: Annotation Consolidation Failed. After retrying consolidation Lambda [3] times.","human-annotated":"true"}}.

Does it have to do anything with the job configuration that I used?

Posting here for the first time. Please let me know if I missed out on any important necessary details.