When creating a labeling job through the Amazon Ground Truth console using Amazon Mechanical Turk as the worker type, you are asked to specify the "Number of workers per dataset object".
In order to ensure higher accuracy of the labeling process, I specify that I want 3 workers to label each dataset object.
When the job finishes though and I inspect the output manifest, only a very small portion (sometimes even none) of the dataset objects have actually been annotated by 3 workers.
Is there a way to make sure that an object is labeled by the specified number of workers before being considered as complete? Or at least a way to increase the percentage of objects labeled by the specified number of workers?
Extra information of labeling job:
- The task is "span classification" which falls under the same category as "Named entity recognition" and the worker pay is more than double the amount suggested for this particular task
- The "Task timeout" option is set to 1 hour (not sure if/how it affects the issue I am facing)
- The "Task expiration time" is set to the maximum value of 12 hours
I have tried experimenting with different values for the "Task timeout" option as well as increasing the "Price per task", to no avail.
