How can I limit the number of tasks a worker can respond to in the dataset labeling jobs and how can I assess them based on my control items?

Question

I'm using the Ground truth built-in labeling tool for a text classification task. Since there are only two labels (Yes vs. No), how can I make sure that the MTurk workers are really paying attention to the task? For example, is there a way to add control items to the dataset to check their performance? and also since my dataset contains a large number of sentences, is there a way to limit the number of tasks an MTurk worker can respond to?

Answer

Hi,

Please use MaxConcurrentTaskCount to control the number of objects available to workers - https://docs.aws.amazon.com/sagemaker/latest/dg/sms-batching.html#sms-batching-maxconcurrenttaskcount

You can refer the MTurk best practices here - https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/IntroBestPractices.html

Also, make sure to test your job with a few objects before the complete job, so you can rectify the shortcomings, if any, beforehand.

How can I limit the number of tasks a worker can respond to in the dataset labeling jobs and how can I assess them based on my control items?

Relevant content