Mechanical Turk

Question

I am trying to post a typical psychology experiment as a HIT on mechanical turk. I am presenting a set of a dozen word pairs and I need to randomize/counterbalance the order of the word pairs presented. On the typical HTML forms there seems to be no way to do this. Is there a way to do this using the API or Command Line interface?

Answer 1

Update 5/20/2016: Go with PsiTurk instead. See the answer by @John McDonnel

Update 6/25/2013: This answer is quite old. See Solomon Messing's blog post on MTurk in Social science for a starting point.

Given the amount of psych research that is starting to appear on AMT, you'd think they would add this as an option!

To the best of my knowledge, you can't do this with the web interface without:

1. Creating a bunch of templates (form A, form B, form C, form D).
2. Running the first form (form A).
3. Paying the workers and then blocking all of them.
4. Running the second form (form B).
5. Paying the workers and the blocking all from step 1 and step 2.
6. And repeating until you are done.

This is bad for a couple of reasons. The first being that you have to manually create the different forms. The second being that by blocking the workers who are doing good work, you are negatively impacting their reputation on the site. Your IRB will not likely look kindly on this.

The "best" way to solve this is to not use the web interface, but use the command line tools (CLT) to create ExternalQuestions. That way, you write a webapp that can handle the details of the randomization, give the workers an informed consent without relying on a qualification, and "politely block" workers who have already done the work. (The workers who have already participated would get a short message asking them to return the HIT.)

If you are interested in taking that path, I have some details on my blog about how I've done it.

Answer 2

if you decide to go the external question route listed by @Nathan VanHoudnos, I have built a framework for running Psychology experiments on Mechanical Turk using the external question interface. You can check it out here. The code runs a simple stroop experiment, hopefully you could change it to run whatever you want. Let me know if you want any help setting it up!