Why Does Amazon Mechanical Turk Still Exist?

Amazon Mechanical Turk (MTurk) has provided inspiration for strongly-worded blog posts since its inception in 2005.[1] I will focus here on the specific question of how MTurk will persist in light of dissatisfaction among so many network participants.

Amazon launched MTurk to “crowdsource” Human Intelligence Tasks (HITs) that are relatively easy, or at least possible, for a human to complete, but challenging or impossible for a computer to complete reliably. HITs might include determining whether there is a barber shop in an image or identifying the mood of a song.

More recently, MTurk has played an important role in social science research as a convenient hive of inexpensive survey participants that, according to some studies, are sufficiently representative and reliable.[2]

MTurk is named after an 18th century ruse in which a chess master was hidden inside a contraption–a Mechanical Turk–claimed to be a chess-playing machine.

http://www.palmit.com/assets/images/au_turk.jpg

http://www.palmit.com/assets/images/au_turk.jpg

The structure of MTurk is a classic two-sided network consisting of Requesters and Workers. Requesters set prices for completion of HITs, and Workers may accept HITs after reviewing them. Workers are not penalized for not completing HITs, but receive no compensation for incomplete HITs.

Both Workers and Requesters benefit from the size of the MTurk network. Workers benefit from having many Requesters who provide many HITs from which to choose (increasing demand for labor and availability of work), and Requesters benefit from an abundance of Workers to ensure competition for HITs, driving prices down and increasing speed of HIT completion. MTurk provides value to all parties by monetizing underutilized assets–in this case, human labor–by reducing labor-market friction (transportation, recruitment, etc.) through the efficiency of a technology platform. Just like Airbnb allows someone to monetize their “underutilized” home, MTurk allows Workers to monetize their “underutilized” ability to complete HITs.

This summer, MTurk changed the structure of its commissions (the amount it charges a Requester per HIT) amid complaints from both Workers and Requesters.[3] From the MTurk blog, “These changes will help allow Amazon to continue growing the Amazon Mechanical Turk marketplace and innovating on behalf of our customers.”[4]

The commission increase is one, but not the only, reason why the current structure of MTurk invites competition from upstart competitors. Wages for Workers were very low even before the commission increase, which constrains demand for HITs and drives wages even lower.[5] Further, Workers have little recourse when Requesters provide misleading Task instructions or time estimates. So a competing service that offers more protections could be an attractive alternative. (But wouldn’t Requesters avoid a competing service that protected Workers from abuse? Stay with me.)

Requesters also see diminishing benefits of MTurk. The commission increase directly reduces the ability of social science researchers (an increasing proportion of Requesters)[6] to use MTurk as a tool to collect survey respondents, given that research funds tend to be fixed. Some researchers have decided to reduce the wages they offer, or simply recruit fewer respondents,[7] eroding the quality of MTurk for Workers.

Requesters face another type of challenge in addition to the commission increase: the inherent value of MTurk to social science survey research may be declining. This issue is more fundamentally erosive of the MTurk value proposition than commission increases and less easily remedied (though not impossible to remedy). The quality of MTurk-generated survey research is threatened by the trend toward “super users” who are skillful at answering the same types of questions; social networks such as mturkforum.com or reddit where Workers share tips on answers and strategies; and the lack of enforcement of basic standards of social science research that ensure quality data as well as ethical practice.[8]

Other networks are forming that provide some benefits that MTurk does not. Sticky Crowd touts better rates than MTurk; sites like peopleperhour and Upwork focus on more highly skilled tasks. Notably, a popular Requester, Crowdsource, has insourced and now uses its own platform to accomplish the same end as it did with MTurk.

It seems clear that the technology of MTurk is nothing special, at least not in 2015. (That is to say, it is replicable.) MTurk may be the largest player in the space presently, but because multi-homing costs are low for both Requesters and Workers (it is convenient for either type of user to sign up for multiple services), an upstart competitor could lure Workers and Requesters by providing a superior product with better Worker protections and lower commissions. While Worker protections could force Requesters to pay more for HIT completion, it would ultimately result in a stronger, more sustainable network that would benefit both Requesters as well as Workers. Better Worker conditions would attract a more representative and reliable pool of Workers, as well as establishing accountability among all Requesters and rooting out the rotten apples that spoil the bunch.

Perhaps the reality is that survey responses as cheap as MTurk initially provided cannot yield good data in the long run and were merely the result of the platform’s novelty. So, a company that provides an MTurk-type platform at high quality could ultimately be the Facebook to MTurk’s MySpace.

Why, then, does Amazon maintain MTurk in its current format? Perhaps it is at least slightly profitable. Or the strategy is (ostensibly) frugal, if not directly profitable: if Amazon continues to use MTurk to recruit Workers for its own HITs, which was the original reason for MTurk’s existence, it makes sense to open platform for other Requesters to recruit Workers, thereby increasing Amazon the supply of Workers.

But if Amazon does not cultivate and improve the MTurk network to keep both Requesters and Workers happy, the current strategy will prove penny wise and pound foolish, and we may see the day when nobody wants to use Mechanical Turk anymore, leaving Amazon to outsource its own HITs to a competitor.

1. http://www.theguardian.com/technology/2014/dec/03/amazon-mechanical-turk-workers-protest-jeff-bezos

2. Buhrmester M, Kwang T, and Gosling SD. Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science 6(1) 3–5, 2011. http://datacolada.org/wp-content/uploads/2014/04/Burhmester-Kwang-Gosling-2011.pdf

3. http://blogs.wsj.com/digits/2015/06/23/amazons-mechanical-turk-fee-hike-irks-researchers/

4. http://mechanicalturk.typepad.com/blog/2015/06/following-up-on-our-commission-structure.html

5. http://www.utne.com/science-and-technology/amazon-mechanical-turk-zm0z13jfzlin.aspx

6. http://www.washingtonpost.com/blogs/monkey-cage/wp/2015/05/04/researchers-are-rushing-to-amazons-mechanical-turk-should-they/

7. http://blogs.wsj.com/digits/2015/06/23/amazons-mechanical-turk-fee-hike-irks-researchers/

8. http://www.linkedin.com/pulse/my-experience-amazon-mechanical-turk-mturk-worker-utpal-dholakia

By: Leo Brown