You are here

Conducting Online Research on Amazon Mechanical Turk and Beyond

Conducting Online Research on Amazon Mechanical Turk and Beyond

May 2020 | 296 pages | SAGE Publications, Inc

Conducting Online Research on Amazon Mechanical Turk® and Beyond, written by Leib Litman and Jonathan Robinson, provides both students and experienced researchers with essential information about the online platforms most often used for social science research. This insightful and accessible text answers common questions like, “How do I maintain data quality in online studies?,” “What is the best way to recruit hard-to-reach samples?” and “How can researchers navigate the ethical issues that are unique to online research?” Drawing on their experiences as the founders of CloudResearch (formerly TurkPrime), the authors provide information that guides new users planning their first online studies and engages even the most experienced researchers with detailed discussions about the challenges of online research. The book begins with an overview of Amazon’s Mechanical Turk and its rapid rise within academic research. Then, the authors describe how to set up an MTurk study with screenshots that walk readers through the steps of creating an account, designing a study, collecting data, and using third-party applications to enhance MTurk’s functionality. Later chapters provide readers with a detailed understanding of the MTurk environment and use data from hundreds of thousands of participants and tens of millions of completed tasks to dive into issues like participant demographics, sources of sampling bias, and the generalizability of findings from MTurk. Finally, the book explores the benefits of using other online platforms as a complement to MTurk and the ethical issues that are unique to conducting research with online participant platforms. Throughout the book, the authors share hands-on advice and best practices, such as those for conducting longitudinal studies or carrying out complex studies. Altogether the mix of data, insight, and advice make this book an essential resource for researchers who want to understand the online environment and the most effective ways to conduct research online. 



Chapter 1: Introduction
A Scientific Revolution in the Making

A Brief History of Online Research in the Social and Behavioral Sciences: From HTML 2.0 to Mechanical Turk

The Use of Online Samples in Applied Behavioral Research

Amazon Mechanical Turk

Leib Litman, Cheskie Rosenzweig, Jonathan Robinson
Chapter 2: The Mechanical Turk Ecosystem
How Quality is Maintained

Reputation Mechanism

Selectively Recruiting Specific Workers

Protections for Workers

Communicating with Workers

A Worker’s Perspective

Worker Communities

Chapter 3: Conducting a Study on Mechanical Turk
Sample Project

Setting up a Requester Account on Mechanical Turk

Creating a HIT

The ‘Design Layout’ tab

Monitoring Progress on the Requester’s Dashboard

When a Worker Runs Out of Time

Sample Study Results

Conducting Follow-up Studies Using Requester-Issued Qualifications

Appendix A: Checklist for best practices of setting up a Mechanical Turk HIT

Chapter 4: API and Third Party Apps
Third Party API-based Platforms

Common Uses for API Scripts and Third Party API-based Apps


Jesse Chandler, Gabriele Paolacci, David Hauser
Chapter 5: Data Quality Issues on MTurk
Defining and Measuring Data Quality

Measuring Individual Participant Data Quality

Causes of and Cures for Poor Data Quality

Concluding Thoughts

Chapter 6: Who are the Mechanical Turk Workers?
Sources of Data

Location of Workers in the US

Demographics of Mechanical Turk

Chapter 7: Sampling Mechanical Turk Workers: Problems and Solutions
Sampling on Mechanical Turk

Sources of Sampling Bias

The Problem of Superworkers

Time-of-day Effects

Pay Rate


Sampling Best Practices

Chapter 8: Data Representativeness of Mechanical Turk Samples
Representativeness, surveys, and survey sampling

The methodology of survey sampling

Mechanical Turk as a sampling frame

The fit-for-purpose framework

Chapter Overview:

Chapter 9: Conducting Longitudinal Research on Amazon Mechanical Turk
Why Longitudinal Research?

Retention, Longitudinal Research, and MTurk

Case Studies

Best practices for longitudinal research

Chapter 10: Beyond Mechanical Turk: Using Online Market Research Platforms
Limitations of MTurk

Online probability-based panels

Online market research platforms

Overall comparisons between Mechanical Turk and market research platforms

Chapter 11: Conducting Ethical Online Research: A Data-Driven Approach
Historical background

Risk of harm in online research

Research on sensitive topics

A deeper dive into controversial and complex issues

Economics of Mechanical Turk: Considerations for setting wages

Setting wages: Considerations of ethics and methodology

Considerations for rejecting, blocking and disqualifying workers.

Practical advice for requester/worker interactions


Appendix A

“This is the best practical guide I have seen written in any methodologically written book!” 

Michaela Porubanova
Farmingdale College

“Great overview of literature developing in several fields; it's hugely valuable to have all of this in one place.” 

Kevin Munger
New York University

“They cover the whole gamut of history/background, practical concerns and scholarly concerns.” 

See-yeon Hwang
Sam Houston State University

"I am looking forward to being able to hand this to my students and know they'll be ready to get started on MTurk and really understand the whole system (i.e., not just the mechanics of getting data, but really "getting" the culture and so on). I would probably also find myself citing this book as it discusses many of the issues I personally research.” 

Alice M. Brawley
Gettysburg College
Key features
  • Step-by-step guidance and screenshots for setting up a study on Amazon Mechanical Turk (MTurk) helps researchers quickly navigate this important research platform
  • Coverage of APIs and scripts help researchers more efficiently and effectively conduct research 
  • Challenging designs like longitudinal research and experimentation offer new avenues for researchers to use MTurk
  • Data collected via the CloudResearch database offers researchers new insights into MTurk workers and research participants 
  • Aggregated data on MTurk workers including demographic information, sampling strategies, and compensation helps researchers make informed decisions on the methods and mechanics of their studies
  • Detailed information on data quality issues help researchers avoid some of the most common pitfalls 
  • A full chapter on ethics begins the conversation on ethical implications of working with MTurk for research
  • Description of a "fit-for-purpose" framework that describes theoretical and practical considerations relevant for non-probability sampling

For instructors

Select a Purchasing Option

Rent or Buy eBook
ISBN: 9781506391120

ISBN: 9781506391137

This title is also available on SAGE Research Methods, the ultimate digital methods library. If your library doesn’t have access, ask your librarian to start a trial.