Thanks to advances in web-based technology and the vast number of Internet users seeking to participate in research activities, more and more experiments, surveys, and qualitative studies are being conducted online. Recognizing the significance of this shift, the BRL hopes to help researchers explore and leverage the various online research tools and services available today. While we encourage researchers at MIT to conduct lab and online studies through the BRL, we understand that other avenues of data collection may be more suitable for certain projects. This webpage introduces two of these avenues: research panels (which can be used for both online and offline studies, though the former usage seems more common) and crowdsourcing platforms.
We begin by describing how research panels are created and managed, before moving on to name a few panel vendors with which the BRL has established contact. Next, we explain what crowdsourcing is and provide information on several crowdsourcing platforms frequently used by academic researchers. We then make a comparison between research panels, crowdsourcing platforms, and the BRL by listing the advantages each of these participant sources offers. Finally, we present a knowledge base that contains helpful resources on a wide range of topics pertaining to web-based research.
Please contact the BRL Coordinator at email@example.com or at (617) 253-1959 if you have any questions or requests regarding online data collection. In particular, please reach out to the BRL Coordinator if you need the following:
- Guidance on choosing the right panel vendor or crowdsourcing platform for your project
- Assistance in contacting a panel vendor to obtain information or to launch a study
- Technical support in using a particular crowdsourcing platform
- General advice on conducting online studies
A research panel is comprised of individuals who have expressed willingness to participate in research activities conducted by or through a particular company or organization. Before taking a look at how research panels operate, we would like to clarify some of the terminology associated with this topic:
- The terms “research panels” and “online panels” are often used interchangeably, but the former is in fact broader in scope. That is, members of a research panel may be asked to complete research activities online, by phone, in person, or through other communication channels. On the other hand, members of an online panel should, by definition, only be able to take part in research activities over the Internet.
- While most research panels are business-owned, some are managed by nonprofits and universities. Therefore, we will use the term “panel vendors” (rather than “panel companies”) when referring to these entities.
- Members of a research panel are often called “panelists”.
There are two main types of research panels: probability-based panels and non-probability panels. Members of probability-based panels are recruited using random sampling techniques, such as random digit dialing (RDD) and addressed-based sampling (e.g., randomly selecting addresses from a database maintained by the United States Postal Service). The goal of these methods is to ensure that all individuals within the target population have a known chance of being selected, which in turn allows researchers to determine, at a certain confidence level, the maximum difference between the data obtained from the sample and the actual values for the whole population (i.e., the margin of error). To boost recruitment response rates, probability-based panel vendors may send multiple promotional letters, make follow-up calls, or offer prepaid incentives to prospective panelists. Furthermore, if a panelist does not have the necessary resources (e.g., a phone, a computer, or Internet access) to participate in research activities, a vendor may provide such resources to the panelist for free.
Whereas membership in probability-based panels is by invitation only, anyone with Internet access can sign up for non-probability panels — a process that typically takes place on a panel vendor’s website. Apart from getting sign-ups through organic web traffic, some vendors create online advertisements to attract panelists, and some even recruit individuals directly from other panels. Since non-probability panels are formed through convenience sampling rather than random sampling, they may not be representative of the general population. Although results from studies conducted using non-probability panels may have lower external validity (i.e., generalizability) than research findings based on probability samples, many academic researchers have turned to non-probability panels in recent years because of their lower prices, larger panel sizes, and shorter turnaround times.
Most panel vendors ask newly registered panelists to provide detailed demographic, psychographic, and behavioral information by completing a “profiling survey”, which gives researchers the ability to limit the target audience of their studies to individuals with certain characteristics. Vendors differ from one another in how they recruit panelists for specific research activities: some notify panelists by email whenever a new opportunity becomes available; others allow panelists to log into an online portal where they can view a list of studies they are eligible for; and still others use a routing system that automatically directs panelists to web surveys where they are “needed the most”. After completing a study, panelists are usually rewarded with points, which can be redeemed for cash, gift cards, airline miles, consumer goods, and so on.
There are numerous research panels in the U.S. today, with sizes ranging from a few hundred to several million members. While the majority of panel vendors specialize in market research or public opinion polling, some also cater to academic researchers. Besides conducting online surveys, many vendors offer researchers the option to gather data by other means, including:
- Web-based experiments
- Mail surveys
- Traditional phone surveys/interviews
- Interactive voice response (IVR) surveys/interviews (i.e., automated phone surveys/interviews)
- Computer-assisted telephone interviewing (CATI)
- Computer-assisted web interviewing (CAWI)
- Computer-assisted personal interviewing (CAPI)
- Traditional face-to-face interviews
- Online focus groups
- In-person focus groups
Researchers sometimes employ more than one of the above methods for a single study — an approach known as “mixed-mode” or “multi-mode” data collection. For instance, a researcher may allow panelists to complete a survey online, by phone, or by mail in order to increase response rates and sample representativeness. Another example of mixed-mode data collection would be when a survey researcher administers sensitive questions online and non-sensitive questions over the phone so that panelists might be less inclined to give socially desirable responses.
Typically, when a panel vendor receives a new research request, a project manager would set up an initial meeting or phone call with the researcher to discuss the details of the proposed study. The project manager would then provide a quote of how much the research project is expected to cost. Once the researcher accepts the quote, the project manager would begin recruiting participants and continue to work with the researcher as the study progresses.
The cost of conducting a study through a panel vendor is mainly determined by the following factors:
- Desired sample size — The number of panelists the researcher hopes to have in the final sample.
- Eligibility criteria — The requirements panelists must meet to qualify for the study. Stricter requirements usually result in higher costs.
- Study duration — The amount of time each panelist is likely to spend on the research activity.
- Mode of data collection — The data collection method(s) the researcher plans to use. It is generally cheaper to conduct studies online than by phone, by mail, or in person.
- Add-on services — Many panel vendors not only assist researchers with data collection, but also provide a range of other services, such as research design consultation, survey programming, and data analysis. Researchers can use these services by paying additional fees.
The BRL has built a connection with three popular panel vendors: Dynata, NORC, and Qualtrics. Details about each of these vendors can be found below.
We especially encourage researchers to conduct studies through Dynata, our preferred panel vendor. As a result of our partnership with the company, MIT researchers can now enjoy a 15% discount when using the Dynata panel for online studies. Please contact us if you would like to take advantage of this special offer or learn more about Dynata’s pricing.
Dynata is one of the world’s leading providers of first-party data contributed by consumers and business professionals. With a reach that encompasses 60+ million people globally and an extensive library of individual profile attributes collected through surveys, Dynata is the cornerstone for precise, trustworthy quality data. The company has built innovative data services and solutions around its core first-party data offering to bring the voice of the customer to the entire marketing spectrum, from market research to marketing and advertising. Dynata serves nearly 6,000 market research agencies, media and advertising agencies, consulting and investment firms, and healthcare and corporate customers in North America, South America, Europe, and Asia-Pacific.
Dynata also supports a variety of telephone-based data collection methods, such as RDD, mobile-only RDD, directory sampling, demographic targeting, and CATI. Other research services offered by Dynata include:
- Mobile surveys, mail surveys, and in-person interviews
- Survey design, hosting, and programming
- Data processing, analysis, and reporting
- Self-service online samples
Dynata is committed to delivering high-quality data to researchers. It has adopted a series of quality control measures designed to verify panelist identity, flag duplicate accounts, and detect satisficing behavior (e.g., speeding, straight-lining) in web surveys.Visit Website
Crowdsourcing is the recruitment of a large, often unassociated group of individuals to collectively undertake a project, typically over the Internet. These individuals may be asked to generate ideas, process data, create content, solve complex problems, or, as discussed below, participate in online research studies. Much like how companies use online job boards to advertise employment opportunities, one can visit certain websites to recruit Internet users for different types of tasks. These websites, known as crowdsourcing platforms, give researchers easy, on-demand access to prospective participants from all around the world. This section provides an overview of four crowdsourcing platforms: Amazon Mechanical Turk (MTurk), CloudResearch (formerly TurkPrime), psiTurk, and Prolific.
Mechanical Turk (MTurk) is a widely used crowdsourcing platform developed by Amazon in 2005. The website features a large number of “human intelligence tasks” (HITs), which are usually tasks that humans can perform more accurately or efficiently than computers. Common types of HITs include categorizing images, transcribing media files, extracting information from websites, and recording the contents of receipts.
A “requester” is an individual, business, or organization that creates HITs on MTurk. Many companies now use MTurk regularly to outsource their services, collect data for machine learning, solicit opinions from potential customers, and so on. In recent years, MTurk has also become increasingly popular among researchers in the social sciences.
People from all around the world can sign up to be MTurk “workers” (or “Turkers”) and complete HITs for monetary compensation. However, not everyone who signs up is granted access to MTurk, as all worker accounts must first be approved by Amazon. Over time, a small portion of users become “master workers” — that is, workers who have “consistently demonstrated a high degree of success in performing a wide range of HITs across a large number of requesters”. Amazon has not disclosed the specific criteria it uses for determining who can receive the master worker designation, or even who can join the workforce in the first place. When setting up HITs on MTurk, a requester can target a subset of the worker population based on location, performance history, master worker status, and other “qualifications” provided by the system or previously assigned by the requester.
Each HIT on MTurk represents an individual task that can typically be done by a human rather quickly (e.g., labeling an image, finding the address of a restaurant). However, some HITs, such as transcribing an audio track or answering a survey, may take longer to complete. In most cases, requesters would create a “project” (also known as a “HIT group”, a “HIT type”, or a “batch”) that contains many HITs of the same kind. Furthermore, a single HIT can have multiple “assignments”. This allows more than one worker to complete each HIT, which is useful to requesters hoping to seek consensus or different viewpoints on certain subjects.
Requesters have the option to post HITs and manage workers through MTurk’s graphical user interface (GUI), command line interface (CLI), or application programming interface (API). The GUI, which can be accessed by logging into the MTurk requester site, enables users to build HITs visually from a set of HTML templates. If a requester hopes to conduct a study hosted on an external website (e.g., Qualtrics), he or she can simply create a HIT in the GUI with a link to the study. By contrast, the CLI requires requesters to interact with MTurk programmatically. Although the GUI is easier to use, the CLI offers more features, including the ability to add “qualification tests” that workers must pass before they can complete certain HITs. Finally, requesters can use the API to perform all the operations available in the GUI and the CLI, as well as some additional functions, such as emailing workers. One of the main advantages of the API is that it can be integrated into business applications and back-end systems, which allows companies to automatically generate HITs, process data, pay workers, etc.
Amazon gives requesters the freedom to decide how much workers can earn for each HIT and to deny payment for incomplete or unsatisfactory work. However, a requester who frequently underpays workers or rejects HITs will likely acquire a poor reputation on popular MTurk forums such as Turkopticon and Turker Nation, which may in turn lead to lower HIT completion rates. After receiving payment on MTurk, workers can transfer their earnings to a bank account or to an Amazon.com gift card. Aside from worker remuneration, Amazon charges requesters a 20% service fee, plus an additional 20% fee for HITs with 10 or more assignments. Requesters must also pay extra fees for recruiting master workers and for posting HITs with certain qualifications.
Please be sure to explore the knowledge base at the bottom of this webpage for MTurk-related tutorials, support documentation, and articles. In addition, we encourage you to utilize the MTurk sandbox, which is a simulated environment where requesters can create HITs without spending any real money and preview the HITs from the perspective of a worker.Visit Website
Panel vs. Crowdsourcing vs. BRL
At this point, you may still be uncertain whether you should use a research panel or a crowdsourcing platform for your study. More broadly, you may be wondering whether your study should take place online or at the BRL. In this section, we help researchers choose between research panels, crowdsourcing platforms, and the BRL by discussing the main strengths of each option. Please note that individuals in the BRL participant pool can be recruited for both lab and online studies; however, as explained below, conducting research in a lab environment is preferable in certain situations.
Reasons to Use Research Panels
- Range of research services — In contrast with the self-service nature of crowdsourcing platforms, panel vendors often have the personnel who can help researchers program surveys and experiments, collect and process data, and perform other research administration tasks. This allows researchers to focus on areas such as study design and in-depth data analysis without having to personally carry out research. The BRL’s scope of services currently falls between that of a panel vendor and a crowdsourcing platform. Specifically, the BRL is able to offer researchers advisory and logistical support, but is not directly involved in the execution of research activities (though we have plans to expand our services in the near future).
- Modes of data collection — Before online survey and experiment platforms were developed, panel vendors collected data for their clients by phone, by mail, or in person. Today, many vendors continue to support these traditional modes of data collection. This gives researchers the opportunity to conduct mixed-mode studies or use data collection methods that are difficult (if not impossible) to implement through crowdsourcing platforms or the BRL participant pool.
- Sample representativeness — Probability-based panels, which use random sampling techniques to recruit panelists, can often provide nationally representative samples. On the other hand, the demographic composition of non-probability panels, crowdsourced labor, and the BRL participant pool may differ from that of the general population, as is the case with any convenience sample. Nonetheless, these participant sources are all considerably more diverse than university subject pools consisting solely of undergraduate students. Researchers may also be able to enhance the external validity of their studies by employing methods such as quota sampling and statistical weighting. In addition, researchers should make an effort to increase the response rates of their studies in order to minimize any systematic differences between participants and non-participants.
Reasons to Use Crowdsourcing Platforms
- Cost — Crowdsourcing platforms provide a way for individuals and organizations to collect data at relatively low costs. Many companies use MTurk to recruit workers for short, repetitive tasks that pay anywhere between $0.01 and $1. Lengthier tasks, such as surveys and experiments posted by academic researchers, typically offer rewards ranging from $1 to $5. The BRL urges MIT researchers to pay participants no less than the federal minimum wage (prorated based on the length of a study), despite the fact that minimum wage laws do not apply to crowdsourced workers. Even when a researcher compensates participants adequately and takes all additional costs (e.g., service fees) into account, it is usually still cheaper to conduct studies through crowdsourcing platforms than through panel vendors or the BRL.
- Speed — The constant supply of participants on crowdsourcing platforms (especially MTurk) enables researchers to collect all the data they need in a matter of days or even hours. Although some non-probability panel vendors may be able to provide data at a comparable speed, most panel studies take longer to complete. As for studies conducted at the BRL, data collection times can range from less than a week to several months depending on the availability of the lab, the eligibility requirements of the study, the desired sample size, the number of participants allowed in each lab session, and other factors.
- Flexibility — Researchers can easily manage their studies and participants through the various options and features available on crowdsourcing platforms. For instance, MTurk users have the freedom to create custom qualification tests, reject low-quality submissions, change the properties of a task after it is published, and contact participants at any time. Due to technical and policy constraints, researchers may not be able or permitted to perform these types of actions when conducting studies through panel vendors or the BRL.
Reasons to Use the BRL
- Level of standardization/supervision — Even though it may be more convenient to collect data online than in person, running studies in a lab setting provides several unique benefits. First, participants are less likely to multitask or be distracted by their surroundings when a study takes place in a lab. While attention checks (also known as instructional manipulation checks or IMCs) can be used to identify inattentive participants in online studies, there is a lack of consensus in the academic community as to what constitutes an effective attention check and how researchers should handle data from inattentive participants. Second, by having participants complete surveys and experiments on BRL computers, researchers can ensure that their study results are not affected by extraneous variables such as the particular device, operating system, or web browser a participant is using. Finally, the presence of research personnel in a lab may deter participants from engaging in dishonest behavior, which includes searching for factual information on the Internet during a study.
- Participant naïveté — As the usage of crowdsourcing platforms and online panels for data collection has grown in recent years, so has the number of “professional participants” who spend a significant portion of their time working on MTurk and various other websites. These individuals present a challenge to researchers who do not want to recruit participants with prior knowledge of (or exposure to) certain experimental paradigms. BRL participants tend to be less familiar with such paradigms, as most of them do not actively take part in studies on crowdsourcing platforms or panel websites. Furthermore, when setting up a study on Sona, researchers have the option to exclude participants who have completed studies of a similar nature in the past, even if those studies were carried out by other researchers.
- Feasibility — Some researchers choose to conduct studies at the BRL purely for practical reasons. For example, studies that require the use of locally installed software (e.g., z-Tree) or special devices (e.g., professional recording equipment) can only take place in a lab environment. The BRL is also better suited for studies involving group discussions, cooperative tasks, and other types of real-time interactions between participants. As web-based behavioral research tools continue to emerge and evolve, however, many technological and logistical limitations of online data collection may soon be overcome.
We have compiled the knowledge base below to help researchers gain a better understanding of online research tools, methods, and trends. The knowledge base contains three types of resources:
- Scholarly articles — Peer-reviewed articles published in academic journals. All of these articles are accessible to MIT-affiliated individuals.
- Web articles — News articles, blog posts, and webpages, as well as any research reports not published in academic journals.
- Tutorial/documentation — Instructions on how to use a specific research tool, such as MTurk.
All items in the knowledge base were selected and reviewed by the BRL Coordinator, who adds new entries to the system on a regular basis. Since we hope to highlight the latest developments and findings in the domain of web-based research, we have mainly chosen to include content that was created or updated within the past five years.