Online Studies

A research panel is comprised of individuals who have expressed willingness to participate in research activities conducted by or through a particular company or organization. Before taking a look at how research panels operate, we would like to clarify some of the terminology associated with this topic:

• The terms “research panels” and “online panels” are often used interchangeably, but the former is in fact broader in scope. That is, members of a research panel may be asked to complete research activities online, by phone, in person, or through other communication channels. On the other hand, members of an online panel should, by definition, only be able to take part in research activities over the Internet.
• While most research panels are business-owned, some are managed by nonprofits and universities. Therefore, we will use the term “panel vendors” (rather than “panel companies”) when referring to these entities.
• Members of a research panel are often called “panelists”.

There are two main types of research panels: probability-based panels and non-probability panels. Members of probability-based panels are recruited using random sampling techniques, such as random digit dialing (RDD) and addressed-based sampling (e.g., randomly selecting addresses from a database maintained by the United States Postal Service). The goal of these methods is to ensure that all individuals within the target population have a known chance of being selected, which in turn allows researchers to determine, at a certain confidence level, the maximum difference between the data obtained from the sample and the actual values for the whole population (i.e., the margin of error). To boost recruitment response rates, probability-based panel vendors may send multiple promotional letters, make follow-up calls, or offer prepaid incentives to prospective panelists. Furthermore, if a panelist does not have the necessary resources (e.g., a phone, a computer, or Internet access) to participate in research activities, a vendor may provide such resources to the panelist for free.

Whereas membership in probability-based panels is by invitation only, anyone with Internet access can sign up for non-probability panels — a process that typically takes place on a panel vendor’s website. Apart from getting sign-ups through organic web traffic, some vendors create online advertisements to attract panelists, and some even recruit individuals directly from other panels. Since non-probability panels are formed through convenience sampling rather than random sampling, they may not be representative of the general population. Although results from studies conducted using non-probability panels may have lower external validity (i.e., generalizability) than research findings based on probability samples, many academic researchers have turned to non-probability panels in recent years because of their lower prices, larger panel sizes, and shorter turnaround times.

Most panel vendors ask newly registered panelists to provide detailed demographic, psychographic, and behavioral information by completing a “profiling survey”, which gives researchers the ability to limit the target audience of their studies to individuals with certain characteristics. Vendors differ from one another in how they recruit panelists for specific research activities: some notify panelists by email whenever a new opportunity becomes available; others allow panelists to log into an online portal where they can view a list of studies they are eligible for; and still others use a routing system that automatically directs panelists to web surveys where they are “needed the most”. After completing a study, panelists are usually rewarded with points, which can be redeemed for cash, gift cards, airline miles, consumer goods, and so on.

There are numerous research panels in the U.S. today, with sizes ranging from a few hundred to several million members. While the majority of panel vendors specialize in market research or public opinion polling, some also cater to academic researchers. Besides conducting online surveys, many vendors offer researchers the option to gather data by other means, including:

• Web-based experiments
• Mail surveys
• Traditional phone surveys/interviews
• Interactive voice response (IVR) surveys/interviews (i.e., automated phone surveys/interviews)
• Computer-assisted telephone interviewing (CATI)
• Computer-assisted web interviewing (CAWI)
• Computer-assisted personal interviewing (CAPI)
• Traditional face-to-face interviews
• Online focus groups
• In-person focus groups

Researchers sometimes employ more than one of the above methods for a single study — an approach known as “mixed-mode” or “multi-mode” data collection. For instance, a researcher may allow panelists to complete a survey online, by phone, or by mail in order to increase response rates and sample representativeness. Another example of mixed-mode data collection would be when a survey researcher administers sensitive questions online and non-sensitive questions over the phone so that panelists might be less inclined to give socially desirable responses.

Typically, when a panel vendor receives a new research request, a project manager would set up an initial meeting or phone call with the researcher to discuss the details of the proposed study. The project manager would then provide a quote of how much the research project is expected to cost. Once the researcher accepts the quote, the project manager would begin recruiting participants and continue to work with the researcher as the study progresses.

The cost of conducting a study through a panel vendor is mainly determined by the following factors:

• Desired sample size — The number of panelists the researcher hopes to have in the final sample.
• Eligibility criteria — The requirements panelists must meet to qualify for the study. Stricter requirements usually result in higher costs.
• Study duration — The amount of time each panelist is likely to spend on the research activity.
• Mode of data collection — The data collection method(s) the researcher plans to use. It is generally cheaper to conduct studies online than by phone, by mail, or in person.
• Add-on services — Many panel vendors not only assist researchers with data collection, but also provide a range of other services, such as research design consultation, survey programming, and data analysis. Researchers can use these services by paying additional fees.

The BRL has built a connection with three popular panel vendors: Dynata, NORC, and Qualtrics. Details about each of these vendors can be found below.

We especially encourage researchers to conduct studies through Dynata, our preferred panel vendor. As a result of our partnership with the company, MIT researchers can now enjoy a 15% discount when using the Dynata panel for online studies. Please contact us if you would like to take advantage of this special offer or learn more about Dynata’s pricing.

Crowdsourcing is the recruitment of a large, often unassociated group of individuals to collectively undertake a project, typically over the Internet. These individuals may be asked to generate ideas, process data, create content, solve complex problems, or, as discussed below, participate in online research studies. Much like how companies use online job boards to advertise employment opportunities, one can visit certain websites to recruit Internet users for different types of tasks. These websites, known as crowdsourcing platforms, give researchers easy, on-demand access to prospective participants from all around the world. This section provides an overview of four crowdsourcing platforms: Amazon Mechanical Turk (MTurk), CloudResearch (formerly TurkPrime), psiTurk, and Prolific.

At this point, you may still be uncertain whether you should use a research panel or a crowdsourcing platform for your study. More broadly, you may be wondering whether your study should take place online or at the BRL. In this section, we help researchers choose between research panels, crowdsourcing platforms, and the BRL by discussing the main strengths of each option. Please note that individuals in the BRL participant pool can be recruited for both lab and online studies; however, as explained below, conducting research in a lab environment is preferable in certain situations.

1. Range of research services — In contrast with the self-service nature of crowdsourcing platforms, panel vendors often have the personnel who can help researchers program surveys and experiments, collect and process data, and perform other research administration tasks. This allows researchers to focus on areas such as study design and in-depth data analysis without having to personally carry out research. The BRL’s scope of services currently falls between that of a panel vendor and a crowdsourcing platform. Specifically, the BRL is able to offer researchers advisory and logistical support, but is not directly involved in the execution of research activities (though we have plans to expand our services in the near future).
2. Modes of data collection — Before online survey and experiment platforms were developed, panel vendors collected data for their clients by phone, by mail, or in person. Today, many vendors continue to support these traditional modes of data collection. This gives researchers the opportunity to conduct mixed-mode studies or use data collection methods that are difficult (if not impossible) to implement through crowdsourcing platforms or the BRL participant pool.
3. Sample representativeness — Probability-based panels, which use random sampling techniques to recruit panelists, can often provide nationally representative samples. On the other hand, the demographic composition of non-probability panels, crowdsourced labor, and the BRL participant pool may differ from that of the general population, as is the case with any convenience sample. Nonetheless, these participant sources are all considerably more diverse than university subject pools consisting solely of undergraduate students. Researchers may also be able to enhance the external validity of their studies by employing methods such as quota sampling and statistical weighting. In addition, researchers should make an effort to increase the response rates of their studies in order to minimize any systematic differences between participants and non-participants.

1. Cost — Crowdsourcing platforms provide a way for individuals and organizations to collect data at relatively low costs. Many companies use MTurk to recruit workers for short, repetitive tasks that pay anywhere between $0.01 and $1. Lengthier tasks, such as surveys and experiments posted by academic researchers, typically offer rewards ranging from $1 to $5. The BRL urges MIT researchers to pay participants no less than the federal minimum wage (prorated based on the length of a study), despite the fact that minimum wage laws do not apply to crowdsourced workers. Even when a researcher compensates participants adequately and takes all additional costs (e.g., service fees) into account, it is usually still cheaper to conduct studies through crowdsourcing platforms than through panel vendors or the BRL.
2. Speed — The constant supply of participants on crowdsourcing platforms (especially MTurk) enables researchers to collect all the data they need in a matter of days or even hours. Although some non-probability panel vendors may be able to provide data at a comparable speed, most panel studies take longer to complete. As for studies conducted at the BRL, data collection times can range from less than a week to several months depending on the availability of the lab, the eligibility requirements of the study, the desired sample size, the number of participants allowed in each lab session, and other factors.
3. Flexibility — Researchers can easily manage their studies and participants through the various options and features available on crowdsourcing platforms. For instance, MTurk users have the freedom to create custom qualification tests, reject low-quality submissions, change the properties of a task after it is published, and contact participants at any time. Due to technical and policy constraints, researchers may not be able or permitted to perform these types of actions when conducting studies through panel vendors or the BRL.

1. Level of standardization/supervision — Even though it may be more convenient to collect data online than in person, running studies in a lab setting provides several unique benefits. First, participants are less likely to multitask or be distracted by their surroundings when a study takes place in a lab. While attention checks (also known as instructional manipulation checks or IMCs) can be used to identify inattentive participants in online studies, there is a lack of consensus in the academic community as to what constitutes an effective attention check and how researchers should handle data from inattentive participants. Second, by having participants complete surveys and experiments on BRL computers, researchers can ensure that their study results are not affected by extraneous variables such as the particular device, operating system, or web browser a participant is using. Finally, the presence of research personnel in a lab may deter participants from engaging in dishonest behavior, which includes searching for factual information on the Internet during a study.
2. Participant naïveté — As the usage of crowdsourcing platforms and online panels for data collection has grown in recent years, so has the number of “professional participants” who spend a significant portion of their time working on MTurk and various other websites. These individuals present a challenge to researchers who do not want to recruit participants with prior knowledge of (or exposure to) certain experimental paradigms. BRL participants tend to be less familiar with such paradigms, as most of them do not actively take part in studies on crowdsourcing platforms or panel websites. Furthermore, when setting up a study on Sona, researchers have the option to exclude participants who have completed studies of a similar nature in the past, even if those studies were carried out by other researchers.
3. Feasibility — Some researchers choose to conduct studies at the BRL purely for practical reasons. For example, studies that require the use of locally installed software (e.g., z-Tree) or special devices (e.g., professional recording equipment) can only take place in a lab environment. The BRL is also better suited for studies involving group discussions, cooperative tasks, and other types of real-time interactions between participants. As web-based behavioral research tools continue to emerge and evolve, however, many technological and logistical limitations of online data collection may soon be overcome.