Overview

This section provides a conceptual overview of how NivTurk operates. See the following sections for a more technical explanation.

NivTurk workflow

At its core, the NivTurk application functions to guide participants through a series of webpages in sequential fashion. The order of these pages is as would be expected. By default, a new participant will see the following pages in order:

1. Consent form: where participants are informed about the details of the study and are given the choice to accept or decline participation.
2. Alert page: where participants are informed about the mechanics of participation (e.g. not to refresh the experiment page).
3. Experiment page: where participants actually view and complete the experiment.
4. Completion page: where participants receive their completion code and/or are redirected back to the recruitment platform (e.g. Prolific, MTurk).

Of course, this workflow may be broken at any point in the chain. For example, a participant may decline to consent to participate. Alternatively, a participant may try restarting the experiment when they were told not to. In such an event, participants may instead be directed to an error page with one of a number of error messages.

Record keeping

NivTurk keeps a record of every participant that starts a study (i.e. accesses the consent page). By default, this record includes technical information that is likely of use to an experimenter, including worker ID (i.e. their Prolific or MTurk ID); computer operating system; browser type and version; IP address; and a timestamped log of every webpage access. These records are written to disk that are stored in the metadata folder under the corresponding worker ID. (Note: NivTurk does not maintain a database of user activity, but instead relies on these files and fast I/O operations.)

NivTurk keeps track of participants through two pieces of information: their worker ID (e.g. their Prolific or MTurk ID) and an experiment-specific browser cookie automatically generated by NivTurk. Through these pieces of information, NivTurk is able to prevent re-participation in previously started or completed experiments. (Of course, this can be relaxed if desired as in longitudinal studies.)

Importantly, at the start of a study, every participant is assigned a random alphanumeric subject ID. At the end of an experiment, a participant’s behavioral data is saved under this anonymized subject ID. Thus, participants’ behavioral data is deidentified by default (i.e. does not contain any identifiable information like worker ID) and can only be linked back to a participant via their metadata file.

If the metadata files are destroyed, there is no way of linking a participant to their behavioral data file. Only delete metadata files after all participants have been paid and there is no further need to identify participants.

Data saving

By default, participants’ data are only saved at one of two points: (1) at the successful completion of an experiment, or (2) on an attempted page close or refresh during an experiment. In other words, NivTurk does not write data to disk after every trial or experiment section.

This is by design and motivated by two principles. First, we wanted to minimize the number of calls made to a server at any point in time, so as not to create lag. Second, in online experiments, data is cheap and incomplete datasets are almost always expendable. That said, NivTurk is flexible and can be made to write data to disk as often as needed.

At the successful conclusion of an experiment, the behavioral data is written to the data folder under a participant’s deidentified subject ID. On a page close or refresh, the incomplete behavioral data is written to the incomplete folder under the same ID.