How SMAAT times cognitive tasks

How stimulus onsets and response times are measured on the participant's phone, what each trial and each run records, which conditions of a session the app records, which sessions it flags, and what this means for the analysis.

Overview

A stimulus onset is the time the frame that shows the stimulus was drawn. A response time runs from that onset to the moment the finger touched the screen.
Each trial records its response time and the clock it came from; each run records how the times were measured, the phone's model and system, and the conditions of the session (screen brightness, battery saving, interruptions and more).
Runs with possible problems are flagged on the study's Quality tab and in the export. Flags never exclude data; the researcher decides which ones matter.
Tasks run on the phone and do not use the network while they run.

This timing has not yet been validated against an external measurement on real phones. A check with high-speed video (240 frames per second) is planned, and its results will be added to this page. Until then, treat absolute response times with the caution described under Known limits.

What onset and response time mean

Stimulus onset

When a task shows a stimulus, it changes the screen and waits for two animation frames. By the second frame, the frame showing the stimulus has been committed, and the time of that frame is the onset. Taking the time of the screen change instead would place the onset up to a frame too early. Tasks with pictures (Approach-Avoidance, Dot-probe, IAT, Picture Rating) wait until the picture reports that it has loaded before they show it and take the onset; a picture that never reports it is shown after a time limit, and the trial records that.

The onset can only fall on a frame, so its precision is bounded by the screen's refresh rate: 16.7 ms at 60 Hz, 8.3 ms at 120 Hz. Each run stores the phone's typical frame interval.

Response time

Responses are taken at touch-down, when the finger touches the screen, not when it lifts. The time is the touch's own timestamp from the operating system, which removes the delay between the touch and the moment the app handles it. Onsets and touches are on the same clock. When a touch timestamp is not plausible (later than the moment the app handles it, or more than a second before), the app uses the time it handled the touch instead and records that it did (rtSource js instead of native).

Requested and measured durations

Where a duration is part of the procedure, the task records the duration it asked for and the one it measured: the stop-signal delay in the Stop-Signal Task, and the stimulus timing in the Cued CPT, Dot-probe and N-back tasks. The difference shows how closely the phone kept to the procedure.

What each trial and each run records

Each trial records its phase (practice or live), whether the answer was correct or missing, the response time (rt, in ms, to 0.1 ms) and its clock (rtSource), plus the fields of the task. In the long export these are the columns TrialPhase, TrialCorrect, TrialOmission, TrialRT, TrialRTSource and TrialDetails.

Each run records the summary scores, the duration, and timing: how onsets and responses were taken, the phone's typical frame interval and how many responses each clock timed (column TaskTiming). From the app version with session conditions on, it also records session, described below (column TaskConditions, with the most used values in their own columns).

Each response to a survey records the phone's operating system and version, its model and manufacturer, and the app version at the time (columns Platform, OSVersion, DeviceModel, Manufacturer and AppVersion).

Session conditions

When a task starts, the app records what the phone reports about its state, and while the task runs it records interruptions and the frames at each onset. All of this is stored with the run as session. A value the phone does not report is left empty.

Condition	Field	What it is
Screen brightness	`brightness (0–1), brightnessMode`	On iPhone, the position of the brightness slider. On Android, the brightness setting, which the app can read only when adaptive brightness is off; with adaptive brightness on, the level is empty and brightnessMode is “automatic”. The value is a setting, not the light the screen gives off: the same setting is brighter on some phones than on others.
Battery saving and battery	`lowPowerMode, batteryLevel, batteryState`	Low Power Mode on iPhone, Battery Saver on Android. Battery saving can lower the screen's refresh rate and the processor's speed.
Screen size and orientation	`windowWidth, windowHeight, screenWidth, screenHeight, pixelRatio, orientation`	Sizes in the points the task layouts use; the pixel ratio converts them to pixels. The app runs in portrait.
Refresh rate	`timing.frameIntervalMs; onsetFrames`	The typical interval between the app's frames when the task opened (16.7 ms at 60 Hz, 8.3 ms at 120 Hz), and the interval at each stimulus onset (see Late frames below).
Font size and reduce motion	`fontScale, reduceMotion`	The system text size (1 is the default) and the accessibility setting that reduces animations.
Network	`network.type, network.connected, network.cellularGeneration`	Wi-Fi, cellular (with 4G or 5G where the phone says), or none, as the phone last reported it to the app. Tasks do not use the network (see below).
Local time	`localTime, timeZone`	The time on the phone when the task started, with its offset from UTC, and the time zone.
Interruptions	`interruptions: kind, atMs, durationMs`	Every time the app left the foreground while the task ran: “background” (another app, the home screen, the lock button, a full-screen call), “inactive” (iPhone: Control Center, Notification Center, the app switcher, a system alert) or “focusLost” (Android: the notification shade or a system dialog), with when it started (ms after the task started) and how long it lasted. A notification that arrived while the task ran is listed as “notification”, with its time only, never its content.

Not recorded

Whether Do Not Disturb is on: the app does not read it. The checklist below can ask participants to switch it on.
Headphones and volume: no SMAAT task plays sound (the Stop-Signal Task uses a visual signal on purpose, because audio latency on Android phones varies too much), so neither is recorded.
A call or notification shown as a banner while the app stays in front, other than the app's own notifications.
Ambient light and the screen's actual luminance.

Controlling conditions

Two options in the task editor, under Session conditions, set conditions instead of only recording them.

Fixed screen brightness

The app sets the chosen brightness when the task starts and gives the participant's own setting back when it ends. On iPhone, the app also gives it back while the participant is in another app, and sets it again on return. On Android, the setting applies only while the app is in front. The record says whether the brightness was set (brightnessControl), and the run is flagged when it could not be. A fixed setting makes sessions on one phone more alike; it does not make phones alike, because the same setting gives different light on different screens.

Checklist before the task

Before the task, participants see the chosen points in their language, tick what applies and continue. They can continue with points unticked; the record lists the points shown and the points confirmed, and the run is flagged when one was left unticked. The points are:

I am sitting down.
I am holding the phone in one hand.
I am holding the phone with both hands.
Do Not Disturb is on, so no calls or notifications will interrupt me.
I am in a quiet place where nobody will interrupt me.

The app cannot switch on Do Not Disturb itself, so that point tells participants where to find it on their phone.

Quality flags

Each run gets the flags below. They are shown on the study's Quality tab (Data Quality, Cognitive task runs) and exported in the column TaskFlags, separated by semicolons. The wide export has them in the column “[flags]” of each task. Practice trials are not counted for the response rules.

Flag	Rule
Stopped part-way	The participant stopped the task with ✕ (or Android's back button) before the end. The trials so far are kept, and TaskStatus says where the task was stopped.
Interrupted	At least one interruption or notification while the task ran. The Personal Cue Photos step opens the phone's camera itself, so its interruptions are recorded but not flagged. The first run of a recording task (Verbal Fluency, List Learning) can be flagged when the phone asks for microphone access.
Late frames	More than 5% of the stimulus onsets fell on a late frame: the interval between the two frames of the onset was at least twice the phone's typical interval and at least 25 ms. The app was busy then, and the stimulus may have appeared later than its recorded onset.
Handler timing	More than 10% of the responses were timed when the app handled the touch, because the touch's own timestamp was not usable. These response times include the delay before the app handled the touch.
Too-fast responses	More than 10% of the timed responses came less than 100 ms after the stimulus onset, too soon to be a reaction to it. The share follows the IAT rule of 10% of responses under 300 ms (Greenwald et al., 2003); the 100 ms limit is the one SMAAT's DCCS scoring uses (Zelazo et al., 2014). Both are screening rules, not a published standard for every task.
Practice not passed	A task with a practice criterion: the participant did not reach it, and the task went on after the last practice attempt.
Checklist incomplete	The participant left at least one point of the checklist before the task unticked.
Brightness not fixed	The study asked for a fixed brightness, and the app could not set it, for example because the participant's app version is older than the option.

Flags are never used to exclude data automatically. Decide before data collection which flags lead to exclusion in your study, ideally in the preregistration, and report how many runs each rule removed. Runs recorded by app versions before session conditions have no session record; flags that need it (Interrupted, Late frames, Checklist incomplete, Brightness not fixed) are then absent, and the others are computed from the trials.

Export columns

The long export has, for every task run: TaskFlags; TaskBrightness (the fixed level when the app set it, otherwise the level at the start); TaskBrightnessFixed; TaskLowPowerMode; TaskBatteryLevel; TaskNetwork; TaskInterruptions (a count); TaskLateFrameShare; and TaskConditions, the whole session record as JSON. The wide export has, per task, [flags], [brightness], [lowPowerMode] and [interruptions].

The network does not affect timing

A task runs entirely on the phone. Pictures from a researcher's image set are downloaded to the phone before the Start button becomes active, and the task then shows the downloaded files; the built-in pictures and word lists are part of the app. Results are saved on the phone and sent when the phone is online; audio recordings and photos are uploaded afterwards in the same way. A slow or missing connection can therefore delay when the data arrive, not when a stimulus appears or how a response is timed. The network type is recorded anyway, so that sessions can be compared.

Known limits

Display and touch latency. A screen shows a frame some time after the app drew it, and a touchscreen reports a touch some time after it happened. Neither delay can be measured by the app, so both are part of every response time. Across 26 smartphones, the two together ranged from 35 to 140 ms (Nicosia et al., 2023).
Response times are too long, by an amount that depends on the device. On touchscreen and keyboard devices, response times were consistently overestimated, and differences in this overestimation between devices may lower the reliability of individual differences, of absolute response times more than of differences between conditions (Pronk et al., 2020). In a large web sample, users of mobile devices, particularly Android phones, had slower measured response times than users of computers (Passell et al., 2021).
Refresh rate. An onset can only be placed on a frame. Phones with 90 or 120 Hz screens may lower the rate, for example with battery saving on, and the run records the rate at the time.
Not yet validated on real phones. See the note at the top of this page.

Analysis advice

Compare within participants where possible: differences between conditions (e.g. Stroop interference, a bias score) and change over time within a person are less affected by device latency than absolute response times (Pronk et al., 2020).
Account for the device: include the phone model (DeviceModel) or the platform as a covariate or a random effect when absolute response times are compared between people. A participant who changes phones during the study shows a new model in the per-response device columns.
Aggregate across sessions. Averages of brief ambulatory cognitive assessments reached high between-person reliability (Sliwinski et al., 2018); a single short session is less reliable.
Use the session conditions as covariates where they are plausible influences, for example brightness for visual tasks or battery saving for timing-sensitive tasks, and fix the brightness when it should not vary.
Decide in advance how to treat flagged runs, and report it. See also Data Quality & Limitations.

References

Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85(2), 197–216. https://doi.org/10.1037/0022-3514.85.2.197
Nicosia, J., Wang, B., Aschenbrenner, A. J., Sliwinski, M. J., Yabiku, S. T., Roque, N. A., Germine, L. T., Bateman, R. J., Morris, J. C., & Hassenstab, J. (2023). To BYOD or not: Are device latencies important for bring-your-own-device (BYOD) smartphone cognitive testing? Behavior Research Methods, 55(6), 2800–2812. https://doi.org/10.3758/s13428-022-01925-1
Passell, E., Strong, R. W., Rutter, L. A., Kim, H., Scheuer, L., Martini, P., Grinspoon, L., & Germine, L. (2021). Cognitive test scores vary with choice of personal digital device. Behavior Research Methods, 53(6), 2544–2557. https://doi.org/10.3758/s13428-021-01597-3
Pronk, T., Wiers, R. W., Molenkamp, B., & Murre, J. (2020). Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices. Behavior Research Methods, 52(3), 1371–1382. https://doi.org/10.3758/s13428-019-01321-2
Sliwinski, M. J., Mogle, J. A., Hyun, J., Munoz, E., Smyth, J. M., & Lipton, R. B. (2018). Reliability and validity of ambulatory cognitive assessments. Assessment, 25(1), 14–30. https://doi.org/10.1177/1073191116643164
Zelazo, P. D., Anderson, J. E., Richler, J., Wallner-Allen, K., Beaumont, J. L., Conway, K. P., Gershon, R., & Weintraub, S. (2014). NIH Toolbox Cognition Battery (CB): Validation of executive function measures in adults. Journal of the International Neuropsychological Society, 20(6), 620–629. https://doi.org/10.1017/S1355617714000472