Abstract

People have never played more video games, and many stakeholders are worried that this activity might be bad for players. So far, research has not had adequate data to test whether these worries are justified and if policymakers should act to regulate video game play time. We attempt to provide much-needed evidence with adequate data. Whereas previous research had to rely on self-reported play behaviour, we collaborated with two games companies, Electronic Arts and Nintendo of America, to obtain players’ actual play behaviour. We surveyed players of Plantsvs.Zombies: Battle for Neighborville and Animal Crossing: New Horizons for their well-being, motivations and need satisfaction during play, and merged their responses with telemetry data (i.e. logged game play). Contrary to many fears that excessive play time will lead to addiction and poor mental health, we found a small positive relation between game play and affective well-being. Need satisfaction and motivations during play did not interact with play time but were instead independently related to well-being. Our results advance the field in two important ways. First, we show that collaborations with industry partners can be done to high academic standards in an ethical and transparent fashion. Second, we deliver much-needed evidence to policymakers on the link between play and mental health.

1. Introduction

Video games are an immensely popular and profitable leisure activity. Last year, the revenues of the games industry were larger than the film industry’s [] and the number of people who report playing games has never been higher []. Across the globe, the rise of games as a dominant form of recreation and socializing has raised important questions about the potential effect of play on well-being. These questions concern players, parents, policymakers and scholars alike: billions of people play video games, and if this activity has positive or negative effects on well-being, playing games might have worldwide health impacts. Therefore, empirically understanding how games might help or harm players is a top priority for all stakeholders. It is possible games are neutral with respect to health and enacting policies that unnecessarily regulate play would restrict human rights to play and freedom of expression [3]. Decisions on regulating video games, or promoting it as a medium for bolstering health, thus come with high stakes and must not be made without robust scientific evidence.

Unfortunately, nearly three decades of research exploring the possible links between video games and negative outcomes including aggression, addiction, well-being and cognitive functioning have brought us nowhere near a consensus or evidence-based policy because reliable, reproducible and ecologically valid studies are few and far between (e.g. [4,5]). In recent years, researchers and policymakers have shifted focus from concerns about violent video games and aggression (e.g. ) to concerns about the association between the amount, or nature, of the time people spend playing video games and well-being (e.g. in the UK [7]). In other words, they are interested in the effect of game play behaviours on subjective well-being and by extension mental health. Yet, instead of measuring such behaviour directly, research has relied on self-reported engagement. Historically, this methodological decision has been taken on practical grounds: first, self-report is a relatively easy way to collect data about play. Second, the video games industry has in the past hesitated to work with independent scientists. As time has gone on, it has become increasingly clear that defaulting to self-report is not tenable. Recent evidence suggests self-reports of digital behaviours are notoriously imprecise and biased, which limits the conclusions we can draw from research on time spent on video games and well-being [8,9].

The lack of accurate behavioural data represents a formidable shortcoming that deprives health policymakers of the high-quality evidence they require to make informed decisions on possible regulations to the video games industry [10]. A range of solutions have been proposed including active and passive forms of online engagement [11] and measuring engagement using device telemetry (i.e. logged game play) [12,13]. Therefore, there is a need for directly measured video game behaviour to inform policymakers. To obtain such data, researchers must collaborate, in a transparent and credible way, with industry data scientists who can record objective measures of video game engagement. In this paper, we detail such a collaboration and report our investigation of the relation between the actual time people devote to playing a game and their subjective sense of well-being. We believe our study addresses the primary impediment to past research, delivers high-quality evidence that policymakers require, and provides a template for transparent, robust and credible research on games and health.

1.1. Video game behaviour

Globally speaking, the most contentious debates surrounding the potential effects of video game engagement are focused on the mental health of players. For example, the American Psychiatric Association did not identify any psychiatric conditions related to video games in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), but it does recommend Internet Gaming Disorder as a topic for further research [14]. The World Health Organization adopted a more definitive approach and included Gaming Disorder in the International Classification of Diseases (ICD-11), emphasizing excessive play time as a necessary component [15]. In sharp contrast, the US Food and Drug Administration recently approved the use of a so-called ‘serious video game’ for treatment of children with attention deficit hyperactivity disorder, providing some evidence that there are mental health benefits of some kinds of play time [16]. These examples illustrate the central role video game engagement plays as a potential public health issue.

Given this, it is critical to understand that the quality of the evidence underlying possible classifications of video game play as potentially psychopathological has been criticized strongly. Many experts have argued that there is insufficient evidence that gaming disorder definitions and diagnostic tools meet clinical standards [15,17–22]. Excessive use has been flagged as a key criterion for many gaming disorder definitions, yet researchers exclusively operationalize excessive use by way of self-reported estimates. This is an important shortcoming, as an increasing number of scholars are now aware that self-reported behaviour is a poor predictor of actual behaviour, particularly for technology use (e.g. [,23,24]). Self-reported video game play is thus an unsuitable proxy of actual video game play—yet researchers and those advising health bodies are depending on self-reports for diagnosis and policy decisions (e.g. [21]).

Although there have been calls for more direct measures of video game behaviour, these efforts have stalled because scientists do not have the resources or access to data necessary for independent scientific research. For example, on the issue of social media use and well-being, a UK parliamentary select committee called on ‘social media companies to make anonymized high-level data available, for research purposes’ in January 2019 [25]. A year later, another committee report on addictive and immersive digital technologies recommended that government ‘require games companies to share aggregated player data with researchers’ [26]. There is a need for collaborations between games companies and independent scientists, but we are unaware of any successful collaborations investigating player well-being. Game developers have in-house expertise in directly measuring video game engagement via telemetry—the automated logging of users’ interaction with content. But so far, efforts have been futile to connect with scientists who have experience in combining such telemetry data with methods that assess subjective well-being (e.g. surveys or experience sampling) and it is not clear if the data, collected for commercial purposes, could be applied to scientific ends.

Collaboration with industry partners not only has the promise to make objective player behaviours accessible for independent analysis; it also provides an opportunity to address a related problem which has plagued games research for decades: a lack of transparency and rigour. Much research in the quantitative social sciences does not share data for others to independently verify and extend findings (e.g. [27]). Sharing resources and data contribute to a more robust knowledge base [28,29]. It also gives other scientists, the public and policymakers the opportunity to better judge the credibility of research [30,31]. A lack of transparency allows selective reporting and thus contributes to unreliable findings that regularly fail to replicate (e.g. [32–34]). Work by Elson & Przybylski [35] showed that this issue arises regularly in research focused on the effects of technology, including in video games research. Carras et al. [19] summarized systematic reviews on gaming disorder and found a high degree of selective reporting in the literature. To increase public trust in their findings, scientists have an obligation to work as transparently as possible, particularly when they collaborate with industry [36]. Greater transparency will provide a valuable tool for informing policy [3.7] and the heated academic debates that surround the global health impacts of games.

1.2. Video game behaviour and well-being

Research and policymakers have been interested in a wide range of mental health outcomes of video game play. Mental health comprises both negative mental health (e.g. depression) and positive mental health. Positive mental health can be further divided into emotional well-being (i.e. the affective component) and evaluative well-being (i.e. the cognitive component) [38]. Nearly all non-experimental studies examining the links between video games and mental health rely on subjective, self-reported estimates of video play time, either by players themselves or by parents. For example, Maras et al. [39] found a sizeable positive correlation between video game play time and depression in a large sample of Canadian adolescents. The focus of research is often on excessive or problematic video game use, routinely reporting positive correlations between problematic video games and mental health problems in both cross-sectional (e.g. [40]) and longitudinal designs (e.g. [41]).

Because self-reported technology use has shown to be a poor proxy of actual behaviour, such associations will necessarily be biased (e.g. [8]). The same caveat holds for research reporting both positive (e.g. [42]) and nonlinear (e.g. [43]) associations between video game play time and psychological functioning. For example, studies suggest that self-reported technology use can lead to both overestimates and underestimates of the association with well-being compared to directly logged technology use [44–46]. Therefore, our scientific understanding of video game effects is limited by our measures. In other words, the true association could be positive or negative, small or large, irrelevant or significant.

A handful of efforts have combined server logs with survey data [47]. However, these studies mainly used a network approach, modelling offline to online dynamics in leadership [48] and friendship formation in games [49]. Studies combining objective play and well-being are lacking. We need accurate, direct measures of play time to resolve the inconsistencies in the literature on well-being and to ensure the study of games and health is not as fruitless as the study of games and aggression [].

Whereas the perceptions of players in recalling their video game play time can introduce bias, a decade of research indicates perceptions of the psychological affordances provided by games are important to player experiences in games. According to self-determination theory, any activity whose affordances align with the motivations of people will contribute to their well-being [50]. Motivations can be intrinsic, driven by people’s interests and values which result in enjoyment, or extrinsic, inspired by rewards or a feeling of being pressured to do an activity. If an activity also satisfies basic psychological needs for competence, relatedness and autonomy, people will find the activity more motivating, enjoyable and immersive—ultimately leading to higher well-being.

The interplay of the affordances of video games, motivation and needs has shown to be important for subjective well-being. If a game satisfies basic needs people will experience more enjoyment and higher well-being [51]. Conversely, if those needs are not met, frustrated, or play is externally motivated, it is associated with lower psychological functioning [52]. In other words, how play time relates to well-being probably depends on players’ motivations and how the game satisfies basic needs. Player experience would thus moderate the association between play time and well-being: if players are intrinsically motivated and experience enjoyment during play, play time will most likely be positively associated with well-being [53,54]. By contrast, when players only feel extrinsic motivation and feel pressured to play, play time might have negative effects on well-being. Such a mechanism aligns well with a recent review that concludes that motivations behind play are likely to be a crucial moderator of the potential effect of play time on well-being [55,56]. However, it is unclear whether such a mechanism only holds true for self-reported play time and perceptions, or whether self-reported perceptions interact with directly measured play time.

1.3. This study

In this study, we investigate the relations between video games and positive mental health, namely affective well-being of players (from here on called well-being). We collaborated with two industry partners, Electronic Arts and Nintendo of America, and applied an approach grounded in an understanding that subjective estimates of play time are inaccurate and the motivational experiences of player engagement are important to well-being. To this end, we surveyed players of two popular video games: Plantsvs.Zombies: Battle for Neighborville and Animal Crossing: New Horizons. Our partners provided us with telemetry data of those players. The data allowed us to explore the association between objective play time and well-being, delivering a much-needed exploration of the relation between directly measured play behaviour and positive mental health. We also explored the role of player motivations in this relation, namely whether feelings of autonomy, relatedness, competence, enjoyment and extrinsic motivation interacted with play time.

In the light of calls for more transparency in the Social Sciences (e.g. [31]), we aimed for a transparent workflow to enable others to critically examine and build upon our work. We, therefore, provide access to all materials, data and code on the Open Science Framework (OSF) page of this project (https://osf.io/cjd6z/). The analyses are documented at https://digital-wellbeing.github.io/gametime/. This documentation and the code have been archived on the OSF at https://doi.org/10.17605/OSF.IO/5EF8H.

2. Method

2.1. Participants and procedure

For this project, we combined objective game telemetry data with survey responses. We did not conduct a priori power analyses. Instead, we followed recent recommendations and aimed to collect as many responses as we had resources for [57]. We surveyed the player base of two popular games: Plants vs. Zombies: Battle for Neighborville (PvZ) and Animal Crossing: New Horizons (AC:NH).

We designed a survey measuring players’ well-being, self-reported play and motivations for play, and discussed the survey structure with Electronic Arts. Electronic Arts (EA) programmed and hosted the survey on Decipher, an online survey platform, and sent invite emails to adult (at least 18 years old) PvZ players in the US, Canada and the UK. The survey was translated to French for French-speaking Canadians. Participants received an invitation to participate in the survey on the email address they had associated with their EA account. The email invited them to participate in a research project titled ‘Understanding Shifting Patterns of Videogame Play and Health Outcomes’. Participants were informed that the aim of the study was to investigate how people play video games and how they feel over time. We also informed them that Electronic Arts would link their survey responses to their play data. Further, the study information explained that the research team would act independently of Electronic Arts in data analysis and scientific reporting. We obtained ethical approval from our institute (SSH_OII_CIA_20_043) and all respondents gave informed consent. Participants could halt their participation at any time and did not receive compensation for their participation.

Electronic Arts then pulled telemetry game data of players that got invited in the first wave of data collection. They matched telemetry data with survey invitations by a securely hashed player ID. Afterwards, they transferred both the survey and the telemetry datasets to the researchers. Neither dataset contained personally identifiable information, only a hashed player ID that we used to link survey and telemetry data. Electronic Arts sent out the invitations in two waves. The first wave happened in early August 2020 and was sent to 50 000 player (response window: 48 h).¹ We inspected the data quality of their telemetry and survey responses and checked whether the data were suitable to address our research questions. After confirming that the data were suitable and that we could join the telemetry and survey information, Electronic Arts sent out a second wave of invitations to 200 000 PvZ players from the same population at the end of September 2020 (response window: 96 h). In total, 518 PvZ players (approx. 0.21% response rate) finished the survey (M_age = 35, s.d._age = 12; 404 men, 94 women, two other, 17 preferred not to disclose their gender), of whom 471 had matching telemetry data.

For Animal Crossing: New Horizon (AC:NH) players, the procedure was similar. We hosted the survey with formr [58], an open source survey tool, and Nintendo of America sent invitations with survey links to a 342 825 adult players in the US on 27 October. The survey was identical to the one sent to PvZ players except for aesthetic differences. The response window was 7 days and in total, 6011 players responded (1.75% response rate; M_age = 31, s.d._age = 10; 3124 men, 2462 women, 153 other, 88 preferred not to disclose their gender). We then provided the hashed IDs of the survey respondents to Nintendo of America, who sent us the telemetry data for those players, of whom 2756 had telemetry within the two-week window. Neither the survey nor the telemetry data had any personally identifiable information. We followed the same workflow as described above and linked survey responses with play data.

2.2. Measures

2.2.1. Well-being

We assessed well-being with the validated scale of positive and negative experiences (SPANE, [59]), which measures the affective dimension of well-being [38]. We asked respondents to think about how they had been feeling in the past two weeks and report how often they experienced each of six positive and six negative feelings. Respondents could indicate the frequency of experiencing those feelings on a scale from 1 (Very rarely or never) to 7 (Very often or always). We then took the mean of the positive feelings and negative feelings and subtracted the negative affect mean score from the positive affect mean score to obtain a measure of well-being (figure 1).

2.2.2. Player experience and need satisfaction

We assessed player experiences and motivations with the player experience and need satisfaction scale (PENS, [51]), which has recently been validated [60]. We asked respondents to rate items reflecting on when they had been playing PvZ/AC:NH in the past two weeks on a scale from 1 (Strongly disagree) to 7 (Strongly agree). The scale consisted of five subscales. Participants reported their sense of autonomy on three items such as ‘I experienced a lot of freedom in [PvZ/AC:NH]’; their sense of competence on three items such as ‘I felt competent at [PvZ/AC:NH]’; their sense of relatedness on items such as ‘I found the relationships I formed in [PvZ/AC:NH] fulfilling’, but only when they reported to have played with others, either online or in couch co-op, in the past two weeks. They also reported their enjoyment with four items such as ‘I think [PvZ/AC:NH] was fun to play’ and their extrinsic motivation on four items such as ‘I played [PvZ/AC:NH] to escape’.

2.2.3. Self-reported play

Participants also reported how much total time they estimated to have spent playing the game in the past two weeks on two open numerical fields, where they could report hours and minutes (figure 2). We transformed both to total time in hours.² For PvZ, the Decipher survey platform restricted the maximum time players could report to 40 h 59 min. Such large values were rare and affected only a handful of participants. The AC:NH time scale was unrestricted.

2.2.4. Telemetry

Telemetry data for both games were available in different levels of granularity. For both PvZ and AC:NH, the games companies provided game sessions per player over the two-week window before finishing the survey. Each game session had a start time and an end time. For example, when a player turned on their console, launched PvZ and entered the game hub world (where they can select what type of level or mode to play), opening the hub world counted as the start time. However, determining an end time can be difficult. Players could immediately play another round, return to the hub world, take a break while leaving the game on etc. Therefore, there were instances where game sessions for a given player overlapped (e.g. two game sessions had the same start time, but different end times). In such cases, we condensed multiple overlapping game sessions into one game session, taking the game start time that the sessions shared and the last end time. As a result, players had multiple unique game sessions without overlap. Afterwards, we aggregated the durations of all game sessions per player to obtain the total objective time they spent playing the game in the two-week window before they filled out the survey. In the case of AC:NH, Nintendo of America provided telemetry including game session start and end times, as well as session durations. Start and end times were not always accurate for the same reasons as with PvZ, but we used them in order to only use the session information from the two weeks preceding the survey. However, the session durations were verified by the Nintendo of America Team. Therefore, we aggregated durations in the two weeks preceding the survey per participant for AC:NH (figure 2). Readers can find more details on data processing on https://digital-wellbeing.github.io/gametime/.

PvZ telemetry featured fine-grained records of game events. The data contained information on the game mode of a game session (e.g. online or split screen), the levels a player played in during a game session, and the game type (i.e. single player versus multiplayer). Furthermore, there were indicators of how a player fared during a game session, such as total kill counts, death counts, scores, total damage dealt, shots fired and hit, and critical hit counts. PvZ telemetry also contained information on a player’s progression, such as when a player gained a level, how much XP they gained, and when they gained a prestige level. Last, there were measures of social interactions with other players, such as what in-game gestures a player used and when they became in-game friends with other players. We did not analyse these data or report on them here. Readers can find them on the OSF page of this article.

2.3. Statistical analysis

Before analysis, we first excluded data from players who gave the same response to all SPANE and motivations items (PvZ: 1 [0.2%]; AC:NH: 8 [0.1%]). This so-called straight lining is an indicator of poor data quality [61]. We then identified as outliers all observations that were more than six standard deviations away from the variable’s mean. We aimed to exclude as few data points as possible, which is why we did not follow the common rule of thumb of three standard deviations and only identify truly extreme, implausible values. We replaced those extreme values with missing values to not bias resulting analyses. For PvZ, 1 (0.2%) objective play time values were excluded. For AC:NH, 12 (0.2%) actual and 40 (0.7%) estimated play time values were excluded. In the models reported below, play time refers to units of 10 h, and the well-being and motivation variables were standardized. Note that the sample sizes per analysis will differ because of missing values either due to exclusions, not having telemetry data, or not having played socially (i.e. no responses on relatedness). We conducted all statistical analyses with R (v. 4.0.3, [62]).

3. Results

3.1. Play time and affective well-being

We first focused on the relationships among objective and subjective play time, and well-being. The actual and estimated play times are shown in figure 2. On average, participants overestimated their play time (PvZ: M = 1.6, s.d. = 11.8; AC:NH: M = 0.5, s.d. = 15.8).

Then, to better understand how actual and estimated play times were related, we regressed the subjective estimates of play time on objective play time. This correlation was positive (PvZ: β = 0.34, 95%CI [0.27, 0.41], R² = 0.15, N = 469; AC:NH: β = 0.49, 95%CI [0.45, 0.54] R² = 0.16, N = 2714; figure 3a). Readers can find the full regression tables on https://digital-wellbeing.github.io/gametime/analyses.html#analyses.