Contents lists available at ScienceDirect
International Journal of Information Management
journal homepage: www.elsevier.com/locate/ijinfomgt
The rise of motivational information systems: A review of gamification
research
Jonna Koivistoa,⁎, Juho Hamaria,b
aGamification Group, Faculty of Information Technology and Communications, Tampere University, Finland
bGamification Group, Faculty of Humanities, University of Turku, Finland
A R T I C L E I N F O
Keywords:
Gamification
Games
Motivational information system
Affordance
Literature review
A B S T R A C T
Today, our reality and lives are increasingly game-like, not only because games have become a pervasive part of
our lives, but also because activities, systems and services are increasingly gamified. Gamification refers to de-
signing information systems to afford similar experiences and motivations as games do, and consequently, at-
tempting to affect user behavior. In recent years, popularity of gamification has skyrocketed and manifested in
growing numbers of gamified applications, as well as a rapidly increasing amount of research. However, this vein
of research has mainly advanced without an agenda, theoretical guidance or a clear picture of the field.
To make the picture more coherent, we provide a comprehensive review of the gamification research
(N=819 studies) and analyze the research models and results in empirical studies on gamification. While the
results in general lean towards positive findings about the effectiveness of gamification, the amount of mixed
results is remarkable. Furthermore, education, health and crowdsourcing as well as points, badges and leader-
boards persist as the most common contexts and ways of implementing gamification. Concurrently, gamification
research still lacks coherence in research models, and a consistency in the variables and theoretical foundations.
As a final contribution of the review, we provide a comprehensive discussion, consisting of 15 future research
trajectories, on future agenda for the growing vein of literature on gamification and gameful systems within the
information system science field.
1. Introduction
During recent decades, we have witnessed glimpses of a fascinating
emerging development where utilitarian and hedonic systems are in a
state of spiraling convergence. Today, the spiral has made a full re-
volution, and we now see hedonic or entertainment-oriented technol-
ogies being re-appropriated for productive use. This development has
been titled “gamification” and the phenomenon has quickly cemented
itself as being one of the major developments in the information sys-
tems (IS) field and other domains. Hedonic information systems in-
itially came about through the re-appropriation of instrumental in-
formation technology. Most notably, the first video games emerged
from a playful re-appropriation of oscilloscopes – a seemingly utili-
tarian system (“Tennis for Two” developed by Higinbotham in 1958 –
see e.g. Tavinor, 2009). Since then, we have witnessed a wide diffusion
of game consoles (e.g. Pong in 1972, Atari 2600 in 1977, Nintendo in
1983, Xbox in 2002 etc.) and other video game applications. For-
warding to today, hedonic systems and software are everywhere, and
are developed for the sole purpose of promoting user enjoyment.
Furthermore, digital games have penetrated our everyday lives at an
increasing pace and have now become a mainstream form of en-
tertainment, enjoyed by people from all demographic groups (see e.g.
Williams, Yee, & Caplan, 2008; Williams, Consalvo, Caplan, & Yee,
2009). However, especially during the last ten years, we have come a
full circle, and hedonic systems (and especially game designs) are
currently merging back into utilitarian systems and even perhaps new
strains of utilitarian systems are emerging from hedonic systems.
Games are especially known for their ability to engage and excite,
and when playing games, people commonly experience e.g. mastery,
competence, enjoyment, immersion, or flow, all of which are char-
acteristic of intrinsically motivated human behavior (e.g. Huotari &
Hamari, 2017; Ryan, Rigby, & Przybylski, 2006; Deci & Ryan, 2000;
Ryan & Deci, 2000; Agarwal & Karahanna, 2000; Venkatesh, 1999;
Webster & Martocchio, 1992; Csíkszentmihályi, 1975, 1990). An es-
sential aspect of playing games is the self-purposeful nature of the ac-
tivity, as well as the engagement and enjoyment of the activity. It is this
nature of playing games that gamification technology attempts to
capture, harness and implement into contexts that commonly have a
https://doi.org/10.1016/j.ijinfomgt.2018.10.013
Received 27 June 2017; Received in revised form 13 July 2018; Accepted 15 October 2018
⁎ Corresponding author at: Gamification Group, Faculty of Information Technology and Communications, FI-33014 Tampere University, Finland.
E-mail address: jonna.koivisto@tut.fi (J. Koivisto).
International Journal of Information Management 45 (2019) 191–210
0268-4012/ © 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license 
(http://creativecommons.org/licenses/BY-NC-ND/4.0/).
T
more instrumental purpose (Hamari & Koivisto, 2015b; Huotari &
Hamari, 2017; Liu, Xun, & Santhanam, 2013; Santhanam, Liu, & Shen,
2016; Vesa, Hamari, Harviainen, & Warmelink, 2017; Liu, Santhanam,
& Webster, 2017). When starting a game, a player accepts the con-
tingency of the end result, however, the process is often enjoyable re-
gardless of the outcome (see e.g. Malaby, 2007). Incorporating the
engagement and enjoyment of the gameful process into activities out-
side games is at the core of what commonly is titled gamification; a
design approach of employing game elements into different types of
systems and services, with the goal of affording gameful experiences
(Huotari & Hamari, 2017).
Since its conceptual inception around 2010, gamification has in-
creasingly drawn the attention of academics and practitioners (see
Hamari, Koivisto, & Sarsa, 2014). In addition to gaining popular pro-
ponents, the approach has gained traction from positive prospects
published in business analyses by Gartner (2011) and IEEE (2014)
which predict that most companies and organizations will implement
gamification in the near future. Consequently, operators in various
fields have been attracted by the potential of gamification for inducing
motivation and engagement for a diverse range of activities. This has
led to gamification being implemented in domains such as enterprise
resource planning (Alcivar & Abad, 2016; Herzig, Strahringer, &
Ameling, 2012), intra-organizational communication and activity
(Farzan et al., 2008b, Farzan et al., 2008a; Thom, Millen, & DiMicco,
2012), science (Sørensen et al., 2016), government services and public
engagement (Bista, Nepal, Paris, & Colineau, 2014; Tolmie,
Chamberlain, & Benford, 2014; see also Hassan & Hamari, 2019 for a
review), work (see Warmelink, Koivisto, Mayer, Vesa, & Hamari, 2018
for a review) and crowdsourcing (Eickhoff, Harris, de Vries, &
Srinivasan, 2012; Lee, Ceyhan, Jordan-Cooley, & Sung, 2013; Ipeirotis
& Gabrilovich, 2014; see also Morschheuser, Hamari, Koivisto, &
Maedche, 2017 for a review), commerce (Hamari, 2015; Hamari,
2013), exercise (Hamari & Koivisto, 2015a; Koivisto & Hamari, 2014),
health (Jones, Madden, & Wengreen, 2014; see also Alahäivälä & Oinas-
Kukkonen, 2016 for a review), education (e.g. Bonde et al., 2014;
Christy & Fox, 2014; de-Marcos, Domínguez, Saenz-de-Navarrete, &
Pagés, 2014; Denny, 2013; Domínguez et al., 2013; Farzan &
Brusilovsky, 2011; Filsecker & Hickey, 2014; Hakulinen, Auvinen, &
Korhonen, 2013; Simões, Díaz Redondo, & Fernández Vilas, 2013; see
also Majuri, Koivisto, & Hamari, 2018 for a review), environmental
behavior (Lee et al., 2013; Lounis, Pramatari, & Theotokis, 2014), as
well as marketing and advertising (Cechanowicz, Gutwin, Brownell, &
Goodfellow, 2013; Terlutter & Capella, 2013; Xi & Hamari, 2019), to
name a few.
The literature on gamification is rapidly increasing and spreading in
many directions, but this is similar to any development that has great
potential and which is surrounded by a crowd of hyped enthusiasts. In
order to control and take advantage of this development, concerted
efforts are needed to harness the literature and existing knowledge to
productive use, and to provide the field with an agenda for further
research. Gamification is still in its infancy and rapidly developing, but
what is actually known of the phenomenon tends to stem from frag-
mented pieces of knowledge, and from a variety of perspectives. While
some attempts have been made to synthesize the literature on gamifi-
cation, previous reviews have been very focused in their scope. In order
to provide both academics and practitioners with a more widespread
view on the gamification phenomenon, a larger scale review of the
phenomenon should help to map its development and progress, as well
as aid in steering future literature and agendas. We firmly believe that
gamification is especially an IS/IT phenomenon, since it has at its core
the use of leisure information systems (more specifically (video) games)
and their design in a variety of utilitarian information system contexts.
However, if we consider the host of literature on gamification that has
been produced thus far, it appears to be relatively under-represented in
IS literature, regardless of it clearly being an IS phenomenon. This
suggests that other fields (especially those of education and human-
computer interaction) have perhaps shown more innovation and
openness in their approach to this prominent technological develop-
ment. Therefore, it is also important to more broadly initiate a discus-
sion about gamification in IS literature.
In this study, we aim to, firstly, comprehensively review and syn-
thesize the extant literature on the concept of gamification; and sec-
ondly, to theorize and delineate a further research agenda for the re-
search of gamification and motivational information systems within the
information systems research field. The review draws together the ex-
istent knowledge on the topic and presents it in a structured manner.
The review process mainly follows the guidelines described by Webster
and Watson (2002) and Paré, Trudel, Jaana, and Kitsiou, (2015). Over
800 papers have been categorized, and 273 empirical studies are ana-
lyzed in detail to outline the domains in which gamification is being
implemented, how it is being implemented, how it has been studied, as
well as identifying the kinds of results that have been produced thus far.
The findings of the review indicate where research knowledge is al-
ready abundant, where further research is needed, and what steps
should be taken in future research to develop knowledge on the topic.
2. Background
Information systems discipline has traditionally been characterized
as the pursuit of knowledge pertaining to productivity and efficiency
(see e.g. Hirschheim & Klein, 2012), and improvement of these. A
substantial body of knowledge has sprung from this rational, utility-
seeking premise of aiding in the development and construction of ef-
ficiently managed and operated organizations and information systems
within them. However, this utility-driven lens of information systems
was not geared towards capturing the use of varied non-utilitarian in-
formation systems that started to heavily appear when information
technology had advanced enough in its graphics and calculation power.
Information systems seeking to fulfill entertainment-oriented needs
challenged the utilitarian premise previously dominating the research
and understanding in the IS field (e.g. van der Heijden, 2004). There-
fore, the scope of information systems science was expanded by in-
troducing the study of hedonic information systems that deviates from
the utility/rational-core of the IS discipline. The first wave of literature
started to widen the perspective of IS research into intrinsic and he-
donic motivations in the early 1990s by studying the concepts of
playfulness and enjoyment in relation to technology acceptance and use
(see e.g. Webster & Martocchio, 1992; Davis, Bagozzi, & Warshaw,
1992), and later in 2004 by e.g. van der Heijden (2004) via the de-
velopment of models that addressed the acceptance and use of hedonic
information systems. During the same period, the marketing research
field and literature also witnessed a surge of research on hedonic as-
pects of consumption (see Hirschman & Holbrook, 1982 for an early
account). Since then, however, a disconnect between the rational and
hedonic veins of IS literature has existed until very recent literature on
dual and multipurpose systems (as can be seen e.g. in Wu & Lu, 2013).
When considering these various types of information systems, it
seems that gamification has a rather interesting and peculiar role.
Traditionally, the information systems field has distinguished between
two system types that are designed to address different needs, either
utilitarian or hedonic (see e.g. van der Heijden, 2004). Systems defined
as utilitarian information systems are commonly designed to serve
purposes related to productivity. They seek to increase the efficiency of
a given task, and therefore, at their core they serve mainly instrumental
purposes. From a motivational perspective, the use of utilitarian sys-
tems can be considered as being extrinsically motivated (see e.g. Deci &
Ryan, 1985; van der Heijden, 2004); i.e. the system aids the user in
reaching a goal that is separate from the system use itself (Davis et al.,
1992). The usefulness and benefits of the system thus rise from reaching
the external goal more efficiently. Conversely, the use of hedonic in-
formation systems is mostly entertainment-driven (van der Heijden,
2004). These systems aim at creating experiences of enjoyment and are
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
192
used for the purposes of recreation and entertainment, and for the sake
of using the system itself. Therefore, the use of hedonic systems is
considered to be autotelic and intrinsically motivated (Deci & Ryan,
1985; Ryan & Deci, 2000), as opposed to systems with utilitarian use
objectives. Examples of hedonic systems are entertainment-oriented
websites and services, video games, blogs, and social networking sites.
In recent years, information systems have increasingly been de-
signed to meet more varied user motivations and orientations (Gerow,
Ayyagari, Thatcher, & Roth, 2013; Sun & Zhang, 2006). The research
field has started to acknowledge that many systems serve both utili-
tarian as well as hedonic needs (Hamari & Keronen, 2017), and in-
creasingly, systems are fundamentally being designed to cater for these
needs, i.e. as mixed systems (Gerow et al., 2013). One such type of
mixed systems are those with the objective of motivating users toward
different individually and collectively beneficial behaviors (Hamari &
Koivisto, 2015b; Hamari et al., 2014). These systems are an intriguing
combination of both utilitarian (Davis, 1989) and hedonic aspects (van
der Heijden, 2004): the goals of the systems’ use are related to pro-
ductivity, however, the means and the design by which the systems
promote productivity are hedonic in nature. Whereas traditional utili-
tarian systems aim for productivity through efficiency and traditional
hedonic systems aim for creating fun experiences, these Motivational
Information Systems can be characterized to aim for “productivity
through fun”. Therefore, Motivational Information Systems differ from
the utilitarian and hedonic information systems in one important,
crucial way: the acceptance is mainly driven by usefulness as in utili-
tarian systems, but the usefulness is determined by the enjoyment of the
use. One of the most prominent solutions for addressing motivational
challenges has been to draw from one of the pinnacle forms of hedonic
information systems, i.e. digital games. This approach is commonly
referred to as gamification.
Gamification refers to a design approach of enhancing services and
systems with affordances for experiences similar to those created by
games (Deterding, Dixon, Khaled, & Nacke, 2011; Hamari, Huotari, &
Tolvanen, 2015; Huotari & Hamari, 2012; Liu et al., 2017; Santhanam
et al., 2016; Vesa et al., 2017). These “gameful” affordances aim at
supporting and motivating the user toward the behavior that the ga-
mified system is targeting, such as healthy behaviors and exercise,
participation in learning activities etc. The experiences created by
games refer e.g. to senses of enjoyment, flow, autonomy, mastery, ac-
complishment etc., that are considered to be induced by games and
game play (e.g. Ryan et al., 2006). In the context of games, these ex-
periences are often considered to be what make games intrinsically
motivating, so that the user wishes to engage with the system simply for
the sake of using it. Harnessing similar experiences by implementing
gameful affordances in the contexts of the utilitarian functions de-
scribed above aims at transferring similar motivational effects into the
new environment. What makes motivational information systems such
as gamification interesting is the fact that the systems at their core
motivate and support the user toward a given activity or behavior. This
conveys that the system should increase the efficiency and productivity
regarding the target behavior. Thus, their usefulness is determined on
the basis of whether they manage to do so. Furthermore, in many of the
commonly gamified contexts, such as learning or healthy behavior, the
activities require long-term commitment and persistence for the results
to actualize. On the other hand, the systems features hedonic design,
and therefore aim at making the process of using the system enjoyable.
When the system use is enjoyable, the chances of engaging with it in
long-term may be increased (see e.g. van der Heijden, 2004; Atkinson &
Kydd, 1997; Moon & Kim, 2001; Venkatesh, 1999; Mäntymäki,
Merikivi, Verhagen, Feldberg, & Rajala, 2014; Mäntymäki & Riemer,
2014; Hamari, 2015; Mäntymäki & Salo, 2013; Mäntymäki & Salo,
2015).
The potential of gamification lies in the restructuring of tasks and
activities with game elements and gameful affordances. This may be by
dividing a larger whole into sub-tasks with clear goals and providing
direct feedback for accomplishments, reframing an activity by estab-
lishing a meaningful narrative, or by gathering a social community to
provide support. A commonly used theoretical frame for understanding
the motivational potential of games is that of self-determination theory
(SDT) and its sub-theories (Deci & Ryan, 2000; Deci & Ryan, 1985; Ryan
et al., 2006) that consider human motivation to be either intrinsically or
extrinsically motivated, depending on whether the activity is performed
for the sake of the activity itself or for reasons external to the activity.
For a behavior to be intrinsically motivated, it is likely that it results
from motivational needs for competence, autonomy and relatedness
(Deci & Ryan, 2000; Ryan & Deci, 2000). Competence refers to feelings
of mastering the challenge at hand. Autonomy refers to the freedom of
choosing what challenges to undertake, and relatedness refers to ex-
periences of recognition and acceptance (Deci & Ryan, 2000). All of
these motivational needs are well-documented as being commonly sa-
tisfied by playing games (Ryan et al., 2006), which is generally con-
sidered to be a highly intrinsically motivated behavior (Rigby & Ryan,
2011). Playing games is usually a voluntary behavior, conducted at
one’s own instigation, and thus a behavior that promotes autonomy
(Ryan et al., 2006). Furthermore, encountering and overcoming chal-
lenges that are often adjusted to the optimal level for the player (see
e.g. Csíkszentmihályi, 1975, 1990) is essential to gameplay, and is often
considered to be a core component of games (Deterding, 2015; Ryan
et al., 2006). Thus, playing games commonly provides experiences of
competence as the player tackles the challenges of the game. Moreover,
the player’s relatedness is often catered for by social environments
created either within the game or around it (Huang, Cheng, Huang, &
Teng, 2018; Ryan et al., 2006).
On an overarching level, gamification can be seen to comprise of
three main elements (Hamari et al., 2014): the affordances im-
plemented to a system or service lead to psychological outcomes, and
these gameful experiences further lead to behavioral outcomes, i.e. the
activities and behaviors that the gamification aims to support and
motivate (see Fig. 1). Furthermore, all these elements and the supported
activity are situated within a certain context (Hamari et al., 2014;
Huotari & Hamari, 2017; Deterding, 2015). The affordances refer to the
various elements and mechanics that structure games and aid in indu-
cing gameful experiences within the systems. The psychological out-
comes refer to psychological experiences such as competence, au-
tonomy and relatedness, or for example enjoyment and engagement,
which games and gamification are commonly considered to promote.
The behavioral outcomes of gamification refer to behaviors and activ-
ities that are supported through use of the gamification system, such as
continued or increased physical activity in the context of exercise ga-
mification, or better learning results in the context of education gami-
fication.
The topic of motivational information systems and especially ga-
mification has gained significant popularity in recent years, which has
Fig. 1. Overall conceptualization of gamifica-
tion following Hamari et al. (2014); Huotari
and Hamari (2017) and Deterding (2015).
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
193
manifested in a great amount of literature being produced on the topic
from both academic and non-academic sources. The first attempts to
map the literature under the flag of gamification were conducted in
early 2014, and formed a review examining the then-current body of 24
peer-reviewed internationally published research papers (Hamari et al.,
2014). Since then, the popularity of the topic has increased significantly
and the concept has been trending among practitioners and academics.
As confirmation of the growth of the topic, academic databases provide
proof of the expanding volume of literature found with the keyword
‘gamification’. In their review, Hamari et al. (2014) reported that a
literature search in the Scopus database with the search terms ‘gami-
fication’, ‘gamif*’, ‘gameful’ and ‘motivational affordance’ resulted in
330 hits. In June 2015, a similar literature search using only the search
term ‘gamif*’ provided 807 hits. At the time of writing, in June 2016,
the latter search provides 1767 hits. In order to maintain currency with
the research area, an overview of how the field of gamification is de-
veloping is desperately needed. While some literature mapping has
been conducted since the Hamari et al. (2014) review (see e.g. Seaborn
& Fels, 2015), a general overview of how the phenomenon has been
researched and developed is lacking.
3. The Review
3.1. Review procedure
The literature searches were conducted in the Scopus database and
the Association for Information Systems (AIS) Electronic Library
(AISeL), which were chosen for the reason that they index all of the
other potentially relevant databases, for example ACM, IEEE, Springer,
and the DBLP Computer Science Bibliography. Conducting the searches
in as few comprehensive databases as possible instead of several ones is
seen as preferable for purposes of rigor and clarity (see Paré et al.,
2015).
The search for literature in the Scopus database was conducted
using the search query: TITLE-ABS-KEY (gamif*). The search was lim-
ited to include conference papers, articles, articles in press, reviews and
book chapters, in order to exclude non-academic publications. In the
AISeL database, the search query ‘abstract:gamif* OR subject:gamif*
OR title:gamif*’ was used. The search term gamif*was chosen as it takes
into account all possible forms derived from the root, i.e. the noun
gamification, and the verb gamify in all its forms.
The search fields were defined in Scopus as title, abstract, and
keywords, and in AISeL as title, abstract, and subject. These search
parameters were used as it is considered that inclusion of the search
term in the metadata would indicate that the term is actually relevant
for the given paper. With these search queries, we aimed to reach all the
academic literature (within the used databases) that is published under
the flag of gamification. Inevitably, this strategy also led to hits where
the term gamification was not of actual relevance to the paper, but
where, for example, the abstract contained a mention of gamification as
a future prospect. These hits were included in the review, but high-
lighted as not directly relevant and therefore not included in the full
analyses.
The literature search was conducted in June 2015 and resulted in
807 hits from the Scopus database and 35 from the AISeL database. All
of the hits were collected into reference management software, where
Fig. 2. Flowchart describing the literature review process.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
194
the references were managed, organized and categorized. The review
process was conducted by the first author of this paper. Any cases and
categorizations that were unclear were discussed amongst the research
team.
On further examination, 2 hits were removed as duplicate entries in
the Scopus database and 18 hits were identified as overlapping between
the Scopus and AISeL search results. Of the remaining papers, 3 studies
were identified as false hits where the paper concerned used the letter
combination ‘gamif’ to refer to something other than gamification. The
final body of literature amounted to 819 papers.
The full versions of the papers were retrieved. Where papers were
initially not available for us, we contacted the authors either by email
or through academic social network tools such as ResearchGate and
requested the full version. Around 30 papers were obtained in this way.
The final number of unavailable papers was 16. Additionally, 16 of the
papers were identified as not being written in English, and were
therefore excluded from further analyses.
The remaining 787 papers were categorized based on the type of
publication, whether the papers were full research papers, and whether
they contained empirical data. Fig. 2 outlines the literature review
process.
286 full papers were identified as containing empirical data. Of
these, 13 were further identified as explicitly stating that the studied
system contained no gamification, that gamification was part of future
plans to develop the featured solution, or the paper contained a men-
tion of the term solely as a keyword. These papers were removed from
the further analyses which focused on empirical studies. The final
amount of analyzed empirical papers was 273.
In the literature analysis, the studies were coded. As mentioned
above, the amount of information coded from the retrieved papers
depended on whether they were categorized as empirical or non-em-
pirical. Full analyses were conducted only on empirical full papers.
Table 1 outlines the details that were retrieved during the coding
process from both empirical and non-empirical papers.
Following the guidelines of Webster and Watson (2002), all of the
full papers were analyzed, firstly author-centrically, and then concept-
centrically. The units of analysis were defined prior to the analytical
process. Author-centric coding was conducted, where the pre-defined
units of analysis were checked and coded for each paper as it was read.
Through this procedure, a matrix of the coded literature was produced.
Continuing the analysis process, a concept-centric approach was then
taken where coded literature was then organized based on further units
of analysis. As suggested by Webster and Watson (2002), the coded
concepts were comprised into frequency tables, which form the core of
this review.
3.2. Bibliometric descriptors
Since 2011, there has been rapidly increasing general interest in
gamification, and this has also been reflected by the academic interest
shown in the topic. As the literature search hits by year indicate
(Table 2), the amount of academic literature on gamification has con-
sistently risen over the past few years.
From the 819 total hits, 260 papers were identified as not being full
papers (Table 3). In this category we included all the papers that, e.g.
reported research-in-progress or other types of incomplete research,
were not peer-reviewed, or were in other formats than research reports.
Many of these papers were in fact short conference publications and
workshop papers.
The full research papers were categorized based on the form of
Table 1
Details retrieved from the full papers in the reviewed body of literature.
Coded information Non-empirical, full
papers
Empirical full
papers
Empirical full papers -
Quantitative experimental
studies
Bibliometric data: Author(s), publication year, publication venue X X X
Type of study: Empirical, (preliminary) description of a study/system, conceptual papers; frameworks;
think pieces, review papers, simulations; modelings, heuristics, definition papers
X X X
Domain: E.g. education, health and exercise, business and management etc. X X X
Data gathering method: E.g. Implementation/prototype, interviews, observation etc. X X
Data type used in analyses: E.g. Log data, interview data, observation data etc. X X
Analysis method: Qualitative, quantitative (descriptive, inferential) X X
Affordances: E.g. points, narratives, social features etc. X X
Outcomes: Psychological outcomes, behavioral outcomes X X
Sample sizes X
Results: Positive, mixed with mostly positive, null or equal positive and negative, mixed with mainly
negative, negative; categorized by domain and affordances
X
Table 2
All Scopus and AISeL hits for gamif* by year at the
time of data gathering (6/2015). False hits, dupli-
cates and overlapping hits removed.
Year Number of hits
2011 26
2012 92
2013 263
2014 368
06/2015 70*
Total 819
*It is noteworthy that due to the natural delay in
addition of entries to the repositories, a proportio-
nately smaller number of studies had yet appeared
during 2015 at the time of the data gathering.
Table 3
Types of not full papers (N=260).
Types of not full papers Frequency
Short papers, research notes 69
Workshop papers 60
Posters 28
Magazine articles, journal paper articles, newsletters, commentaries 19
Extended abstracts 17
Research-in-progress 19
Demonstrations, tutorials 11
Technical reports, industry track papers 8
Panel or track proposals/descriptions 8
PhD papers 3
Editorials 2
Business reports 1
Books 1
Other not full 14
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
195
research reported, e.g. empirical research, descriptive non-empirical
research, conceptual work, or literature reviews (see Table 4). After the
removal of duplicate studies, 514 full research papers were examined to
determine the type of the research conducted and reported. Of the 514
papers, 286 were identified as empirical. However, 13 of these ex-
plicitly stated that either gamification was not examined in the paper,
that gamification was considered as a future possibility, or the paper
simply contained gamification as a keyword without any further men-
tions. These studies were therefore not included in further analyses.
Thus, the final analysis included 273 studies reporting empirical data
that was analyzed in the paper. These publications varied greatly in
terms of the extent of data gathering and analysis, ranging from large-
scale or multi-year experiments to preliminary trials with only a few
participants. However, all of the studies were categorized as empirical
research papers for the purposes of this review, and for the purpose of
gaining the widest possible overview of the research field. Papers de-
scribing a future study or a system under development were prominent
(105 studies), but as these papers were identified not to contain any
actual empirical data, they were excluded from further content analysis.
Conceptual papers, frameworks and think pieces (84 studies) mostly
considered ideas and perspectives for developing the understanding of
gamification or the study of the phenomenon. The remainder of the
body of literature consisted of reviews, simulations on gamification,
heuristic papers, as well as papers concentrating on defining the con-
cept of gamification.
Of the full research papers, a clear majority (352 papers) were
published in conference proceedings, and only 155 were published in a
journal (see Table 5). What potentially follows from the fact that con-
ference publications are the main venue for publication of gamification
research is related to the comprehensiveness of the research work.
Conference publications are usually limited in length and often not as
comprehensive in their discussion of various aspects of the conducted
research when compared to journal publications. Thus, the great
number of conference proceedings has a potential effect on the theo-
retical depth of the research on gamification. In their review of gami-
fication literature, Seaborn and Fels (2015) found that the majority of
the reviewed papers did not address the theoretical foundations which
guided the research, and the lack of theoretical discussions has been
acknowledged as a problem in the field. Seaborn and Fels (2015) also
observe a disconnection between theory and applied research in their
reviewed literature. While the theoretical stances and foundations of
gamification were not explicitly mapped in this review, similar ob-
servations can be made from the body of current literature; a large
proportion of the research papers addressed the theoretical foundations
of the work only in passing, via a definition of gamification or pre-
senting related prior research.
Nevertheless, the bibliometric descriptors of the current body of
literature demonstrate the increasing amount of research that has been
published on the topic of gamification. While there is no possibility of
knowing whether this trend will continue, the bibliometric results may
provide some perspectives which could be applied to future develop-
ments. As conference papers often present early research or developing
ideas, the abundance of conference publications in the body of litera-
ture can reflect how interest in the concept of gamification will likely
develop in the future. Additionally, as a large part of the literature
reported work-in-progress or preliminary results, this suggests that the
amount of literature will also continue to increase in the future.
3.3. Domains
Table 6 outlines the domains of empirical studies, non-empirical
descriptive, and conceptual papers. Altogether, 462 research papers
were analyzed for their domain. Reviews, definition papers, heuristic
and simulations/modelings were excluded, since their contexts were
often not clearly definable due to the type of research presented. While
domains may evidently overlap (e.g. an empirical study about gamifi-
cation in education may actually be more concentrated on the workload
it creates for the teachers, instead of the effects it has on students), for
the purposes of the review, all of the studies were assigned to a single
domain which describes the general context where the gamification has
Table 4
Types of full papers.
Types of papers (duplicates removed) Frequency %
Empirical (also unclear methods/data) 273 53.1
Explicitly not about gamification 13 2.5
(Preliminary) description of a study/system, no empiricism 105 20.4
Conceptual papers; frameworks; think pieces 84 16.3
Review papers 19 3.7
Simulations; modelings 11 2.1
Heuristics 5 1.0
Definition papers 4 0.8
Total 514 100
Table 5
Publication venues of full papers.
Publication venue Frequency %
Conference 352 68.5
Journal 155 30.2
Book chapters 7 1.4
Total 514 100
Table 6
Domains in empirical, non-empirical descriptive and conceptual papers
(N=462).
Empirical
papers
Non-
empirical
papers
Domain Frequency % Frequency %
Education/Learning 129 46.7 67 35.4
Health/Exercise 40 14.5 15 7.9
Crowdsourcing (includes
information gathering,
knowledge sharing and citizen
science)
25 9.1 7 3.7
Social behavior/networking/
sharing
14 5.1 2 1.1
Software development/design * 11 4.0 25 13.2
Business/Management 10 3.6 19 10.1
Ecological/environmental behavior 9 3.3 9 4.8
eCommerce/eServices 8 2.9 1 0.5
Software engineering ** 7 2.5 5 2.6
Marketing/Consumer behavior 4 1.4 5 2.6
Citizen/public engagement/activity 3 1.1 2 1.1
Entertainment (includes gaming,
watching TV, media capturing)
3 1.1 1 0.5
Innovation 3 1.1 2 1.1
Transportation/Mobility 3 1.1 4 2.1
Culture/Tourism 2 0.7 4 2.1
Architecture 1 0.4 0 0.0
Communication 1 0.4 0 0.0
Emergency planning 1 0.4 0 0.0
Politics 1 0.4 3 1.6
Welfare/city/human/public
services
1 0.4 4 2.1
Work 1 0.4 2 1.1
Theory *** 0 0.0 12 6.3
Total 276**** 189
* Studies related to design and development of gamification services.
** Software engineering as a field of work/industry.
*** Studies where gamification or an aspect related to it is discussed on an
abstract/theoretical level.
**** One study contains 4 different cases in different domains. All of the 4
domains are included in the categorizations.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
196
been implemented or considered.
The analysis of domains in the body of literature shows that a clear
majority of the empirical research on gamification is conducted in the
domain of education and learning. Second largest category of empirical
studies is health and exercise, followed by research papers relating to
crowdsourcing. These three categories comprise over 70% of the em-
pirical research in the current body of literature. The fourth largest
category consists of various social behavior and networking domains,
followed by empirical studies which are related to the design and de-
velopment of gamification services, as well as papers in the business
and management domain. The remainder of the domain categories
comprise less than ten empirical research papers.
Interestingly, in the non-empirical research papers, education and
learning is again the most common domain, but the second largest
domain is formed by studies related to the design and development of
gamification services. The third most common domain of non-empirical
research is business and management. Thus it can be seen that in non-
empirical research which potentially contains studies at a preliminary
phase of the research process, there is the promise of certain domains
gaining more empirical research in the future.
In more detail, the domain of education and learning constitutes
nearly half of the published empirical research on gamification, with
129 published empirical papers. In the non-empirical research, more
than one third of the publications also stem from the education and
learning domain. The education domain has previously been noted as
the most popular gamification context for empirical research (Hamari
et al., 2014; Seaborn & Fels, 2015). This review stands as further con-
firmation of these findings indicating that in the expanding field of
online learning and training (Panigrahi, Srivastava, & Sharma, 2018)
gamification has its share. The attractiveness of gamification in the
education domain is rather intuitive, as games in general commonly
promote learning and the developing of skills in an inherent manner,
and often provide a structured environment where these skills can be
practiced (see e.g. Landers & Armstrong, 2015). However, the educa-
tional domain has been somewhat riddled with conceptual unclarity as
the terms of gamification, game-based learning and serious games are
all commonly used to refer to the use of games in education contexts
(Landers, 2015). In this review, no distinctions between the concepts
were made, and any paper labeled by the authors as gamification has
been considered, as specified above.
In the empirical research, health and exercise forms the second
largest category for studies. Especially with activities such as healthy
eating and physical exercise, extra support and a motivational push
may be needed in order to maintain routines (Hamari & Koivisto,
2015a). The gamification approach has been shown to have positive
results in this domain in several of the featured studies (Allam, Kostova,
Nakamoto, & Schulz, 2015; Cafazzo, Casselman, Hamming, Katzman, &
Palmert, 2012; Chen & Pu, 2014; Chen, Zhang, & Pu, 2014; Hamari &
Koivisto, 2015a; Jones et al., 2014; Koivisto & Hamari, 2014; Riva,
Camerini, Allam, & Schulz, 2014; Watson, Mandryk, & Stanley, 2013).
Furthermore, an interesting development is the large number of
research papers on gamification in crowdsourcing. Research examining
the combinations of these two fields has previously been mapped by
Morschheuser, Hamari, and Koivisto, 2016; Morschheuser et al., 2017,
who suggest that the addition of game elements for increasing the
motivations of volunteer crowdsourcees is an efficient approach based
on empirical research findings.
Regarding the largest domains of empirical gamification research, it
can be concluded that the findings of this review support those of
Seaborn and Fels (2015), who found health, crowdsourcing and social
networking to be the biggest domains for empirical research of gami-
fication, in addition to education.
As a further conclusion, based on the domains of the empirical re-
search, it is evident that gamification seems to be implemented espe-
cially in domains where long-term commitment and perseverance are
needed for gaining results. These may include learning or the
development of healthy or beneficial habits. This idea is supported by
the fact that education, health and e.g. domains relating to ecological
behavior are among the most popular categories.
Among the non-empirical research, the prevalence of the domains
differs from the pattern seen in empirical works, with the exception of
education and learning. Non-empirical work has been conducted more
in domains such as general software development and the design of
gamification services. This is rather expected, as within a field that is
still very much in development, plenty of work is required on the
conceptualization and design of solutions, before it is feasible to pro-
duce empirically testable prototypes or systems. This is evident also
from the noticeable number of theoretical papers, present among the
non-empirical works.
3.4. Methods and data
As shown in Table 7, the empirical research papers most frequently
employed quantitative research methods. Mixed methods studies were
also numerous, with qualitative research papers featuring in the min-
ority. The category labeled as ‘mixed’ refers to empirical papers which
combined any forms of qualitative and quantitative approach.
The quantitative research papers were further categorized based on
the approach taken in their analyses, as either descriptive or inferential.
The majority of the empirical quantitative research on gamification is
descriptive, meaning that the analyses of the data are often reported as
percentages and means drawn from the numerical data. In the in-
ferential studies, some relationship between variables has been ex-
amined, and the results are reported. For studies using a mixed meth-
odology, the type of quantitative analyses was not recorded. However,
if studies combining qualitative and quantitative research approaches
were considered, then the amounts of both descriptive and inferential
quantitative research would be higher.
The prevalence of quantitative research in the empirical research on
gamification is potentially due to a large part of the research being
published in technical and computer science related venues, which
commonly employ quantitative approaches. The small number of fully
qualitative research papers (16.8% of the empirical studies) is also
noteworthy, and may again be a consequence of the disciplines and
research fields where the research is published. However, if the mixed
methods studies are accounted for, then qualitative research ap-
proaches feature in nearly 40% of the empirical studies.
Table 8 reports the data gathering methods employed in the em-
pirical research papers, the types of data used for their analyses, and the
analysis methods that were used. The most commonly used method of
gathering data (and consequently the most common form of data) was
by surveys and questionnaires, with the data being analyzed either
qualitatively or quantitatively. The second most common data gath-
ering method used a system implementation or prototype, from which
some form of usage or log data was gathered. Also popular were ex-
perimental settings and interviews methods. For the data types, various
forms of feedback and observation data formed also large categories.
Regarding the analysis methods employed, quantitative descriptive
analyses as well as qualitative analyses were reported in most studies.
While there was a wide variety of data gathering methods and data
types used in the analyses, the research approaches tended to
Table 7
Type of empirical research.
Empirical type Frequency Frequency % %
Quantitative 165 60.4
descriptive 85 51.5
inferential 80 48.5
Qualitative 46 16.8
Mixed 62 22.7
Total 273 100
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
197
concentrate on certain methods and data types. A popular structure for
data gathering was where a gamification implementation or a prototype
had been built, then a group of study participants used the im-
plementation and some kind of usage data was gathered. The partici-
pants also tended to take part in a quantitative or a qualitative survey
that was either numerical or with open-ended questions.
3.5. Affordances
Altogether, 47 different affordances were identified in the studies
(see Table 9). These were subsequently grouped based on their type,
into achievement/progression-oriented (10 affordances), social-or-
iented (7), immersion-oriented (5), real world-related (8) and mis-
cellaneous (17) elements. On average, the 273 empirical studies ex-
amined implementations or systems with 3.5 affordances. The
affordance identifications were formed, depending on how the authors
referred to the elements in their paper, without further analysis on si-
milarities or differences between the affordances featured in different
studies. This decision was taken due to the amount of the reviewed
papers, which would have made a more detailed analysis of the im-
plemented affordances more challenging. Although some overlap be-
tween the individual elements may exist (e.g. displaying a score might
be very tightly connected to a progress bar in a system), for the pur-
poses of this overview, each element was assigned to only one category.
The most commonly implemented gameful affordances involve
various forms of points and scoring. Also different forms of challenges,
clear goals, achievements and leaderboards were among the most fre-
quent elements. Generally in game design, points, achievements and
leaderboards have been categorized as goal metrics that provide per-
formance feedback to the player (Zagal, Mateas, Fernandez-Vara,
Hochhalter, & Lichti, 2005). The prevalence of these affordances in
gamification implementations is potentially due to them being easily
applicable to various types of existing system (Mekler, Brühlmann,
Tuch, & Opwis, 2015).
In this study, the frequency of various challenge/clear goal im-
plementations differs from the findings of previous reviews (Hamari
et al., 2014), where such affordances were not as common. In this re-
view, all of the studies which explicitly stated to contain some form of
challenge, quest, mission, task or clear goal were categorized into this
group. However, it is clear that challenges and goals overlap heavily for
example with badge-type affordances, which generally provide a goal
to work towards (Hamari, 2017). Similarly for instance, leaderboards
and levels may act as goals that a user aims to reach. Therefore, the
difference between this and prior reviews with regards to the pre-
valence of the challenge/clear goal affordances might be due to dif-
ferences in the coding of the literature.
In the reviewed literature, the application of various achievement or
progression signaling affordances is generally the most common way to
gamify activities. The second most frequent provision of affordances
involved social elements in various forms. Especially, different features
common to social networking services, e.g. friending, liking, status
updates, commenting and profile pages are often implemented as ga-
mification features. Cooperation and team-based activities also featured
frequently among social affordances. Only 25 papers clearly articulated
competition as an element, however, gamification implementations
with leaderboards or other means of social comparison also promote a
sense of competition among users. Therefore, competition as a concept
is most likely to be more prevalent in actual implementations, but not
necessarily specified in the research reports.
Various immersion-oriented affordances such as the use of stories
and narratives, avatars, virtual worlds etc. were featured, but these
were not as commonly implemented as achievement and social affor-
dances.
The analysis of the affordances employed in the reviewed empirical
literature indicates that the triad of points, badges and leaderboards
continues to dominate the landscape of gamification. Several critical
Table 8
Data gathering methods employed in the reviewed body of literature.
Data gathering method Frequency Data type Frequency
Survey, questionnaire (qualitative and
quantitative)
179 Survey data 167
Implementation, prototype 161 Use data, log data 128
Experiment, trial 78 Interview data 48
Interviews 53 Feedback 25
Course, learning session 18 Observation data 21
Observation 16 Test scores, assignment scores 15
Focus group 9 Audio, video recordings, photos 6
Diaries 6 Diary entries, daily reports 5
Course assignments, education related
assignments, tests
6 Focus group data, discussion data 5
Data from a system or a platform (existing
systems; commercial products)
5 Course attendance data, other course data 3
Workshop 4 Field notes, experiential data (phenomenological data) 3
Data mining 3 Blog texts, forum discussions 2
Video data gathering 2 Delphi method data 1
Case study 2 Psychophysiological data 1
Ethnography 2 Brainstorming data 1
Google blog search 1 Analysis method Frequency
Audio data gathering 1 Quantitative descriptive 136
Phenomenological assessment 1 Qualitative 100
Photography 1 Quantitative modeling
(T-tests, Regressions, Mann-Whitney U-test, Wilcoxon rank-sum test, Structural equation
modeling, Wilcoxon signed rank test, Welch’s t-test)
70
Delphi method 1 Quantitative comparisons
(ANOVA/ANCOVA/MANOVA, Kruskal-Wallis)
44
Work session 1 Quantitative association-based
(Correlations, Chi squares, Factor analysis, Crosstabs, Spearman’s rho)
30
Contest 1 Statistical quantitative
(Binomial tests, Social network analysis, Growth curve analysis, Cluster analysis, Logistic models,
Fuzzy AHP, Granger causality test, Hierarchical linear modeling, Z-test for proportions)
11
Cognitive ability test 1
Forum discussions 1
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
198
views regarding the prevalence of these elements have been voiced,
suggesting that employing such affordances without further con-
sideration of the context or the users results in mere “pointsification” of
the activities (see Deterding, 2015). One reason for the continued po-
pularity of these elements may be that inserting them as an additional
layer to an existing system can be achieved without undue effort
(Mekler et al., 2015). Another potential explanation for the popularity
of the blueprint design of points, badges and leaderboards may be that
many gamification design guides and frameworks approach gamifica-
tion design from a pattern-based perspective (Seffah & Taleb, 2012),
and suggest designing gamification implementations by basically
choosing the affordances from a given list of elements (Deterding,
2015).
In reality, gamification is difficult to design for three main reasons:
1) games are complex, multifaceted, and therefore difficult to holi-
stically transfer to other environments; 2) gamification involves moti-
vational information system design (Hamari, 2015) which entails an
understanding of (motivational) psychology; and 3) the goal of gami-
fication is often to affect behavior, and this adds yet another layer to the
scope of gamification design. Moreover, gamification design is targeted
to a variety of audiences and activities, as well as serving a range of
motivational needs which individuals may have in the varying gamified
contexts. Deterding (2015) highlights the tendency of gamification
guides and frameworks to promote the idea of a certain design pattern
leading to a specific effect. However, as Deterding (2015) notes, from a
psychological perspective, the motivational effect of every given si-
tuation is the result of “situated, active interpretation”. In other words,
how an individual perceives the gamification is highly dependent on
the nature of the activity, the contextual factors related to it, as well as
the specific situation where the system is being used - all in addition to
the individual’s own personal and demographic characteristics.
As called for by Hamari et al. (2014), some studies looking at in-
dividual elements and their effects have started to appear (see e.g.
Mekler et al., 2015; Christy & Fox, 2014). These studies have provided
valuable information for research and practice by examining a specific
relationship between an affordance and an outcome. Given their
context dependency, the results of such studies are not universally ap-
plicable to all gamification settings, but they still provide much needed
support for making design choices.
3.6. Outcomes
The empirical studies were examined for any psychological and
behavioral outcomes that were featured in the papers (reported in
Tables 10 and 11 respectively). Psychological outcomes were quanti-
tatively studied in 138 of the 273 empirical studies. The analysis of
psychological outcomes indicates that the empirical research on gami-
fication is mostly interested in how gamification implementations are
perceived and experienced as systems, whether they are enjoyable or
useful, and whether the users feel motivated by the systems.
Most commonly, the empirical research papers examined percep-
tions of using a system, some specific system features, or some other
assessments of use experiences. In a large portion of the studies, an
implementation or a prototype had been developed and was subse-
quently studied with its users, so it is not surprising that inquiries re-
garding user experiences and perceptions are commonplace.
Other frequently studied psychological outcomes reflect the most
common reasons for implementing gamification. As games are gen-
erally associated with experiences of enjoyment, the application of
gamification is often laden with an intention of creating enjoyment for
the user. Enjoyment and experiences of “fun” were the second most
frequent psychological outcome featured in the empirical research pa-
pers. Similarly, gamification is commonly framed as a method for in-
creasing motivation towards various activities and tasks. Consequently,
many of the empirical studies examine motivation as a psychological
outcome.
Further aspects such as perceived usefulness or effectiveness, or the
ease of use or effort required to use a system were frequently examined
as psychological outcomes. According to theories on technology ac-
ceptance and adoption, these aspects are considered to be important
determinants for the continued use of various systems.
Social aspects were also studied in many of the empirical research
Table 9
Affordances studied in the empirical research papers.
Affordances Affordances
Achievement/progression Non-digital elements
Points, score, XP 138 Real world/financial reward 16
Challenges, quests, missions, tasks, clear goals 91 Check-ins, location data 16
Badges, achievements, medals, trophies 85 Motion tracking 10
Leaderboards, rankings 82 Physical cards 4
Levels 59 Physical playboard 2
Performance stats (includes visualization of agreement in crowdsourcing), performance
feedback
46 Real world interactive objects 1
Progress, status bars, skill trees 32 Physical objects as game resources 1
Quizzes, questions 32 Physical dice 1
Timer, speed 23 Miscellaneous
Increasing difficulty 11 Full game (also board games), also commercial gamification systems not
described
17
Social Assistance, virtual helpers 15
Social networking features 49 Virtual currency 10
Cooperation, teams 47 Reminders (to create engagement), cues, notifications, annotations 9
Competition 25 Retries, health, health points 7
Peer-rating, also betting to review work of others 17 Onboarding (safe environment to practice the rules), benefits for
beginners
3
Customization, personalization 7 Adaptive difficulty 3
Multiplayer 3 Game rounds 2
Collective voting 1 Warnings 1
Immersion Penalties 1
Avatar, character, virtual identity 29 Game slogans 1
Narrative, narration, storytelling, dialogues, theme 22 Funny movies 1
Virtual world, 3D world, game world 14 Virtual pets 1
In-game rewards 13 Trading 1
Role play 6 Making suggestions 1
Virtual objects as augmented reality 1
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
199
papers, but there was a wide variation in how the associated variables
were measured or studied. Social influence or subjective norms (i.e.
one’s perceptions of the opinions of meaningful others: Ajzen, 1988,
1991; Fishbein, 1979; Fishbein & Ajzen, 1975) are the most commonly
investigated social aspects. The recognition from others and the sense of
relatedness with other system users were also frequently studied.
The variety of psychological outcomes in the empirical research is
vast, however, due to the wide dispersion of the research models, there
is no notable accumulation of knowledge from any given perspective.
For example, enjoyment/fun, or usefulness/effectiveness outcomes are
often examined with specific instruments developed for a particular
study. Furthermore, while many studies have examined, e.g. the per-
ceptions of use of a certain system, these results tend to be specific to
the system, and thus do not provide much possibility for generalization.
Compared to the psychological outcomes studied in the empirical
papers, the variety of behavioral outcomes (studied in 166 of the 273
empirical papers) is more limited. A clear majority of the empirical
works studied interaction with a system, or some specific performance-
metric related to its use. Among these metrics, time-related variables
for performance were the most frequently examined. Also, a measure-
ment of the amount and quality of contributions to a system was seen to
be common.
The behavioral outcomes also reflect the popularity of education as
the main domain for the study of gamification. Course or assignment
grades, and other forms of measuring academic performance were
among the behavioral outcomes that were more frequently studied.
Badges, points, and leaderboards are often considered to be the
blueprint of gamification, as many gamification implementations rely
on them as affordances. The more frequently examined performance-
related outcomes also include the number of badges or points gained, or
leaderboard positions.
The psychological and behavioral outcome categories both feature a
wide variety of different aspects that have been studied, but even
within the outcomes that have been grouped together for this review,
there is considerably little consistency across measurement instru-
ments. Even though gamification implementations and their outcomes
are highly dependent on context, an improved consistency of how
outcomes are measured would increase the comparability of research
results, despite their differences in implementation.
3.7. Results in reviewed literature
In their 2014 review, Hamari et al. (2014) noted that the research
on the topic of gamification showed several methodological short-
comings. Hamari et al. suggested for example paying attention to
samples sizes, experiment timeframes, the use of validated measure-
ment instruments, and the use of controls in experimental studies in
future research. Several studies have since acknowledged these sug-
gestions, and have sought to strengthen the depth of research by ad-
dressing known gaps (see e.g. Hanus & Fox, 2015; Mekler et al., 2015).
In this review, only the results and the sample sizes from experi-
mental quantitative studies (N=66) were analyzed, and not those of
the whole body of empirical studies. This decision was made in order to
limit the analysis to studies where hypotheses were tested, and thus
clear indications of the results were provided.
Within this group of controlled experimental quantitative studies,
the median sample size was 74.5 and the sample mean was 1165. The
sample sizes ranged from 1 to 42,724. A clear increase in the sample
sizes is thus visible when compared to the earlier review by Hamari
Table 10
Psychological outcomes studied in the empirical research papers.
Psychological outcomes Psychological outcomes
Overall assessment / general attitude of
the use of the gamified system
Social interaction
Perceptions of use, use experience,
perceptions of system and
features
54 Subjective norm, social
influence
7
Preference of system type/features 7 Recognition 5
Perception of course, perception of
gamification in education
4 Relatedness 4
Affective Reciprocity 3
Perceived enjoyment, fun 34 Network effects 3
Engagement 12 Perceived socialness, social
context
3
Affect, emotional experience 7 Perceived competition 3
Flow 6 Social comparison 1
Playfulness 3 Social skills 1
Immersion 2 Psychological states and traits /
personality features
Cognitive Motivation (also orientation
towards various motivations)
25
Perceived usefulness, perceived
effectiveness
23 Interest 10
Perception of learning 8 Perceived competence 9
Perceptions of additional benefits,
customer ROI
6 Autonomy 4
Involvement, participation 2 Quality of life, flourishing 4
Perception of contribution 1 Empowerment 3
Effort in use / Experienced challenge Awareness 3
Ease of use 14 Personality, user types 3
Effort, perceived difficulty,
challenge
10 Mood 2
Perceived stress, cognitive load 4 Self-efficacy, confidence 2
Frustration, annoyance 3 Attentional bias 1
Workload 3 Anxiety 1
Perceived physical exertion 1 Perceived control 1
Attitude Familiarity 1
Satisfaction 8 Identification 1
Attitude 7 Loyalty 1
Predisposition to changes 1 Disengagement 1
Comfort with sharing data 1 Vigilance 1
Perception of one's work 1 Focus 1
Table 11
Behavioral outcomes studied in the empirical research papers.
Engagement/interaction with the
system
Physical etc. measures
Behavioral outcomes Behavioral outcomes
Participation in a system, system use 39 Physical activity 10
Use intentions, willingness to use,
intentions to continue activity
13 Health care activity 6
Participation in discussions 10 Medication over/misuse 2
Course material views, downloads 9 Stress level 2
Course attendance, exam attendance 6 Energy use in exercise, intensity
of exercise
2
Effects of gamification on site use 1 Psychophysiological measures 2
Purchase intentions 1 Anxious behavior 1
Knowledge transfer 1 Mental processes 1
Performance Pain burden 1
Speed, time 34 Social interaction
Amount of contributions/content
produced
27 Cooperation 4
Course grade, assignment grade,
academic performance
26 Social actions 3
XP, points, score gained 17 Word of mouth 2
Quality of contributions 16 Requests for help 2
Learning, skill progression 12 Recommending intentions 1
Badges earned, tracking of badges 12 Size of network, amount of
friends
1
Number of assignments, amount of
contributions in class
7 Agreement over content 1
Number of attempts 5 Miscellaneous
Accuracy 3 Ecological behavior 3
Leaderboard positions 3 Functionality of software 3
Acting on time 2 Retention and attrition of users 2
Number of transactions, number of
trade proposals
1 Disease knowledge 1
Behavioral strategies 1
Behavior change 1
Amount of problems 1
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
200
et al. (2014) where sample sizes were seen to include around 20 study
participants. However, it should be noted that in their study, the body
of literature contained empirical research in general, while in this re-
view, sample sizes were analyzed only for a particular type of empirical
research papers. Including the whole body of current empirical litera-
ture in the analysis could change the median sample sizes reported
here, however, improvements are clearly visible in this area.
The results reported in the 66 identified controlled experimental
quantitative studies (Table 12) show that while positive research
findings are frequent (reported in 28.7% of the papers), a clear majority
of the studies still report somewhat mixed results, i.e. the papers report
negative or inconclusive results in addition to positive results. In order
to further examine the mixed results, they were categorized as either
‘Mixed with positive’, ‘Null or equal positive and negative’ or ‘Mixed
with negative’, depending on whether the majority of tests had yielded
positive or negative results. Based on this analysis, mixed but mainly
positive results can be seen to have been reported in nearly half (47.0%)
of the 66 controlled experimental studies. Fully negative results were
only reported in 2 of the 66 quantitative experimental studies.
The results of the controlled experimental quantitative studies were
also analyzed by grouping the results by study domain, and also cate-
gorizing the results based on the affordances employed in the studies.
Grouping the results by domain shows that in the largest domain of
education/learning, most of the studies reported fully positive results
(35.7%, 10 studies of 28), and mixed but mostly positive results were
reported in nearly as many studies (32.1%, 9 studies of 28). In the next
largest domains (health/exercise and crowdsourcing), most of the stu-
dies report mixed but mostly positive results. For the rest of the do-
mains, the number of the studies in each domain was 3 or less, so no
meaningful conclusions could be drawn.
The results of the studies were also grouped by the affordances they
employed (see Table 13). It is evident that within this set of studies, the
triad of points, badges and leaderboards form the top three most
common affordances. Drawing further conclusions based on the re-
lationships of the implemented affordances and the results of the stu-
dies is difficult, as most of the studies have examined the effects of a
gamification system as a whole, or tested several affordances at once.
Thus, there is little possibility of identifying which of the affordances
actually produced the effects. However, this is not a new observation,
and has previously been acknowledged as a problem within the field of
gamification research (see e.g. Hamari et al., 2014).
We also identified controlled experimental studies in which the ef-
fects of certain affordances had been controlled for (see Table 14). In
the current body of controlled experimental studies, only 11 of the 66
studies examine the effects of only a single affordance at a time. Among
these, the most studied affordances are points, badges and leader-
boards, with points being examined in 4 papers, badges or other
achievements in 4 papers, and leaderboards or rankings in 4 papers.
The finding that most of the controlled experimental studies re-
ported mixed results implies that some effects of the featured gamifi-
cation experiment were positive, while others showed inconclusive or
negative effects. This provides further support to the conclusions of
previous gamification reviews (Hamari et al., 2014; Seaborn & Fels,
2015) that gamification is not a silver-bullet type of solution for
achieving positive results and success, in either the research sphere, or
in practice.
The small number of studies with fully negative results is however
interesting. The possibility of a publication bias (as mentioned in
Hamari et al., 2014) must be taken into account when considering these
findings, and authors as well as publication venues may be more likely
to publish positive rather than negative or inconclusive results. Whe-
ther such a phenomenon has had a significant effect on the current body
of literature is not known.
4. Future Research Agenda
This review has presented thus far the most comprehensive and
widest look at gamification research; a field which has seen noteworthy
increases in the past few years. The reason for conducting such a wide
review was twofold: Firstly, attaining a wide overview of the devel-
oping field is beneficial for comprehending how gamification research
has developed, and what type of knowledge has been gained; and
secondly, taking an overarching look at the literature can offer valuable
information that will guide future research endeavors (see Paré et al.,
Table 12
Results of the controlled experimental quantitative studies by domain. The number in italics refers to the percentage from the total number of papers in the given
domain.
Domain Positive Mixed with
positive
Null or equally positive and
negative
Mixed with
negative
Negative Sum
Education/Learning 10
35.7
9
32.1
7
25.0
1
3.6
1
3.6
28
Health/Exercise 4
30.8
7
53.8
2
15.4
13
Crowdsourcing (includes information gathering, knowledge sharing and
citizen science)
1
9.1
7
63.6
2
18.2
1
9.1
11
Ecological/environmental behavior 1
33.3
2
66.7
3
Social behavior/social networking/social sharing 2
66.7
1
33.3
3
Marketing/Consumer behavior 2
100.0
2
Business/Management 1
100.0
1
eCommerce/eServices 1
100.0
1
Innovation 1
100.0
1
Software development/design 1
100.0
1
Transportation/Mobility 1
100.0
1
Work 1
100.0
1
Total 19
28.7
31
47.0
12
18.2
2
3.0
2
3.0
66
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
201
Table 13
Results of the controlled experimental quantitative studies by affordances implemented in the studies. The number in italics refers to the percentage from the total
number of papers featuring the given affordance type.
Affordances Positive Mixed with
positive
Null or equal
positive and negative
Mixed with
negative
Negative Sum
Points, score, XP 11
31.4
18
51.4
4
11.4
1
2.9
1
2.9
35
Badges, achievements, medals, trophies 9
36.0
9
36.0
5
20.0
1
4.0
1
4.0
25
Leaderboards, ranking 7
29.2
10
41.7
5
20.8
1
4.2
1
4.2
24
Challenges, quests, missions, tasks, clear goals 7
36.8
8
42.1
2
10.5
2
10.5
19
Performance stats (includes visualization of agreement in
crowdsourcing), performance feedback
4
30.8
8
61.5
1
7.7
13
Levels 2
18.2
5
45.5
3
27.3
1
9.1
11
Timer, speed 5
50.0
3
30.0
1
10.0
1
10.0
10
Social networking features 4
40.0
4
40.0
1
10.0
1
10.0
10
Real world/financial reward 1
11.1
7
77.8
1
11.1
9
Narrative, narration, storytelling, dialogues, theme 4
44.4
4
44.4
1
11.1
9
Progress, status bars, skill trees 5
62.5
2
25.0
1
12.5
8
Competition 5
62.5
3
37.5
8
Cooperation, teams 2
25.0
4
50.0
1
12.5
1
12.5
8
Quizzes, questions 1
20.0
4
80.0
5
Assistance, virtual helpers 2
40.0
2
40.0
1
20.0
5
Full game (also board games), also commercial gamification systems
not described
3
60.0
2
40.0
5
Increasing difficulty 3
75.0
1
25.0
4
Avatar, character, virtual identity, 1
25.0
2
50.0
1
25.0
4
Virtual world, 3D world, game world, simulation 1
25.0
3
75.0
4
Reminders (to create engagement), cues, notifications, annotations 2
66.7
1
33.3
3
Motion tracking 3
100.0
3
Customization, personalization 2
66.7
1
33.3
3
In-game rewards 1
50.0
1
50.0
2
Virtual currency 1
50.0
1
50.0
2
Check-ins, location data 1
50.0
1
50.0
2
Warnings 1
100.0
1
Adaptive difficulty 1
100.0
1
Retries, health, health points 1
100.0
1
Collective voting 1
100.0
1
Physical dice 1
100.0
1
Physical cards 1
100.0
1
Physical play board 1
100.0
1
Game rounds 1
100.0
1
Total 84 110 29 11 5 239
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
202
Ta
bl
e1
4
Co
nt
ro
lle
de
xp
eri
me
nt
al
stu
die
se
xa
mi
nin
gt
he
eff
ec
ts
of
on
ea
ffo
rd
an
ce
at
at
im
e.
Re
fer
en
ce
Do
ma
in
Aff
or
da
nc
es
Stu
die
dp
sy
ch
olo
gic
al
ou
tco
me
s
Stu
die
d
be
ha
vio
ral
ou
tco
me
s
Re
su
lts
At
tal
i&
Ar
iel
i-A
tta
li,
20
14
Ed
uc
ati
on
Po
int
s,
sco
re,
XP
Ti
me
r,
sp
ee
d
Qu
izz
es,
qu
est
ion
s
Ac
cu
rac
y
Sp
ee
d,
tim
e
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Bu
ism
an
&
va
nE
ek
ele
n,
20
14
Ed
uc
ati
on
Po
int
s,
sco
re,
XP
Le
ad
erb
oa
rd
s,
ran
kin
gs
Mo
tiv
ati
on
(al
so
or
ien
tat
ion
tow
ard
sv
ari
ou
s
mo
tiv
ati
on
s)
Pe
rce
ive
de
njo
ym
en
t,
fun
En
ga
ge
me
nt
Inv
olv
em
en
t,
pa
rti
cip
ati
on
Nu
mb
er
of
as
sig
nm
en
ts,
am
ou
nt
of
co
nt
rib
ut
ion
si
nc
las
s
Co
ur
se
gr
ad
e,
as
sig
nm
en
tg
rad
e,
ac
ad
em
ic
pe
rfo
rm
an
ce
XP
,p
oin
ts,
sco
re
ga
ine
d
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Ch
oi
et
al.
,2
01
4
Cr
ow
ds
ou
rci
ng
Re
al
wo
rld
/fi
na
nc
ial
rew
ard
Pe
rce
ive
de
njo
ym
en
t,
fun
Am
ou
nt
pr
od
uc
ed
Qu
ali
ty
of
co
nt
rib
ut
ion
s
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Ch
ris
ty
&
Fo
x,
20
14
Ed
uc
ati
on
Le
ad
erb
oa
rd
s,
ran
kin
gs
Ide
nt
ifi
ca
tio
n
Nu
ll
res
ult
so
re
qu
all
yp
os
iti
ve
an
d
ne
ga
tiv
er
esu
lts
Ha
ku
lin
en
,A
uv
ine
n
&
Ko
rh
on
en
,
20
13
Ed
uc
ati
on
Ba
dg
es,
ac
hie
ve
me
nt
s,
me
da
ls,
tro
ph
ies
Sp
ee
d,
tim
e
XP
,p
oin
ts,
sco
re
ga
ine
d
Ba
dg
es
ga
ine
d,
tra
ck
ing
of
ba
dg
es
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Ha
ku
lin
en
,A
uv
ine
n
&
Ko
rh
on
en
,
20
15
Ed
uc
ati
on
Ba
dg
es,
ac
hie
ve
me
nt
s,
me
da
ls,
tro
ph
ies
Pe
rce
pti
on
so
fu
se,
us
ee
xp
eri
en
ce
,p
erc
ep
tio
ns
of
sy
ste
m
an
df
ea
tu
res
Sp
ee
d,
tim
e
XP
,p
oin
ts,
sco
re
ga
ine
d
Ba
dg
es
ga
ine
d,
tra
ck
ing
of
ba
dg
es
Nu
mb
er
of
att
em
pts
Nu
ll
res
ult
so
re
qu
all
yp
os
iti
ve
an
d
ne
ga
tiv
er
esu
lts
Ha
ma
ri,
20
13
Ec
om
me
rce
Ba
dg
es,
ac
hie
ve
me
nt
s,
me
da
ls,
tro
ph
ies
Ba
dg
es
ga
ine
d,
tra
ck
ing
of
ba
dg
es
So
cia
la
cti
on
s
Nu
mb
er
of
tra
ns
ac
tio
ns
,n
um
be
ro
ft
rad
e
pr
op
os
als
Nu
ll
res
ult
so
re
qu
all
yp
os
iti
ve
an
d
ne
ga
tiv
er
esu
lts
La
nd
ers
&
La
nd
ers
,2
01
5
Ed
uc
ati
on
Le
ad
erb
oa
rd
s,
ran
kin
gs
Sp
ee
d,
tim
e
Co
ur
se
gr
ad
e,
as
sig
nm
en
tg
rad
e,
ac
ad
em
ic
pe
rfo
rm
an
ce
Po
sit
ive
res
ult
s
Lo
ng
&
Al
ev
en
,2
01
4
Ed
uc
ati
on
Ba
dg
es,
ac
hie
ve
me
nt
s,
me
da
ls,
tro
ph
ies
Re
tri
es,
he
alt
h,
HP
s
Pe
rce
ive
de
njo
ym
en
t,
fun
Le
arn
ing
,s
kil
lp
ro
gr
ess
ion
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Me
kle
re
ta
l.,
20
13
Cr
ow
ds
ou
rci
ng
Po
int
s,
sco
re,
XP
Le
ve
ls
Le
ad
erb
oa
rd
s,
ran
kin
gs
Mo
tiv
ati
on
(al
so
or
ien
tat
ion
tow
ard
sv
ari
ou
s
mo
tiv
ati
on
s)
Pe
rce
ive
dc
om
pe
ten
ce
Au
ton
om
y
Sp
ee
d,
tim
e
Am
ou
nt
pr
od
uc
ed
Qu
ali
ty
of
co
nt
rib
ut
ion
s
Mi
xe
dr
esu
lts
wi
th
mo
stl
yp
os
iti
ve
Th
om
,M
ille
n
&
Di
Mi
cc
o,
20
12
So
cia
ln
etw
or
kin
g
Po
int
s,
sco
re,
XP
Am
ou
nt
pr
od
uc
ed
Po
sit
ive
res
ult
s
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
203
2015). Therefore, within the next sections, we provide an agenda for
future gamification research. The agenda is divided into three sections:
thematic, theoretical and methodological agendas. As concrete con-
clusions, we formulate 15 agenda points (five from each perspective)
suggesting future directions and foci for gamification research.
4.1. Thematic agenda
Based on the analysis of the reviewed literature, it is clear that the
empirical research has concentrated mainly on education and learning,
as well as health and exercise. In other words, research has focused on
domains where activities and behaviors typically demand long-term
perseverance, where activities are complex and multifaceted, goals are
difficult to set, and progress is challenging to track and quantify; ac-
tivities that are commonly riddled with procrastination and other in-
efficiencies. Based on the amount of research, gamification is con-
sidered as both applicable and beneficial for these types of activities,
where continuity and long-term engagement is often needed for gaining
lasting results.
In addition to the domains that prevail in the body of literature,
thematically, a large portion of the existing research is aimed at sup-
porting individualistic motivations such as self-care and self-manage-
ment. Encouraging communal engagement and cooperative activity is
considerably less studied, but would provide an interesting potential
avenue of research. This thematic gap can be noted on the level of af-
fordances, as well as on the level of domains. The most commonly
implemented affordances such as points and leaderboards often require
that other users exist in the system and that one can compare results,
but the emphasis of these affordances is still on individual development
and progress. Affordances supporting collective behavior, such as co-
operation, teams or networking are considerably less used elements in
current gamification implementations. On the level of domains, con-
texts such as learning and health would evidently benefit from collec-
tive or cooperative perspectives, but domains that specifically look to
support collective activity or well-being, such as citizen engagement for
communal development, are clearly in the minority.
As social human beings, we seek senses of relatedness (Deci & Ryan,
2000; Deci & Ryan, 1985; Ryan & Deci, 2000) and collectivity, so col-
laboration and cooperation are natural behaviors for us to engage in.
However, inducing collaboration is a well-known challenge. This is
especially so in complex contexts such as societal activities, which are
abstract and difficult to perceive, and may easily seem irrelevant to
one’s everyday life. Games, however, are well-known for their ability to
induce collaborative behavior, even among complete strangers, and this
is often seen in online multiplayer games. There is ample evidence
which indicates that people enjoy playing together (Chen, Sun, & Hsieh,
2008; Cole & Griffiths, 2007; Scharkow, Festl, Vogelgesang, & Quandt,
2015; Teng & Chen, 2014; Teng, 2017; Yee, 2006), and collaboration
often emerges seamlessly, effortlessly and organically. Therefore, in-
stead of focusing strongly on individual motivation and behaviors, ga-
mification research could also be beneficial in increasing our under-
standing of how to induce and maintain collective and collaborative
behaviors.
Some interesting glimpses of collective level affordances and goals
already exist within the gamification research field. Jones et al. (2014)
report a study featuring a gamification system for motivating school-
children to consume more fruit and vegetables during school lunches.
Their study included school-wide collective goals, cooperative action
and collective rewards. Another example comes from Laureyssens et al.
(2014), who reported a study on citizen engagement. Through various
gameful affordances (including teams and cooperation), they aimed for
“augmenting community participation in urban neighborhoods”
(Laureyssens et al., 2014). In the field of knowledge and information
management, Araújo and Pestana (2017) report their work seeking to
support social well-being, team work and skills management in an or-
ganizational context via gamification. Moreover, Morschheuser,
Hamari and Maedche (2018) revealed that gamified solutions com-
bining both cooperative and competitive structures may prove most
effective. More research is, however, needed to better understand how
gamification can be harnessed for inducing collective and collaborative
behavior.
Thematic agenda point 1) Future gamification research should seek
to explore the possibilities of cooperative and collective gamification ap-
proaches.
Based on the findings of this review, current gamification research is
mainly focused on implementing “the blueprint” of gamification: i.e.
points, badges and leaderboards (See Table 9; Deterding, 2015; Hamari
et al., 2014). However, when considering games, the diversity of ele-
ments they contain is vast. Unfortunately, in gamification design, this is
often ignored and the implementations are reduced to a replication of
the blueprint triad (see e.g. Deterding, 2015). In the current gamifica-
tion research, progress and achievement-oriented affordances are
clearly the most commonly used, whereas e.g. immersion-related af-
fordances (such as narratives and avatars) are much less frequent.
While gamification is promoted as inducing experiences character-
istic to games, in non-gameful contexts, the limited perceptions of ga-
mification design signal a limited perception of gameful experiences in
general. Most gamification designs are currently focused on achieve-
ment-oriented mentalities, and the type of experiences and motivations
they afford. However, research on the motivations to play games in-
dicates that the drivers of the behavior are considerably more diverse;
and while some players play games for achievement-related gratifica-
tion, some are motivated by social aspects, some by immersing them-
selves into stories and roleplay, and some by a combination of these
elements (see e.g. Hamari, Hassan & Dias, 2018; Hamari & Tuunanen,
2014; Kallio, Mäyrä, & Kaipainen, 2010; Yee, 2006). Therefore, aiming
to comprehensively explore gameful experiences and design gamifica-
tion to cater for a wider variety of motivations is a theme that may be
considered in future research (see Morschheuser, Hassan, Werder, &
Hamari, 2018 for a gamification design review).
Beyond the diversity of experiences that game mechanics afford,
there have also been new developments in gaming technology. For
example, recent developments and successes in virtual reality tech-
nology and other forms of immersive, reality-augmenting designs (e.g.
those implemented in the recent Pokémon Go game), may offer inter-
esting future directions for gamification research. There are signs that
this is beginning to happen, and interesting findings on virtual and
augmented reality based gamification has been published in the con-
texts of education (Reitz, Sohny, & Lochmann, 2016) and health (Yates,
Kelemen, & Sik Lanyi, 2016).
Thematic agenda point 2) Future gamification research should seek
to diversify the use of gameful affordances, and concurrently develop an
understanding of what constitutes and creates gameful experiences.
It is notable that gamification research is highly concentrated in
terms of the domains in which it is investigated. The popularity of the
domains of education and learning as well as health and exercise has
been highlighted in this review. Especially, the depth and breadth of
research in the education and learning domain sets an example for
future research; in addition to the sheer number of research papers,
studies conducted in the education field range from testing a single
gamification element in a controlled experiment (e.g. Denny, 2013;
Christy & Fox, 2014), to large scale gamification of semester-long
classes or courses (e.g. Hanus & Fox, 2015; de-Marcos et al., 2014).
Other domains such as crowdsourcing, social behavior and networking,
and software development have also started to gain a place in the re-
search field, but the majority of these domains have only been in-
vestigated by a handful of studies. Furthermore, while the potential of
motivational systems such as gamification has been largely noted in
information management and general information system science fields
(see e.g. Liu et al., 2017; Hamari & Koivisto, 2015a; Araújo & Pestana,
2017; Blohm & Leimeister, 2013; Morschheuser et al., 2017), the extent
of empirical research examining e.g. gamification in management and
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
204
business contexts is still very limited. Beyond the fact that some specific
domains remain understudied, the narrow scope of domains sheds a
shadow on the entire field of gamification research. Heavy emphasis
only in a few domains affords an unbalanced view of how gamification
works. The reason for this is that the contextual factors affect the out-
comes of the gamification in the different domains and therefore, ap-
plying results from one field to another might not provide similar re-
sults.
Moreover, domains such as citizen and public activity, or welfare
and social services, are contexts which increasingly call for engagement
and collaborative approaches (see e.g. Bista et al., 2014; Sanchez-
Nielsen & Lee, 2013). However, these domains have not yet attracted
the attention of researchers to any significant degree. Evidently, the
gamification of domains such as citizen and public activity requires
ways of engaging large groups of people with varying characteristics
and backgrounds, and is therefore a challenging task. Nevertheless, by
accepting the challenges and tackling the issues found in these domains,
more impactful gamification solutions could potentially be developed.
Thematic agenda point 3) Future gamification research should seek
to widen its thematic perspective with respect to the domains which are being
investigated.
Gamification commonly is focused on the positive impact of tech-
nology on human motivation and behavior (see e.g. Deterding, 2015;
Seaborn & Fels, 2015): research has tended to take a myopic and
narrow view that has zoomed in on the benefits that can be derived
from gamification. As scholars have assumed and expected positive
effects from gamification, the research settings and experiments have so
far lacked resolution to detect any negative effects that extend beyond
the confines of the dependent variables in any given study. This is also
indicated by the relatively small portion of studies that have either
reported or acknowledged negative results in the research literature
(Table 12; Hamari et al., 2014; Seaborn & Fels, 2015). Nevertheless, a
large part of the theoretical discussions indicate that gamification can
also have adverse effects. For example, although game elements are
often implemented in order to create positive affect, many of the af-
fordances may, for example, increase the sense of competition, even if
creating this type of experiences was not the actual goal. A competitive
environment may potentially discourage users, and thus have detri-
mental effects on the activity that the gamification originally aimed to
support (Liu et al., 2013; Santhanam et al., 2016; Vesa et al., 2017).
Furthermore, the goal of gamification commonly is to provide structure
to activities, and to divide them into steps with clear and attainable
goals (see e.g. Hamari, 2013; Landers & Armstrong, 2015). In the terms
of Caillois (1961): gamification brings activities to the ludus end of the
ludus (structured play) – paidia (freeform play) continuum. While the
structuring provided by a gamified system may help users to reach set
goals, it may also limit the means by which they can be reached. When
provided with paths of action that are too concrete or strictly defined,
then creative action and thinking may diminish. This can further harm
the activities that the gamification was intended to support. This is
noteworthy especially in organizational contexts, in domains related to
management and work. As highlighted by Liu et al. (2017), the gami-
fication aspects implemented to an information system should not ob-
struct the instrumental goals of the system.
Moreover, given the ubiquity of gamification and quantification in
practically all fields of modern-day life, our lives are being increasingly
measured and monitored, be it for our own self-interest or for the in-
terests of some other organization or entity. How this increasing pre-
sence of gamification and quantification will affect our lives on a more
holistic level is still unclear.
Thematic agenda point 4) Future gamification research should seek
to explore the potential negative, adverse or non-preferable effects of ga-
mification and how to mitigate them.
It is clear that gamification emerged as a technological phenom-
enon, and especially as a phenomenon of human-computer interaction.
Gamification is primarily thought to entail computers and software that
affect people (and more specifically the explicit elements which feature
in contemporary games). This is natural if we consider that gamifica-
tion mainly arose from the success and popularity of video and online
games during recent decades (Vesa et al., 2017). However, if we think
about games more universally and from a historical perspective, then
digital games are also a rather new phenomenon (Mäyrä, 2008). In the
past, games have commonly consisted of rituals and other non-artefact
driven activities constructed by human dynamics, and within different
forms of organizing (see e.g. Huizinga, 1955). However, if we consider
the variety and degree of gamification, we can immediately notice that
this broader perspective is almost completely absent, both in terms of
temporality and technology (see e.g. Hamari et al., 2014; Seaborn &
Fels, 2015). So, what if we could broaden our understanding and per-
spective of how gamification can be manifested? What if we could
conceptualize that gamification is not only an (information) technology
or human-computer interface, but also a social innovation which stems
from how social dynamics are being shaped and how organizations are
being structured. Thus, as well as borrowing game design from games,
we should also be borrowing player and organizational practices (Vesa
et al., 2017).
Thematic agenda point 5) Also relating to the theoretical agenda, we
suggest future gamification research considers gamification not only as an
innovation of human-computer interaction or information system, but also
as organizational and individual practices reminiscent of those which may be
observed in games.
4.2. Theoretical agenda
Gamification is a new area of research in information systems, and
in addition to the obvious thematic gaps that exist, gaps also exist in our
theoretical and conceptual understanding of the phenomenon. This not
only leads to a partial view of gamification, but also to biases and
shortcomings in research designs which are deployed in the investiga-
tion of gamification. Therefore, future research should seek not only to
fill the thematic gaps, but also to address the following theoretical gaps
which overshadow current research efforts.
The strongest focus in the discussions around gamification centers
around the effects of gamification on human behavior (see e.g. Tables
10 & 11 ; Seaborn & Fels, 2015). Considering that the main premise
behind gamification is to affect motivations and behavior (Huotari &
Hamari, 2017), this focus is evident and intuitively understandable.
Therefore, it might not be surprising that significantly less attention has
been paid to issues and aspects which precede the effects of gamifica-
tion. This indicates that neither the theoretical nor the empirical issues
of the overall gamification context are yet complete. If no attention is
paid to the determinants which lie behind the success of the phenom-
enon, outside the gamification affordances themselves, then we will fail
to see the forest for the trees. For example, while gamification might
have positive effects on the users who choose to adopt it, what will be
the effect on the bulk of users who will not adopt the gamification
features? Some research has already begun to explore issues such as the
adoption of gamification (Hamari & Koivisto, 2015a, 2015b; Herzig
et al., 2012), moderating demographic factors (Bittner & Schipper,
2014; Koivisto & Hamari, 2014), and the effects of personality (Butler,
2014; Hall, Glanz, Caton, & Weinhardt, 2013). However, these studies
have only begun to scratch the surface of the contextual and individual
aspects which affect gamification use. Targeting specific questions re-
levant to e.g. certain demographic groups could yield new interesting
veins of research, such as the work by Talaei-Khoei and Daniel (2018)
exploring the benefits of gameful interactions for cognitive abilities and
transferability of these abilities for seniors. Therefore, the future re-
search agenda on gamification should look to expand its focus in a way
that gamification research becomes less tightly focused on the affor-
dances and outcomes of gamification, and also investigates aspects that
precede the effects of gamification on human behavior and motivation,
such as attitudes and beliefs or personality and demographic issues.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
205
Research investigating the different determinants of why people play
different types of games could prove useful in these endeavours (e.g
Hamari & Keronen, 2017).
Theoretical agenda point 1) Future gamification research should pay
more attention to the pre-determinants/requirements of gamification success,
instead of only the effectiveness of gamification for those users who have
already chosen to adopt it.
Moreover, gamification applications are inherently motivational
information systems that attempt to support people in their goals and
tasks related to the system use (Hamari & Koivisto, 2015b). In more
specific terms, gamification can be seen to make goals more SMART
(Burke, 2014; Hamari, 2013, 2015); that is, more Specific, Measurable,
Attainable, Realistic and Time-bound. According to goal-setting the-
ories and decades of research, such goals assist individuals towards the
attainment of their objective (Locke & Latham, 2002; Mann, De Ridder,
& Fujita, 2013). This phenomenon has been observed and postulated in
the context of gamification in various works (Burke, 2014; Hamari,
2013, 2015; Landers, Bauer, & Callan, 2015). Although several studies
have discussed the relationship between gamification and goal-setting
(see e.g. Hamari et al. 2018), there is a current dearth of literature
relating to goals themselves in gamification. Therefore, for a future
agenda on gamification, we suggest an investigation into the relation-
ship of the effects of gamification, depending on what kinds of goals
users have, how goal-oriented they are, and what kinds of tasks they
look to accomplish. Users do not share the same types of goals, nor do
they have the same orientations toward them. For example, some users
may be more oriented towards the outcomes of goals, whereas other
users are more concerned about the process of reaching their goals
(Locke & Latham, 2002; Mann et al., 2013). Therefore, a pertinent
question is what kinds of gamification initiatives might be better suited
to users, depending on their goals and the orientation they have to-
wards them? Goals differ regarding their defining characteristics, for
example in their difficulty or specificity (Elliot & Harackiewicz, 1994;
Freund, Hennecke, & Riediger, 2010; Mann et al., 2013), hence they
differ in their attainability and goal seeking outcomes (Freund et al.,
2010; Hackel, Jones, Carbonneau, & Mueller, 2016; Landers et al.,
2015; Lunenburg, 2011; Mann et al., 2013). Therefore, the design
principles will differ depending upon the goals they address, and a
single design solution cannot be expected to fit every situation (Koivisto
& Hamari, 2014; Mann et al., 2013; Wang, Schneider, & Valacich,
2015).
Theoretical agenda point 2) Future research into the effectiveness
and adoption of gamification should take into account the role of the user,
their goals (within the information system), and their individual attributes.
In addition to the factors related to users, the usage context as well
as the nature of the gamified service need to be given more attention.
How the users perceive the gamification is highly dependent not only
on the users’ characteristics, but also on how they perceive the context
for the gamification (the domain or the environment where the gami-
fication takes place), as well as the specific activity they are encouraged
to perform (see e.g. Hamari, 2013). Gamification has been heavily
implemented in the education context where it seems to fit rather in-
tuitively: learning new skills, especially in an institutional setting such
as a school, has traditionally been imbued with a similar structure, i.e.
progressing in steps and receiving feedback for each step. Thus, the
gamification of such a context does not feel particularly inappropriate
or awkward. However, when gamifying areas such as health or social
services, the context is evidently a lot more sensitive. For example, the
gamification of an eCommerce service might be perceived differently
than the gamification of a social networking service. The former is
potentially perceived as a very utilitarian context, and playful or ga-
meful elements might lessen the perception of seriousness and divert
the customers away from the service. However, gamifying a social
networking service would not cause such a reaction, due to the context
being more casual in the first place.
The lack of acknowledging the contextual factors in research
suggests a lack of the theoretical understanding surrounding the phe-
nomenon; the factors affecting human behavior, which in the case of
gamification is more often than not the focus of the systems and the
research conducted on them. When not acknowledging, for example,
the environment of the gamification as well as the specific character-
istics of the gamified activity, we risk producing research results which
in reality are not applicable outside the very specific setting of a given
study. Moreover, we ignore the chance for developing more compre-
hensive theoretical understanding of the phenomenon.
Theoretical agenda point 3) Future gamification research should
incorporate the contexts in which the gamification is deployed and in-
vestigated more strongly into research models.
By way of its many motivational affordances, gamification can
primarily be seen to attempt to provide users with feedback.
Gamification provides three types of feedback: 1) cognitive, 2) affec-
tive, and 3) social. Cognitively, gamification commonly uses data on a
user’s behavior to derive points and other indicators of progress, thus
providing instrumental cognitive data about the users’ actions. From
this perspective, gamification can be seen as a decision support system
for the self, quantifying individual rather than organizational activities.
Affective, motivational feedback is at the core of many gameful affor-
dances as game design elements often aim for positive emotional
arousal, such as enjoyment, excitement or interest. Finally, many ga-
mification affordances are inherently social; e.g. high score lists afford
social comparison (Festinger, 1954), and mutual goals can afford sense
of community and strengthen ‘we-intentions’ (Tuomela, 1995). How-
ever, while gamification functions through these feedback systems and
loops, neither the current theoretical understanding of gamification nor
the empirical literature have made any serious inquiries into the dif-
ferent types of feedback that gamification affords. It appears that
feedback functions as an essential mediator between the interaction
with a gamification and the resulting psychological outcomes. Whilst
prior literature has investigated the resultant psychological states and
experiences (such as e.g. usefulness, enjoyment and perceived compe-
tition), it appears that the link between gamification affordances and
resulting psychological states is still unexplored. Thus, it is unknown
through which mechanisms gamification produces the psychological
effects it aims to achieve.
Theoretical agenda point 4) Future gamification research should pay
more attention to the different types of feedback, which kinds of gamification
affordances are best equipped to deliver them, and the effect that the feed-
back has on system users.
The current understanding of gamification highlights that its effects
proceed in a linear chain of events. This is reflected in both the theo-
retical and empirical literature on gamification. For example, the most
cited definitions of gamification by Deterding et al. (2011) and Huotari
and Hamari (2012); Huotari & Hamari, 2017, both conceptualize ga-
mification as a process within which the implemented elements linearly
proceed to affecting psychological states and experiences, and even-
tually user behavior. Similarly, the main body of empirical literature on
gamification (as seen in this and previous reviews) treats gamification
as following a similar process. Even though this is understandable from
the perspective of economizing and simplifying empirical research de-
sign, it offers a rather limited view of the multifaceted complex systems
and processes that gamification entails. Gamification is a dynamic,
cyclical, two-way process in which the technology, the users, and the
contextual factors of the system all contribute to the outcomes which
are achieved. Gamification affects the behavior of the users, who con-
tinue the behavior, but not as the same “clean slates” as when they first
adopted the system. So, it is clear that the behavior is altered directly
due to the effect of the gamification, and that new patterns of behavior
result in new responses to the gamification itself.
Evidently, studying the process of gamification empirically is very
challenging, and addressing these questions most likely calls for the
adoption of qualitative research approaches that can capture the many
facets of the complex phenomenon. However, simply acknowledging
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
206
the multifaceted nature of the gamification beyond the current linear
conceptualizations that are seen in literature and practice would be a
step forward.
Theoretical agenda point 5) Gamification research and its theoretical
conceptualizations should make an acknowledgement of the dynamic, cy-
clical nature of gamification.
4.3. Methodological agenda
While the body of literature on gamification has been growing
steadily, some common issues still hinder the development of the field.
Firstly, the research field is very scattered in terms of the research
models and variables which are used, especially when studying ex-
periences and attitudes. While variables such as enjoyment are often
included in the research models examining gamification, there are
considerably few studies using similar or validated measurement in-
struments for a particular variable. Therefore, comparing research re-
sults or conducting any comprehensive meta-analyses poses a sig-
nificant challenge. In order for the research field to develop, a
consistency in measurement instruments and research models is needed
across studies, in order to gain comparable results. Furthermore, many
empirical studies rely on reporting only descriptive statistics, even
though conducting some form of inferential analyses could also be
feasible in many cases.
Methodological agenda point 1) Future gamification research
should aim for consistency in measurement instruments and research
models, as well as developing the depth of analyses to go beyond a mere
description of data.
Secondly, most of the empirical research on gamification is con-
ducted without control groups. While many of the studies are motivated
by the question of whether the gamification approach is effective or
not, in many cases, the studies eventually fail to accurately answer the
question due to a lack of control groups. Furthermore, even if empirical
approaches are used, a large proportion of the studies rely simply on
user evaluations. Previous literature reviews have already pointed out
these shortcomings (Hamari et al., 2014), and the amount of controlled,
experimental research settings has increased as the field has matured.
However, as noted in this review that out of 227 studies using quanti-
tative methods (165 quantitative studies and 62 mixed methods stu-
dies), only 66 studies were identified as controlled experiments, there
are still considerable steps to be taken in order to strengthen the re-
search field.
Methodological agenda point 2) Future gamification research
should increasingly employ controlled experimental research methods, in
order to gain knowledge on the actual effects of gamification.
Thirdly, a common methodological problem within the gamification
research field has been the study designs, which more often than not do
not control between the various affordances implemented in the studied
systems. Many studies examine the effects or perceptions of gamified
systems containing several different elements, but as a whole. In these
types of research settings, identifying which element actually causes the
observed effects becomes impossible. Furthermore, investigating how
much the different affordances contribute to the results is similarly very
challenging, if none of the elements are controlled for. As reported in
the analyses, only 11 of the 66 controlled experimental studies ex-
amined the effects of individual affordances. Thus, both research design
and the overall research field would benefit from more work which
identifies the effects of different affordances, as more information could
be gained on the kinds of gamification elements that actually work.
However, it must be remembered that even when controlling for the
effects of a certain affordance, the contextual factors as well as in-
dividual user characteristics are likely to affect the results. For example,
positive findings regarding the effectiveness of leaderboards for a class
of schoolchildren does not guarantee that a similar leaderboard design
would produce similar results in a work place setting.
Methodological agenda point 3) Future gamification research
should seek to control for the effects of the individual affordances used in a
given gamification implementation. In addition, when investigating the ef-
fects of the affordances, the contextual characteristics of the setting should
also be taken into account.
Fourthly, as previously mentioned, the empirical research on ga-
mification has been limited in terms of sample sizes, as well as ex-
perimental timeframes. Developments on this front can be seen to have
taken place, when comparing the results of this review with the findings
of the earlier review by Hamari et al. (2014), and the sample sizes of
quantitative experimental studies have increased considerably. Yet,
there are still several studies with only small groups of study partici-
pants. One explanation for these small samples is the nature of the
studies, as many papers report preliminary exploratory research by
testing a prototype or a concept. However, in order to advance the field
in general, research must eventually move beyond isolated works with
prototypes and aim for more theory-informed confirmatory studies.
Furthermore, short timeframes pose an evident threat to the validity of
study findings. Especially, novelty has been shown to have an effect on
users of gamification services (Farzan et al., 2008b; Koivisto & Hamari,
2014), and with short time periods for data gathering, the risk of
findings being skewed by the novelty of the implementation is elevated.
Methodological agenda point 4) In future gamification research, the
sample sizes of studies should be large enough to increase methodological
rigor, as well as to amplify the transferability and explanatory power of the
results. Furthermore, the time spans of studies should be long enough to
enable novelty effects in the data to be minimized.
Fifthly, in many research papers the reporting of the methods, data,
analysis and results is unclear. Part of this problem is potentially caused
by the abundance of conference publications which duly limits the
space available for research papers, and therefore, limits the details that
can be included. While this is naturally not an excuse for poor re-
porting, in some cases it may have contributed to the quality and clarity
of the reports. In any case, an encouragement of precise and thorough
reporting would enable much more efficient diffusion of research re-
sults.
Methodological agenda point 5) In future gamification research,
attention should be paid to clear and comprehensive reporting of research.
4.4. Limitations of the review
In this review, we have followed the suggestions by Paré et al.
(2015) to ensure its quality, in terms of both rigor and relevance. The
review procedure has been described in detail to ensure the clarity of
the process, and to enable replication of the procedure. Furthermore,
the goals of the review have been explicitly stated in order to ensure the
suitability of the chosen methods for the expressed goals.
However, the chosen perspective and methods limit the review in
different ways. The present review focuses on the phenomenon of ga-
mification on an overview level. There is evidently variation, for ex-
ample in how gamification has been defined in the different publica-
tions, or how the various affordances have been defined and
implemented. Due to the goal of comprehensively overviewing a sub-
stantial body of literature, there has been no possibility to go into
further detail of individual studies. In the coding and analysis processes,
some abstraction has obviously been necessary, which has consequently
caused some specifics of the studies to be lost.
Furthermore, the literature search was limited to the Scopus and
AISeL databases. While we are confident of the comprehensiveness of
our literature search, it is nevertheless possible that some publications
have been missed due to either not being among venues indexed in
these databases, or due to indexing errors within the databases (as is the
case with any review study). In any case, the potential number of
missed publications is likely to be meager, and their inclusion would
not foreseeably affect the results of the review to any notable degree.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
207
Declaration of interest
No conflicts of interest exist.
Acknowledgements
This work has been supported by the Business Finland (5479/31/
2017, 40111/14, 40107/14 and 40009/16) and participating partners,
Satakunnan korkeakoulusäätiö and its collaborators, and Academy of
Finland (Center of Excellence - GameCult).
References
Agarwal, R., & Karahanna, E. (2000). Time flies when you’re having fun: Cognitive ab-
sorption and beliefs about information technology usage. MIS Quarterly, 24(4),
665–694.
Ajzen, I. (1988). Attitudes, personality, and behaviour. Chicago, IL: Dorsey Press.
Ajzen, I. (1991). The theory of planned behaviour. Organizational Behavior and Human
Decision Processes, 50(2), 179–211.
Alahäivälä, T., & Oinas-Kukkonen, H. (2016). Understanding persuasion contexts in
health gamification: A systematic analysis of gamified health behavior change sup-
port systems literature. International Journal of Medical Informatics, 96, 62–70.
Alcivar, I., & Abad, A. G. (2016). Design and evaluation of a gamified system for ERP
training. Computers in Human Behavior, 58, 109–118.
Allam, A., Kostova, Z., Nakamoto, K., & Schulz, P. J. (2015). The effect of social support
features and gamification on a web based intervention for rheumatoid arthritis pa-
tients: Randomized controlled trial. Journal of Medical Internet Research, 17(1), e14.
Araújo, J., & Pestana, G. (2017). A framework for social well-being and skills manage-
ment at the workplace. International Journal of Information Management, 37(6),
718–725.
Atkinson, M. A., & Kydd, C. (1997). Individual characteristics associated with world wide
web use: An empirical study of playfulness and motivation. Data Base for Advances in
Information Systems, 28(2), 53–62.
Attali, Y., & Arieli-Attali, M. (2014). Gamification in assessment: Do points affect test
performance? Computers & Education, 83, 57–63.
Bista, S. K., Nepal, S., Paris, C., & Colineau, N. (2014). Gamification for online commu-
nities: A case study for delivering government services. International Journal of
Cooperative Information Systems, 23(2).
Bittner, J. V., & Schipper, J. (2014). Motivational effects and age differences of gamifi-
cation in product advertising. The Journal of Consumer Marketing, 31(5), 391–400.
Blohm, I., & Leimeister, J. M. (2013). Gamification - design of IT-Based enhancing ser-
vices for motivational support and behavioral change. Business & Information Systems
Engineering, 5(4), 275–278.
Bonde, M. T., Makransky, G., Wandall, J., Larsen, M. V., Morsing, M., Jarmer, H., et al.
(2014). Improving biotech education through gamified laboratory simulations.
Nature Biotechnology, 32(7), 694–697.
Buisman, A. L. D., & van Eekelen, M. C. J. D. (2014). Gamification in educational software
development. Proceedings of the Computer Science Education Research Conference (pp.
9–20).
Burke, B. (2014). Gamify: How gamification motivates people to do extraordinary things.
Bibliomotion, Inc.
Butler, C. (2014). A framework for evaluating the effectiveness of gamification techniques by
personality type. International Conference on HCI in Business (Heraklion, Crete, Greece,
June 22-27, 2014)381–389.
Cafazzo, J. A., Casselman, M., Hamming, N., Katzman, D. K., & Palmert, M. R. (2012).
Design of an mHealth app for the self-management of adolescent type 1 diabetes: A
pilot study. Journal of Medical Internet Research, 14(3), e70.
Caillois, R. (1961). Man, play and games. Urbana, IL: University of Illinois Press.
Cechanowicz, J., Gutwin, C., Brownell, B., & Goodfellow, L. (2013). Effects of gamification
on participation and data quality in a real-world market research domain. Proceedings of
Gamification’ 13 Stratford (Ontario, Canada, October 2–4, 2013)58–65.
Chen, Y., & Pu, P. (2014). HealthyTogether: Exploring social incentives for mobile fitness
applications. Proceedings of Chinese CHI’ 14 (Toronto, Ontario, Canada, April 26–27)
25–34.
Chen, C. H., Sun, C. T., & Hsieh, J. (2008). Player guild dynamics and evolution in
massively multiplayer online games. CyberPsychology & Behavior, 11(3), 293–301.
Chen, Y., Zhang, J., & Pu, P. (2014). Exploring social accountability in pervasive fitness apps.
Proceeding of the UBICOMM2014 (Rome, Italy, August 24–28, 2014)221–226.
Choi, J., Choi, H., So, W., Lee, J., & You, J. (2014). A study about designing reward for
gamified crowdsourcing system. Proceedings of 3rd International Conference on Design,
User Experience, and Usability: User Experience Design for Diverse Interaction Platforms
and Environments (pp. 678–687).
Christy, K. R., & Fox, J. (2014). Leaderboards in a virtual classroom: A test of stereotype
threat and social comparison explanations for women’s math performance. Computers
& Education, 78, 66–77.
Cole, H., & Griffiths, M. D. (2007). Social interactions in massively multiplayer online
role-playing gamers. CyberPsychology & Behavior, 10(4), 575–583.
Csíkszentmihályi, M. (1975). Beyond boredom and anxiety: Experiencing flow in work and
play. San Francisco: Jossey-Bass.
Csíkszentmihályi, M. (1990). Flow: The psychology of optimal experience. New York: Harper
and Row.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of
information technology. MIS Quarterly, 13(3), 319–340.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1992). Extrinsic and intrinsic motivation to
use computers in the workplace. Journal of Applied Social Psychology, 22, 1111–1132.
Deci, E. L., & Ryan, R. M. (1985). Self-determination. New Jersey: John Wiley and Sons Inc.
Deci, E. L., & Ryan, R. M. (2000). The ‘what’ and ‘why’ of goal pursuits: Human needs and
the self-determination of behavior. Psychological Inquiry, 11(4), 227–268.
de-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014). An empirical
study comparing gamification and social networking on e-learning. Computers &
Education, 75, 82–91.
Denny, P. (2013). The effect of virtual achievements on student engagement, Proceedings of
CHI 2013: Changing perspectives (Paris, France, April 27–May 2, 2013). 763–772.
Deterding, S. (2015). The lens of intrinsic skill atoms: A method for gameful design.
Human-Computer Interaction, 30(3-4), 294–335.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to
gamefulness: Defining gamification. Proceedings of the 15th International Academic
MindTrek Conference: Envisioning Future media Environments (Tampere, Finland,
September 28–30, 2011)9–15.
Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernández-Sanz, L., Pagés, C., &
Martínez-Herráiz, J.-J. (2013). Gamifying learning experiences: Practical implica-
tions and outcomes. Computers & Education, 63, 380–392.
Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012). Quality through flow and
immersion: Gamifying crowdsourced relevance assessments. Proceedings of the 35th
International ACM SIGIR Conference on Research and Development in InformaTion
Retrieval (Portland, Oregon, USA, August 12–16, 2012)871–880.
Elliot, A. J., & Harackiewicz, J. M. (1994). Goal setting, achievement orientation, and
intrinsic motivation: A mediational analysis. Journal of Personality and Social
Psychology, 66(5), 968–980.
Farzan, R., & Brusilovsky, P. (2011). Encouraging user participation in a course re-
commender system: An impact on user behavior. Computers in Human Behavior, 27(1),
276–284.
Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008a).
When the experiment is over: Deploying an incentive system to all the users. Symposium on
persuasive technology (Aberdeen, Scotland, April, 2008).
Farzan, R., DiMicco, J. M., Millen, D. R., Brownholtz, B., Geyer, W., & Dugan, C. (2008b).
Results from deploying a participation incentive mechanism within the enterprise.
Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in
Computing Systems (Florence, Italy, April 5–10, 2008)563–572.
Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2),
117–140.
Filsecker, M., & Hickey, D. T. (2014). A multilevel analysis of the effects of external
rewards on elementary students’ motivation, engagement and learning in an educa-
tional game. Computers & Education, 75, 136–148.
Fishbein, M. (1979). A theory of reasoned action: Some applications and implications.
Nebraska Symposium on Motivation Nebraska Symposium on Motivation, 27, 65–116.
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to
the theory and research. Reading, MA: Addison-Wesley.
Freund, A. M., Hennecke, M., & Riediger, M. (2010). Age-related differences in outcome
and process goal focus. The European Journal of Developmental Psychology, 7(2),
198–222.
Gartner (2011). Gartner says by 2015, more than 50 percent of organizations that manage
innovation processes will gamify those processes [WWW document]. (Accessed 14th of
February 2017) http://www.gartner.com/it/page.jsp?id=1629214.
Gerow, J. E., Ayyagari, R., Thatcher, J. B., & Roth, P. L. (2013). Can we have fun @ work?
The role of intrinsic motivation for utilitarian systems. European Journal of
Information Systems, 22(3), 360–380.
Hackel, T. S., Jones, M. H., Carbonneau, K. J., & Mueller, C. E. (2016). Re-examining
achievement goal instrumentation: Convergent validity of AGQ and PALS.
Contemporary Educational Psychology, 46, 73–80.
Hakulinen, L., Auvinen, T., & Korhonen, A. (2013). Empirical study on the effect of
achievement badges in TRAKLA2 online learning environment. Proceedings of Learning
and Teaching in Computing and Engineering (LaTiCE) Conference (Macau, March 21–24,
2013)47–54.
Hakulinen, L., Auvinen, T., & Korhonen, A. (2015). The effect of achievement badges on
students’ behavior: An empirical study in a university-level computer science course.
International Journal of Emerging Technologies in Learning (iJET), 10(1), 18–29.
Hall, M., Glanz, S., Caton, S., & Weinhardt, C. (2013). Measuring your Best you: A gami-
fication framework for Well-being measurement. Proceedings of 2013 International
Conference on Cloud and Green Computing (Karlsruhe, Germany, September 30-October
2, 2013)277–282.
Hamari, J., Hassan, L., & Dias, A. (2018). Gamification, quantified-self or social net-
working? Matching users’ goals with motivational technology. User Modelling and
User-Adapted Interaction, 28(1), 35–74.
Hamari, J. (2013). Transforming homo economicus into homo ludens: A field experiment
on gamification in a utilitarian peer-to-peer trading service. Electronic Commerce
Research and Applications, 12(4), 236–245.
Hamari, J. (2017). Do badges increase user activity? A field experiment on effects of
gamification. Computers in Human Behavior, 71, 469–478.
Hamari, J. (2015). Why do people buy virtual goods? Attitude towards virtual good
purchases versus game enjoyment. International Journal of Information Management,
35(3), 299–308.
Hamari, J., Huotari, K., & Tolvanen, J. (2015). Gamification and economics. In S. P. Walz,
& S. Deterding (Eds.). The gameful world: Approaches, issues, applications (pp. 139–
161). Cambridge, MA: MIT Press.
Hamari, J., & Keronen, L. (2017). Why do people play games? A meta-analysis.
International Journal of Information Management, 37(3), 125–141.
Hamari, J., & Koivisto, J. (2015a). Working out for likes”: An empirical study on social
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
208
influence in exercise gamification. Computers in Human Behavior, 50, 333–347.
Hamari, J., & Koivisto, J. (2015b). Why do people use gamification services? International
Journal of Information Management, 35(4), 419–431.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? – A literature review of
empirical studies on gamification. Proceedings of the 47th Hawaii International Conference
on System Sciences (Hawaii, USA, JanUary 6–9, 2014)3025–3034.
Hamari, J., & Tuunanen, J. (2014). Player types: A meta-synthesis. Transactions of the
Digital Games Research Association, 1(2), 29–53.
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A
longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and
academic performance. Computers & Education, 80, 152–161.
Hassan, L., & Hamari, J. (2019). Gamification of e-participation: A literature review.
Proceedings of the 52nd Annual Hawaii International Conference on System Sciences
(HICSS).
Herzig, P., Strahringer, S., & Ameling, M. (2012). Gamification of ERP systems - exploring
gamification effects on user acceptance constructs, in Multikonferenz
Wirtschaftsinformatik 2012: Tagungsband der MKWI 2012. Berlin: Gito Verlag599–612.
Hirschheim, R., & Klein, H. K. (2012). A glorious and not-so-Short history of the in-
formation systems field. Journal of the Association for Information Systems, 13(4)
Article 5.
Hirschman, E. C., & Holbrook, M. B. (1982). Hedonic consumption: Emerging concepts,
methods and propositions. Journal of Marketing, 46, 92–101.
Huang, H.-C., Cheng, T. C. E., Huang, W.-F., & Teng, C.-I. (2018). Impact of online ga-
mers’ personality traits on interdependence, network convergence, and continuance
intention: Perspective of social exchange theory. International Journal of Information
Management, 38(1), 232–242.
Huizinga, J. (1955). Homo ludens: A study of the play element in culture. Boston, MA:
Beacon Press.
Huotari, K., & Hamari, J. (2012). Defining gamification – A service marketing perspective.
Proceedings of the 16th International Academic MindTrek Conference (Tampere, Finland,
3–5 October, 2012)17–22.
Huotari, K., & Hamari, J. (2017). A definition for gamification: Anchoring gamification in
the service marketing literature. Electronic Markets, 27(1), 21–31.
IEEE (2014). Everyone’s a gamer – IEEE experts predict gaming will be integrated into more
than 85 percent of daily tasks by 2020 [WWW document]. (Accessed 14th of February
2017) http://www.ieee.org/about/news/2014/25_feb_2014.html.
Ipeirotis, P. G., & Gabrilovich, E. (2014). Quizz: Targeted crowdsourcing with a billion
(potential) users. Proceedings of WWW’ 14 (Seoul, Korea, April 7–11, 2014)143–154.
Jones, B. A., Madden, G. J., & Wengreen, H. J. (2014). The FIT game: Preliminary eva-
luation of a gamification approach to increasing fruit and vegetable consumption in
school. Preventive Medicine, 68, 76–79.
Kallio, K. P., Mäyrä, F., & Kaipainen, K. (2010). At least nine ways to play: Approaching
gamer mentalities. Games and Culture 1555412010391089.
Koivisto, J., & Hamari, J. (2014). Demographic differences in perceived benefits from
gamification. Computers in Human Behavior, 35, 179–188.
Landers, R. N. (2015). Developing a theory of gamified learning: Linking serious games
and gamification of learning. Simulation & Gaming, 45, 752–768.
Landers, R. N., & Armstrong, M. B. (2015). Enhancing instructional outcomes with ga-
mification: An empirical test of the Technology-Enhanced Training Effectiveness
Model. Computers in Human Behavior.
Landers, R. N., Bauer, K. N., & Callan, R. C. (2015). Gamification of task performance with
leaderboards: A goal setting experiment. Computers in Human Behavior.
Landers, R. N., & Landers, A. K. (2015). An empirical test of the theory of gamified
learning: The effect of leaderboards on time-on-task and academic performance.
Simulation & Gaming, 45(6), 769–785.
Laureyssens, T., Coenen, T., Claeys, L., Mechant, P., Criel, J., & Vande Moere, A. (2014).
ZWERM: A modular component network approach for an urban participation game.
Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems
- CHI’ 14 (Toronto, ON, Canada, April 26-May 1, 2014)3259–3268.
Lee, J. J., Ceyhan, P., Jordan-Cooley, W., & Sung, W. (2013). GREENIFY: A real world
action game for climate change education. Simulation & Gaming, 44(2–3), 349–365.
Liu, D., Santhanam, R., & Webster, J. (2017). Towards meaningful engagement: A fra-
mework for design and research of gamified information systems. MIS Quarterly,
41(4), 1011–1034.
Liu, D., Xun, L., & Santhanam, R. (2013). Digital games and beyond? What happens when
player compete? MIS Quarterly, 37(1), 111–124.
Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting
and task motivation: A 35-year odyssey. The American Psychologist, 57(9), 705–717.
Long, Y., & Aleven, V. (2014). Gamification of joint student/system control over problem
selection in a linear equation tutor. Proceedings of the 12th International Conference on
Intelligent Tutoring Systems (pp. 378–387).
Lounis, S., Pramatari, K., & Theotokis, A. (2014). Gamification is all about fun: The role of
incentive type and community collaboration. Proceedings of ECIS 2014 (Tel Aviv, Israel,
June 9–11, 2014)1–14.
Lunenburg, F. C. (2011). Goal-setting theory of motivation. International Journal of
Management, Business, and Administration, 15(1).
Majuri, J., Koivisto, J., & Hamari, J. (2018). Gamification of education and learning: A
review of empirical literature. Proceedings of the 2nd International GamiFIN conference
(pp. 11–19).
Malaby, T. M. (2007). Beyond play: A new approach to games. Games and Culture, 2(2),
95–113.
Mann, T., De Ridder, D., & Fujita, K. (2013). Self-regulation of health behavior: Social
psychological approaches to goal setting and goal striving. Health Psychology, 32(5),
487–498.
Mekler, E. D., Brühlmann, F., Opwis, K., & Tuch, A. N. (2013). Do points, levels and
leaderboards harm intrinsic motivation? Proceedings of the First International
Conference on Gameful Design, Research, and Applications - Gamification ’13 (pp.
66–73).
Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2015). Towards understanding the
effects of individual gamification elements on intrinsic motivation and performance.
Computers in Human Behavior.
Moon, J.-W., & Kim, Y.-G. (2001). Extending the TAM for a world-wide-Web context.
Information & Management, 38(4), 217–230.
Morschheuser, B., Hamari, J., & Koivisto, J. (2016). Gamification in crowdsourcing: A re-
view. Proceedings of the 49th Annual Hawaii International Conference on System Sciences
(HICSS) (Hawaii, USA, January 5-8, 2016)4375–4384.
Morschheuser, B., Hamari, J., & Maedche, A. (2018). Cooperation or competition–When
do people contribute more? A field experiment on gamification of crowdsourcing.
International Journal of Human-Computer Studies.
Morschheuser, B., Hassan, L., Werder, K., & Hamari, J. (2018). How to design gamifi-
cation? A method for engineering gamified software. Information & Software
Technology, 95, 219–237.
Morschheuser, B., Hamari, J., Koivisto, J., & Maedche, A. (2017). Gamified crowdsour-
cing: Conceptualization, literature review, and future agenda. International Journal of
Human-computer Studies, 106, 26–43.
Mäntymäki, M., Merikivi, J., Verhagen, T., Feldberg, F., & Rajala, R. (2014). Does a
contextualized theory of planned behavior explain why teenagers stay in virtual
worlds? International Journal of Information Management, 34(5), 567–576.
Mäntymäki, M., & Riemer, K. (2014). Digital natives in social virtual worlds: A multi-
method study of gratifications and social influences in Habbo Hotel. International
Journal of Information Management, 34(2), 210–220.
Mäntymäki, M., & Salo, J. (2013). Purchasing behavior in social virtual worlds: An ex-
amination of Habbo Hotel. International Journal of Information Management, 33(2),
282–290.
Mäntymäki, M., & Salo, J. (2015). Why do teens spend real money in virtual worlds? A
consumption values and developmental psychology perspective on virtual con-
sumption. International Journal of Information Management, 35(1), 124–134.
Mäyrä, F. (2008). Introduction to game studies: Games in culture. London & NY: Sage
Publications.
Panigrahi, R., Srivastava, P. R., & Sharma, D. (2018). Online learning: Adoption, con-
tinuance, and learning outcome—A review of literature. International Journal of
Information Management, 43, 1–14.
Paré, G., Trudel, M. C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems
knowledge: A typology of literature reviews. Information & Management, 52(2),
183–199.
Reitz, L., Sohny, A., & Lochmann, G. (2016). VR-based gamification of communication
training and oral examination in a second language. International Journal of Game-
Based Learning, 6(2), 46–61.
Rigby, S., & Ryan, R. M. (2011). Glued to games: How video games draw us in and hold us
spellbound. Santa Barbara, CA: Praeger.
Riva, S., Camerini, A. L., Allam, A., & Schulz, P. J. (2014). Interactive sections of an
internet-based intervention increase empowerment of chronic back pain patients:
Randomized controlled trial. Journal of Medical Internet Research, 16(8).
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of in-
trinsic motivation, social development, and well-being. The American Psychologist,
55(1), 68–78.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A
self-determination theory approach. Motivation and Emotion, 30(4), 344–360.
Sanchez-Nielsen, E., & Lee, D. (2013). eParticipation in practice in Europe: The case of
"puzzled by policy: Helping you Be part of EU". Proceedings of the 46th Annual Hawaii
International Conference on System Sciences (HICSS) (Hawaii, USA, January 7-10,
20131870–1879.
Santhanam, R., Liu, D., & Shen, W. C. M. (2016). Research note - gamification of tech-
nology-mediated training: Not all competitions are the same. Information Systems
Research, 27(2), 453–465.
Scharkow, M., Festl, R., Vogelgesang, J., & Quandt, T. (2015). Beyond the “core-gamer”:
Genre preferences and gratifications in computer games. Computers in Human
Behavior, 44, 293–298.
Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International
Journal of Human-computer Studies, 74, 14–31.
Seffah, A., & Taleb, M. (2012). Tracing the evolution of HCI patterns as an interaction
design tool. Innovations in Systems and Software Engineering, 8(2), 93–109.
Simões, J., Díaz Redondo, R., & Fernández Vilas, A. (2013). A social gamification fra-
mework for a K-6 learning platform. Computers in Human Behavior, 29(2), 345–353.
Sun, H., & Zhang, P. (2006). Causal relationships between perceived enjoyment and
perceived ease of use: An alternative approach. Journal of the Association for
Information Systems, 7(9), 618–645.
Sørensen, J. J. W., Pedersen, M. K., Munch, M., Haikka, P., Jensen, J. H., Planke, T.,
Andreasen, M. G., Gajdacz, M., Mølmer, K., Lieberoth, A., & Sherson, J. F. (2016).
Exploring the quantum speed limit with computer games. Nature, 532, 210–213.
Talaei-Khoei, A., & Daniel, J. (2018). How younger elderly realize usefulness of cognitive
training video games to maintain their independent living. International Journal of
Information Management, 42, 1–12.
Tavinor, G. (2009). The art of videogames. Malden, MA: Wiley-Blackwell.
Teng, C.-I. (2017). Impact of avatar identification on online gamer loyalty: Perspectives of
social identity and social capital theories. International Journal of Information
Management, 37(6), 601–610.
Teng, C. I., & Chen, W. W. (2014). Team participation and online gamer loyalty. Electronic
Commerce Research and Applications, 13(1), 24–31.
Terlutter, R., & Capella, M. L. (2013). The gamification of advertising: Analysis and re-
search directions of in-game advertising, advergames, and advertising in social net-
work games. Journal of Advertising, 42(2-3), 95–112.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
209
Thom, J., Millen, D., & DiMicco, J. (2012). Removing gamification from an enterprise SNS, in
proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
(Seattle, Washington, USA, February 11–15, 2012). 1067–1070.
Tolmie, P., Chamberlain, A., & Benford, S. (2014). Designing for reportability: Sustainable
gamification, public engagement, and promoting environmental debate. Personal and
Ubiquitous Computing, 18(7), 1763–1774.
Tuomela, R. (1995). The importance of us: A philosophy study of basic social notions.
Stanford, CA: Stanford University Press.
van der Heijden, H. (2004). User acceptance of hedonic information systems. MIS
Quarterly, 28(4), 695–704.
Venkatesh, V. (1999). Creation of favorable user perceptions: Exploring the role of in-
trinsic motivation. MIS Quarterly, 23(2), 239–260.
Vesa, M., Hamari, J., Harviainen, J. T., & Warmelink, H. (2017). Computer games and
organization studies. Organization Studies, 38(2), 273–284.
Wang, X., Schneider, C., & Valacich, J. S. (2015). Enhancing creativity in group colla-
boration: How performance targets and feedback shape perceptions and idea gen-
eration performance. Computers in Human Behavior, 42, 187–195.
Warmelink, H., Koivisto, J., Mayer, I., Vesa, M., & Hamari, J. (2018). Gamification of
production and logistics operations: Status quo and future directions. Journal of
Business Research. https://www.sciencedirect.com/science/article/pii/
S0148296318304594.
Watson, D., Mandryk, R. L., & Stanley, K. G. (2013). The design and evaluation of a
classroom exergame. Proceedings of Gamification’ 13 (Stratford, ON, Canada, October
02–04, 2013)34–41.
Webster, J., & Martocchio, J. J. (1992). Microcomputer playfulness: Development of a
measure with workplace implications. MIS Quarterly, 16(2), 201–226.
Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing
a literature review. MIS Quarterly, 26(2), xiii–xxiii.
Williams, D., Consalvo, M., Caplan, S., & Yee, N. (2009). Looking for gender: Gender roles
and behaviors among online gamers. The Journal of Communication, 59(4), 700–725.
Williams, D., Yee, N., & Caplan, S. E. (2008). Who plays, how much, and why? Debunking
the stereotypical gamer profile. Journal of Computer-Mediated Communication, 13(4),
993–1018.
Wu, J., & Lu, X. (2013). Effects of extrinsic and intrinsic motivators on using utilitarian,
hedonic, and dual-purposed information systems: A meta-analysis. Journal of the
Association for Information Systems, 14(3), 153–191.
Xi, N., & Hamari, J. (2019). The relationship between gamification, brand engagement
and brand equity. Proceedings of the 52nd Annual Hawaii International Conference on
System Sciences (HICSS).
Yates, M., Kelemen, A., & Sik Lanyi, C. (2016). Virtual reality gaming in the rehabilitation
of the upper extremities post-stroke. Brain Injury, 30(7), 855–863.
Yee, N. (2006). Motivations for play in online games. CyberPsychology & Behavior, 9(6),
772–775.
Zagal, J. P., Mateas, M., Fernandez-Vara, C., Hochhalter, B., & Lichti, N. (2005). Towards
an ontological language for game analysis. Proceedings of International DiGRA
Conference: Changing Views - Worlds in Play (Vancouver, Canada, June 16–20, 2005)
3e14.
J. Koivisto, J. Hamari International Journal of Information Management 45 (2019) 191–210
210