Emerging Trends in The Social and Behavioral Sciences

Virtual Worlds as Laboratories

Media

Part of Virtual Worlds as Laboratories

Title

Virtual Worlds as Laboratories

extracted text

Virtual Worlds as Laboratories
TRAVIS L. ROSS, EDWARD CASTRONOVA, and ISAAC KNOWLES

Abstract
A virtual world is a persistent space where tens, hundreds, thousands, or even millions of users interact with each other and a mediated environment defined as a
physical space through rules created by designers and enforced by computer code.
Researchers have argued that these characteristics make virtual worlds particularly
well suited for conducting parallel experiments to test macro-level social theory. The
purpose of this essay is to provide an introduction into virtual worlds research. It is
not an exhaustive resource chronicling the history of virtual worlds, but rather an
introduction broken into three sections for those wishing to learn more about the
past, present, and future directions of the topic. First, it explores what researchers
have said about using virtual worlds research and the fields of research where virtual
worlds have been used. In doing so, it focuses on research in video games studies and
complex systems. Second, it examines cutting-edge work in virtual worlds research,
identifying that both academia and the game industry will play a significant role in
the success and direction. Third, it identifies six key issues that scholars using virtual
worlds research will face as they move forward.

INTRODUCTION
Over the last decade, social scientists have expressed interest in using virtual
worlds to study the societies of the real world. The argument follows that
virtual worlds can track user behavior in a database, that virtual worlds can
be experimentally manipulated and run in parallel for controlled experimentation, and that virtual worlds are inhabited by societies of users. These societies are of interest to researchers because they feature project-based teams,
corporations, governments, and economies. In other words, virtual worlds
have the same complex social arrangements that you find in the real world,
and they afford researchers powerful tools for conducting research.
But what is a virtual world? Bartle (2004), who constructed the first virtual
world with Roy Trubshaw, defines one as a persistent online space with physical rules that is shared by many players who can interact with the world and
each other. When he adopted this definition, he was comparing massive multiplayer online games such as Everquest, Ultima Online, Linage, and Multiuser
Emerging Trends in the Social and Behavioral Sciences. Edited by Robert Scott and Stephen Kosslyn.
© 2015 John Wiley & Sons, Inc. ISBN 978-1-118-90077-2.

1

2

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Dungeon to local multiplayer games such as Half-life, Quake, and Mario Kart.
At that time, it was easy to segment environments that were virtual worlds
from those that were not. There were some hybrids, such as Diablo 2, but they
numbered few.
Ask for an example of a virtual world now and the answer might be World of
Warcraft, EVE Online, Second Life, Minecraft, Modern Warfare, Parallel Kingdoms,
or League of Legends. The point being, there are now many online spaces that
are clearly virtual worlds (World of Warcraft, EVE Online, or Second Life) and
there are also many online spaces that straddle the boundary (Modern Warfare, Diablo 3, Minecraft, or League of Legends). Virtual worlds are now more
difficult to delineate from nonvirtual worlds. They contain elements that are
both persistent and nonpersistent, have a few players upward to millions of
players, and have rules that bridge the virtual and the real.
In order to focus on the virtual worlds that have interested social scientists
over the last few decades, it is important to have a working definition that
limits scope. Environments that fall out of this definition are still interesting and share properties with virtual worlds, some more so than others, but
when scholars discuss virtual world experiments, they always possess the
following four characteristics:
Virtual Worlds are Persistent and Multiplayer. They are inhabited by a group
of users and information about those users continues to exist in a
database when those users are not present.
Virtual Worlds Incorporate Space. Whether one-, two-, or three-dimensional
users of virtual worlds interact in a digital representation of a physical
space. Fantasy football does not have a physical space. Second Life does.
Virtual Worlds are Designed. A user can interact with a virtual world
through an interface that is defined by rules created by a designer or
engineer.
The Rules of a Virtual World are Enforced by Computer Code. What separates a
virtual world from a sports league, the television show Survivor, or The
Stanford Prison Experiment is that they instantiated and enforced by a
computer.
Putting all three together into one definition: A virtual world is a persistent
space where tens, hundreds, thousands, or even millions of users interact
with each other and a mediated environment defined as a physical space
through rules created by designers and enforced by computer code.
Each of these characteristics makes a virtual world a valuable tool for studying human society. A persistent world uses a database. The behaviors of
every user in the world can be tracked and stored. Using the appropriate tools

Virtual Worlds as Laboratories

3

for analysis and visualization, a researcher can ask questions about individuals, groups segmented by variables of interest, or the entire population. By
designing the rules and the physical space, a research can craft an environment that is balanced between abstraction and replication of the real world
for the specific question of interest. Finally, computer code allows a virtual
world to be tinkered with and copied. The experimental method requires
running at least two identical environments in parallel while manipulating
only the variable of interest. Virtual worlds are suited for experiments and
scholars have compared them to Petri dishes or neighboring island nations
(Bradley & Froomkin, 2004; Castronova, 2006). In these examples, each world
is constructed with the same rules except for a change in a variable of interest.
It is then populated with a random sample of users and the society is tracked
over time. If behavior in virtual worlds truly parallels behavior in the real
world, then they present a powerful way to study society. It is no wonder that
virtual world research has generated interest in fields such as public health,
economics, communication, complex systems, law, and public policy.
Like the ghosts of A Christmas Carol, this essay explores the past, present,
and future of virtual worlds research. The section titled “Foundational
Research” introduces the past and explores the foundational research for
experimentation with virtual worlds. It shows that researchers working
in complex systems, simulation, and agent-based modeling were actually
the first to consider the use of virtual worlds for social experimentation.
However, it was the video game industry and video game scholars made
it possible to conduct experiments with real human beings. The section
titled “Cutting-Edge Work” focuses on the present. It explores cutting-edge
research in both academia and the game industry, and it introduces studies
and tools that will shape the future of virtual worlds research. The section
titled “Key Issues” focuses on the key issues moving forward and identifies
six issues that researchers crafting virtual worlds for social science research
will have to address moving forward.
FOUNDATIONAL RESEARCH
In the early 2000s (decade), researchers studying games and media identified
that virtual worlds offered a unique opportunity for conducting controlled
experiments on human societies and social groups. They recognized that virtual worlds were designed and instantiated through computer code and that
a scientist could manipulate the code to study social behavior in a series of
parallel experiments. This was the first time that virtual worlds from the
entertainment industry were identified as tools for social scientific research,
but it was not the first time that virtual worlds had been used to study social
behavior. It is the field of complex systems holds that distinction.

4

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Complex systems are nonlinear systems composed of simpler interworking
parts. Ecosystems, beehives, brains, and governments are all complex systems. Each is a collection of individual organisms, bees, neurons, or people
interacting to produce emergent behavior. The origins of complex systems
research can be traced back a few hundred years to Darwin and Adam Smith,
whose theories demonstrate that complexity can arise from a few simple
rules. Much of this early work relied on a top-down approach and used the
scientific paradigm of reductionism to examine the individual components
of the system. This approach, however, was met with limited success because
it is difficult to understand the behavior of a beehive by studying the behavior of a single bee. A bottom-up approach was required, but it was not until
the creation of the modern computer that scientists were able to build virtual
worlds, populate them with autonomous agents, and test hypothesis about
the system as a whole, a technique known as agent-based modeling.
Perhaps the best-known early work in the agent-based modeling comes
from Conway (1970) whose Game of Life spawned a variety of “life-like”
agents in a virtual world with a few simple behavioral rules. Other early
work includes Schelling’s agent-based models of segregation, which show
how a simple rule regarding tolerance to diversity can lead to highly
segregated communities, and Axelrod’s (1987) famous evolutionary tournament designed to test the evolutionary success of cooperative and selfish
agents playing a repeated prisoners’ dilemma. From this groundbreaking
work, an entire field has emerged. It uses virtual worlds to study complex
systems such as food chains, traffic patterns, flocking, local economies, and
information transmission.
Although scholars studying virtual worlds for entertainment rarely mention it, research in the field of complex systems and agent-based modeling
is important, and the two fields are likely to intersect. As Section titled
“Key Issues” of this essay will show, researchers in complex systems have
identified and are currently tackling some of the most important obstacles
for the study of human society using virtual worlds. That said, there is one
important distinction between the two fields. Virtual worlds research proposes using real human societies in virtual worlds to study social behavior,
while agent-based modeling uses computerized agents driven by simple
rules. Agent-based models have their advantages. They can run iterations
of a complex system and examine the behavior of agents across millions of
rounds or generations in a manner of seconds. They also have a weakness.
The predictions of the model are based on simple agents, not real humans.
The foundational research for using virtual worlds with real human beings
as laboratories for social scientific research is largely theoretical and philosophical in nature. Massive multiplayer online virtual worlds have existed
since the late 1970s, but the potential of virtual worlds for social research did

Virtual Worlds as Laboratories

5

not become apparent until Castronova (2001, 2003) demonstrated, through
the use of econometrics, that virtual economies were similar and connected
to the real economy. Afterward, Bradley and Froomkin (2004) picked up
on this work and were the first to identify that virtual worlds, with their
real economies, were well suited to operate as a test bed for law and policy,
especially when the law or policy was too risky or intractable for experimentation in the real world. In 2006, Castronova (2006) performed a case
study on two natural experiments that tested game theoretic coordination
problems in the online role-playing games Dark Age of Camelot and Everquest.
Since then, Castronova’s research has stressed the importance of developing
virtual worlds as a scientific apparatus for conducting experiments on
societies. He has stated many times that policymakers should invest in
virtual worlds as they do in supercolliders, satellites, or other high-tech
research tools (Castronova & Falk, 2009; Castronova, Bell, et al., 2008; 2009;
Ross, Castronova, & Wagner, 2012).
One of the more famous instances where researchers recognized the power
of virtual world research happened by accident. It was triggered by a glitch
known as the Corrupted Blood incident, which acted like a real virus, spreading
through the game World of Warcraft killing thousands of players. Citing the
glitch as an example, Balicer (2007) proposed that data from virtual worlds
could be used to model the spread of infectious disease, and shortly after,
Lofgren and Fefferman (2007) demonstrated that the Corrupted Blood incident shared many of the characteristics of real-world epidemics. Later that
year, the potential of virtual worlds reached a wider scientific community
when Bainbridge (Bainbridge, 2007) published an article in the journal Science detailing the work of scholars thus far and championing virtual worlds
as the future of social science experimentation.
Most of the foundational research regarding the use of synthetic worlds to
study social science has been philosophical in nature; however, there were a
few attempts in the early 2000s (decade) to build virtual worlds for researcher
purposes. In 2005, Castronova led a team of researchers and developers on a
project named Arden, a virtual world based on the works of William Shakespeare. According to Castronova et al. (2008), Arden had two goals, to educate
players by immersing them in the work of William Shakespeare and to provide a platform for cross-server economic experimentation. By late 2006, it
was apparent that Arden would not meet its ambitious long-term goals, so
the project was scaled back and Castronova acknowledge that the failure of
Arden might serve as a warning of the difficulties of virtual world research
(Baker, 2008; Castronova, Cummings, et al., 2009).
In 2008, following his work on Arden, Castronova, Ross, and a team of
researchers began work on a web-based virtual world named Greenland
(Ross, 2009). Once again the goal was macroeconomic experimentation, this

6

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

time specifically focused on the emergence of currency. Construction of the
world was completed in January 2009, and a pretest was conducted with 700
participants and 5 servers. Although the game maintained an active player
base for 2 months, the targeted economic theories were never tested. Once
again the experiment was considered unsuccessful as unexpected expansion
and warfare in the game disrupted economic activity (Ross, 2009; Ross &
Cornell, 2010).
The most successful examples of research involving online societies can be
found in the field of complex systems. In 2006, Salganik, Dodds, and Watts
(2006) constructed parallel instances of a music sharing website where they
tested social learning theory by comparing the influence of song popularity
and song rating on the number of downloads. Although this experiment did
not use a virtual world as defined here, it was groundbreaking because it
provided a successful example of how to test a hypothesis about macro-level
social behavior in parallel online experiments.
Due to the difficulties of virtual world experiments, some researchers have
turned to using the large datasets of existing virtual worlds as a basis for
comparing real-world behavior to virtual behavior. In 2006, Ducheneaut,
Yee, Nickell, and Moore (2006) collected information about the players of
World of Warcraft using autonomous bots and examined how the design of
World of Warcraft motivated the behavior of players and social groups. In
2009, another group of researchers started The Virtual World Observatory,
which is a multiuniversity research group created to study social behavior
using large datasets from popular virtual worlds. The group has published
a considerable amount of research, with much of it based on findings
generated using a four-terabyte dataset from the virtual world Everquest
(Virtual Worlds Observatory, n.d.). In addition to performing research, the
group also tackled some of the major organizational and access problems
that researchers must face when attempting to store and analyze large
amounts of data from virtual worlds (Williams, 2010a).
CUTTING-EDGE WORK
Over the past 5 years, some of the momentum behind using industry grade
virtual worlds as experimental environments has abated, as a number of
projects have demonstrated that building games and virtual worlds for scientific experiments is a considerable challenge. Yet, the momentum continues in
two directions. First, the video game industry is interested in the using social
science theory, experimentation, and analytics for understanding, predicting,
and shaping the behavior of players in online games. Second, academics have
scaled back virtual worlds research. Instead of trying to build industry grade

Virtual Worlds as Laboratories

7

worlds populated by thousands of people, they are now using simpler virtual
worlds to study the dynamics of small groups.
The most powerful tools for testing macro-level social theory now exist in
the game industry. Due to this phenomenon, Castronova, Ross, and Knowels
(2013) argue that over the coming decades the major advances in social
science will be made by game designers tinkering with online societies and
not social scientists in universities. The move toward advanced tools for
behavior tracking, analytics, and visualization in the game industry has been
swift. Over the past 5 years, companies such as Zynga, Bioware, Electronic
Arts, Activision/Blizzard, Riot Games, Valve Software, and CCP Games
have gone from having almost no capabilities for analytics to possessing
the most advanced tools for studying societies ever created. The Game
Developers Conference, the premier forum for discussing the development
of games, now regularly features discussions about the best practices for
studying, creating, and maintaining economies and social institutions in virtual worlds. Many social scientists also been pulled into this work, as Valve
Software, CCP Games, Disney/Playdom, and Riot Games have all recently
hired social scientists to conduct research and design online societies.
While the game industry has scaled up the practice of virtual worlds
research, researchers in academia have scaled back. Currently, the most
successful examples in academia come from researchers using virtual worlds
to study small groups on a short-time scale. For example, Dabbish, Kraut,
and Patton (2012) recently installed World of Warcraft on a private server
and used it to study behavior in groups of 5–25 players. Team members
of The Augur Project at the Carnegie Mellon ETC have used Amazon’s
Mechanical Turk in combination with three of game-like prototypes to study
and build predicative models of player behavior (Augur, 2013). Wisdom and
Goldstone (2010) recently created a small-scale asynchronous virtual world
based around a puzzle game. It allowed players to see information about
what past players had done and was used to test social learning theory. In
complex systems, Mason and Watts (2012) conducted a series of experiments
with simple web-based games to study collaborative problem solving in
groups, and Centola (2010) examined how social networks influence the
exchange of health information by experimenting with parallel interactive
websites. These examples do not exhaust the current research, but they do
serve to demonstrate the direction of virtual worlds research in academia.
Finally, researchers continue to work with large-scale datasets. Szell,
Lambiotte, and Thurner (2010) examined the structure of social networks in
the game Pardus, a virtual world claiming over 100,000 active users. While
Williams and others have begun to explore the link between behavior and
motivation, using surveys linked to behavioral data in virtual worlds such
as League of Legends and Everquest 2 (Virtual Worlds Observatory, n.d.),

8

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Williams’s work has also moved him into the game industry. He recently
stared the company Ninja Metrics to provide social network analysis and
machine learning to game companies (Ninja Metrics, n.d.).
KEY ISSUES
Virtual worlds present an exciting opportunity for researchers looking to
study social systems, but the research will require overcoming some key
issues. This section identifies the main issues facing researchers studying
social systems in virtual worlds. They are the mapping principle, big data,
the attention economy, player synchronization, connecting to mental states, and
complex and chaotic systems.
THE MAPPING PRINCIPLE
Williams (2010b) coined “the mapping principle” as a problem for
researchers wanting to study virtual worlds. In his article, he argues
that using virtual worlds for research is a good idea, but that researchers
wishing to pursue it must understand the similarities and differences
between real and virtual behavior. Virtual worlds are abstractions of the real
world, and thus behavior does not always match one to one. The researchers
who aimed to study the spread of epidemics using the Corrupted Blood
incident in World of Warcraft encountered the mapping principle first hand.
They witnessed infected individuals trying to infect others because it was
funny or dying was of little consequence.
The mapping principle is not entirely a new problem for experimentalists and follows closely to the problem of maintaining ecological or external
validity. When experimental researchers create controlled environments for
the study of real-world decision-making, they make choices about abstraction and how costs and incentives match those of the real world. Maintaining
external validity can be difficult in even simple experiments. The complexity of virtual worlds and the game design decisions that go into them make
constructing virtual worlds for targeted experimentation even more of a challenge (Ross & Cornell, 2010).
BIG DATA
Another problem that researchers studying virtual worlds must face is
storing and analyzing the large, and growing, behavioral datasets of virtual
worlds. Storage space is becoming less expensive and storing terabytes of
data is no longer a major concern. What is of concern for researchers are
the methods for accessing, analyzing, visualizing, and interpreting this

Virtual Worlds as Laboratories

9

data. As the social sciences continue to harness the powers of big data and
computation, it is becoming apparent that scholars in the social sciences and
the humanities must work more closely with researchers, such as computer
scientists, familiar with the techniques and methods required to harness
big data.
ATTENTION ECONOMY
Human beings have limited attention. In our mediated world, attention is
a valuable commodity and there are millions of experiences competing for
this valuable resource, creating what has been termed an attention economy
(Simon, 1996). Researchers who wish to use virtual worlds for studying social
systems will have to compete for attention. Currently, if a scientist needs a
few hundred participants to come into the lab for an hour, they attract them
with extra credit or money. Depending on the question, a scientist using a
virtual world may need more than a few hundred participants and may also
need them for a long period of time. A number of research projects have used
Amazon’s Mechanical Turk to attract a few thousand participants and independent research has validated the results of participants using the service
(Mason & Suri, 2010). Mechanical Turk may present one low-cost method for
attracting players, but researchers wishing to attract players without paying
them must figure out how to create engaging experiences that can attract the
attention of users and maintain ecological validity.
PLAYER SYNCHRONIZATION
The most interesting questions that can be asked using virtual worlds involve
societies interacting over time. This presents a coordination problem for any
researcher asking these questions using a virtual world because players must
all be present in the world at the same time in order to interact with each
other. To make matters worse, many virtual worlds suffer from high attrition
rates and players dropping out or not playing at the same time can create
validity problems for researchers.
One solution is to let players interact asynchronously. Changing the progression of time from continuous to larger discrete units (turns) allows players to enter the virtual world and make strategic decisions at any time during
the turn. There are many successful virtual worlds that use asynchronous
play, but one concern is that it slows the pace of the game and decreases the
number and type of strategic interactions. When a player finishes their turn,
they must wait for all of the other players in the game to react, or for the turn
timer to expire. Some research questions may not be approachable through

10

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

asynchronous play; therefore, researchers must explore the limitations created by the difficulties of synchronization.
CONNECTING BEHAVIOR TO MENTAL STATES
Virtual worlds are good at answering questions about behavior because they
can track it at a very fine time scale. What they are not good at is connecting behavior to the beliefs and desires. Researchers using virtual worlds and
machine learning can make very accurate predictions about the behavior of
individuals, but are challenged when explaining why an individual makes a
certain decision. To overcome this hurdle, researchers have linked self-report
data to behavioral data in order to explore the motivations of players, and
game developers have discussed collecting biometric feedback through controllers (Caplan, Williams, & Yee, 2009; Sottek & Warren, 2013; Williams, Consalvo, Caplan, & Yee, 2009; Williams, Yee, & Caplan, 2008). Still, the connection between behavior and beliefs and desires is limited in virtual worlds and
a problem that researchers must continue to address.
COMPLEX AND CHAOTIC SYSTEMS
Virtual worlds are complex, they have interworking parts that lead to emergent behavior, and are also chaotic, living on the border of order and randomness. In complex and chaotic systems, small decisions early on can lead
to big differences further in time (Miller & Page, 2007). For example, many are
familiar with the metaphor that describes how flapping butterfly wings may
lead to the formation of a hurricane (Lorenz, 1993). It is an extreme example,
but it depicts chaotic systems quite well. As time progresses, societies can
have multiple branches. Small fluctuations can equal big changes. What if
Gore had won the election instead of Bush? If we run the simulation 100 times
with different people in the key roles, would the outcome remain the same
every time? Probably not. Scientists who are studying virtual worlds must
prepare for uncertainty due to early decisions and differences in populations.
They must consider the choices and rules that they design into the game and
understand the motivations of players, and they must run multiple occurrences of the same exact world in parallel. Only then can they understand
when and why a society takes a different path.
CONCLUSION
Over the last decade, game developers have attempted to build hundreds of
virtual worlds and many of them have failed. In order to navigate the difficult of building virtual worlds, the mapping problem, external validity, and

Virtual Worlds as Laboratories

11

the attention economy designers have to carefully combine theory and game
design. Unanticipated design decisions or the decisions of a few individuals
can lead to very different social outcomes in virtual worlds. This means that
researchers who wish to gain the most from virtual worlds must run multiple instances in parallel to establish consistent outcomes and publish the
design, method, and results so that other scientists can replicate the experiment and compare their own results. How will researchers publish the design
and programing decisions made when constructing virtual worlds? In addition, there is little theory available to those who are building simulations of
reality. The best theories of game design, even in serious games, are institutionalized and internalized within a few talented individuals. In order to
realize the potential of virtual world research, there must be a shared knowledgebase of the mapping principle and difficulties encountered during along
the way. Virtual worlds present a powerful opportunity for social scientists,
but the construction and use of these systems may rival in scale and complexity the challenges of building satellites for space exploration or super
colliders for physics.
REFERENCES
Axelrod, R. (1987). The evolution of strategies in the iterated prisoner’s dilemma.
Genetic Algorithms and Simulated Annealing, 3(1), 32–41.
Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science,
317(5837), 472–476. doi:10.1126/science.1146930
Baker, C. (2008 March 24). Trying to design a truly entertaining game can defeat even
a certified genius. WIRED Magazine. Retrieved January 8, 2013, from http://www.
wired.com/gaming/gamingreviews/magazine/16-04/pl_games.
Balicer, R. D. (2007). Modeling infectious diseases dissemination through online
role-playing games. Epidemiology, 18(2), 260–261.
Bartle, R. A. (2004). Designing virtual worlds. Indianapolis: New Riders.
Bradley, C., & Froomkin, M. (2004). Virual worlds, real rules. New York Law School
Law Review, 49(1), 103–146.
Caplan, S., Williams, D., & Yee, N. (2009). Problematic Internet use and psychosocial
well-being among MMO players. Computers in Human Behavior, 25, 1312–1319.
Castronova, E. (2001). Virtual worlds: A first-hand account of market and society
on the cyberian frontier. The Gruter Institute Working Papers on Law, Economics, and
Evolutionary Biology, 2, 1–68.
Castronova, E. (2003). The price of “man” and “woman”: A hedonic pricing model
of avatar attributes in a synthethic world. CESifo Working Paper Series, No. 957.
Retrieved from http://ssrn.com/abstract=415043.
Castronova, E. (2006). On the research value of large games: Natural experiments in Norrath and Camelot. Games and Culture, 1(2), 163–186. doi:10.1177/
1555412006286686

12

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Castronova, E., Bell, M. W., Carlton, M., Cornell, R., Cummings, J. J., Emigh,
W., … , Starks, R. (2008). A test of the law of demand in a virtual world: Exploring the Petri dish approach to social science. CESifo Working Paper No. 2355.
Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1173642.
Castronova, E., Bell, M. W., Cornell, R., Cummings, J. J., Falk, M., Ross, T. L., … ,
Field, A. (2009). Synthetic worlds as experimental instruments. In B. Perron & M.
J. P. Wolf (Eds) The Video Game Theory Reader 2 (272–294). New York, NY: Routledge.
Castronova, E., Cummings, J., Emigh, W., Fatten, M., Mishler, N., Ross, T. L., & Ryan,
W. (2009). Case study: The economics of Arden. Critical Studies in Media Communication, 26(2), 165–179.
Castronova, E., & Falk, M. (2009). Virtual worlds. Games and Culture, 4(4), 396.
Castronova, E., Ross, T. L., Bell, M., Falk, M., Cornell, R., & Haselton, M. (2008). Constructing Arden: Life inside the machine. IEEE MultiMedia, 15(1), 4–8.
Castronova, E., Ross, T. L., & Knowels, I. (2013). Game analytics and human knowledge. In M. S. El-Nasr, A. Drachen & A. Canossa (Eds.), Game analytics: Maximizing
the value of player data. New York, NY: Springer-Verlag.
Centola, D. (2010). The spread of behavior in an online social network experiment.
Science, 329(5996), 1194–1197. doi:10.1126/science.1185231
Conway, J. (1970). The game of life. Scientific American, 223(4), 4.
Dabbish, L., Kraut, R., & Patton, J. (2012). Communication and commitment in an
online game team (879–888). Presented at the Proceedings of the 2012 ACM Annual
Conference on Human Factors in Computing Systems, ACM.
Ducheneaut, N., Yee, N., Nickell, E., & Moore, R. (2006). Building an MMO with mass
appeal: A look at gameplay in world of warcraft. Games and Culture, 1(4), 281–317.
doi:10.1177/1555412006292613
Lofgren, E. T., & Fefferman, N. H. (2007). The untapped potential of virtual game
worlds to shed light on real world epidemics. The Lancet, 7, 625–629.
Lorenz, E. N. (1993). The essence of chaos. Seattle: University of Washington Press.
Mason, W. & Suri, S. (2010). Conducting behavioral research on Amazon’s mechanical turk. SSRN eLibrary. Retrieved from http://ssrn.com/paper=1691163.
Mason, W., & Watts, D. J. (2012). Collaborative learning in networks. Proceedings of
the National Academy of Sciences, 109(3), 764–769.
Miller, J. H., & Page, S. E. (2007). Complex Adaptive Systems: An Introduction to Computational Models of Social Life. Princeton, NJ: Princeton University Press.
Ninja Metrics (n.d.). Ninja metrics creators of the Katana social analytics engine.
Retrieved January 14, 2013, from http://www.ninjametrics.com/.
Project Augur. (2013, January 10). Project Augur. Retrieved January 10, 2013, from
http://www.etc.cmu.edu/projects/augur/.
Ross, T. L. (2009). Constructing a virtual world as a research tool: Lessons learned
from the first iteration in the sevelopment of Greenland. In Proceedings of the 2009
International Conference on Computational Science and Engineering - Volume 04 (pp.
1163–1168). IEEE Computer Society. doi:10.1109/CSE.2009.405
Ross, T. L., Castronova, E., & Wagner, G. G. (2012). Empirical methods in virtual
worlds. In C. N. Silva (Ed.), Online research methods in urban and planning studies:
Design and outcomes. IGI-Global: Hershey, PA.

Virtual Worlds as Laboratories

13

Ross, T. L., & Cornell, R. D. (2010). Towards an experimental methodology of virtual
world research. In Proceedings of the 2010 Second International Conference on Games
and Virtual Worlds for Serious Applications (pp. 143–150). IEEE Computer Society.
doi:10.1109/VS-GAMES.2010.16
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006). Experimental study of inequality
and unpredictability in an artificial cultural market. Science, 311(5762), 854–856.
doi:10.1126/science.1121066
Simon, H. A. (1996). Designing organizations for an information-rich world. International Library of Critical Writings in Economics, 70, 187–202.
Sottek, T. C., & Warren, T. (2013 January 8). Exclusive interview: Valve’s Gabe
Newell on Steam Box, biometrics, and the future of gaming. The VergeRetrieved January 10, 2013, from http://www.theverge.com/2013/1/8/3852144/gabe-newellinterview-steam-box-future-of-gaming.
Szell, M., Lambiotte, R., & Thurner, S. (2010). Multirelational organization of
large-scale social networks in an online world. Proceedings of the National Academy
of Sciences, 107(31), 13636–13641. doi:10.1073/pnas.1004008107
Virtual Worlds Observatory (n.d.). Virtual worlds observatory | virtual worlds
observatory. Retrieved January 14, 2013, from http://129.105.161.80/wp/?
page_id=234.
Williams, D. (2010a). The promises and perils of large-scale data extraction (TBA).
Chicago, IL: MacArthur Foundation.
Williams, D. (2010b). The mapping principle, and a research framework for virtual worlds. Communication Theory, 20(4), 451–470. doi:10.1111/j.1468-2885.2010.
01371.x
Williams, D., Consalvo, M., Caplan, S., & Yee, N. (2009). Looking for gender (LFG):
Gender roles and behaviors among online gamers. Journal of Communication, 59,
700–725.
Williams, D., Yee, N., & Caplan, S. E. (2008). Who plays, how much, and why?
Debunking the stereotypical gamer profile. Journal of Computer-Mediated Communication, 13(4), 993–1018. doi:10.1111/j.1083-6101.2008.00428.x
Wisdom, T. N., & Goldstone, R. L. (2010). Social learning and cumulative innovations in a networked group. In S. K. Chai, J. J. Salerno & P. L. Mabry (Eds.),
Advances in social computing (Vol. 6007/2010, pp. 32–41). Heidelberg/Berlin, Germany: Springer-Verlag.

FURTHER READING
Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science,
317(5837), 472–476. doi:10.1126/science.1146930
Ducheneaut, N. (2010). Massively multiplayer online games as living laboratories:
Opportunities and pitfalls. In W. S. Bainbridge (Ed.), Online worlds: Convergence of
the real and the virtual (pp. 135–145). Berlin: Springer.
Goldstone, R. L., & Gureckis, T. M. (2009). Collective behavior. Topics in Cognitive
Science, 1(3), 412–438.

14

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Ross, T. L., Castronova, E., & Wagner, G. G. (2012). Empirical methods in virtual
worlds. In C. N. Silva (Ed.), Online research methods in urban and planning studies:
design and outcomes. IGI-Global: Hershey, PA.
Williams, D. (2010). The mapping principle, and a research framework for
virtual worlds. Communication Theory, 20(4), 451–470. doi:10.1111/j.1468-2885.
2010.01371.x

TRAVIS L. ROSS SHORT BIOGRAPHY
Travis L. Ross is a PhD candidate in Telecommunications and Cognitive Science at Indiana University. His research focuses on the forces that shape antisocial and prosocial behavior in online societies and the psychology of motivation. He has worked on a number of projects that used virtual worlds or
games to study social behavior and has written extensively about using virtual worlds as laboratories for social science experimentation.
Travis’s Website: www.travislross.com
Travis’s Work: www.motivateplay.com
EDWARD CASTRONOVA SHORT BIOGRAPHY
Edward Castronova is a Professor of Telecommunications and Cognitive Science, Indiana University. Castronova (PhD Economics, Wisconsin, 1991) is
a founder of scholarly online game studies and an expert on the societies
of virtual worlds. Among his academic publications on these topics are two
books: Synthetic Worlds (University of Chicago Press, 2005) and Exodus to
the Virtual World (Palgrave, 2007). Professor Castronova teaches graduate
and undergraduate courses on the design of games, the game industry, and
the management of virtual societies. Outside his academic work, Professor
Castronova makes regular appearances in mainstream media (60 Minutes,
the New York Times, and The Economist), gives keynotes at major conferences (Austin Game Conference, Digital Games Research Association Conference, and Interactive Software Federation of Europe), and consults for
business (McKinsey, Vivendi, and Forrester).
Edward’s Website: http://mypage.iu.edu/∼castro/
ISAAC KNOWLES SHORT BIOGRAPHY
Isaac Knowles is a PhD student in the Department of Telecommunications at
Indiana University. His work focuses on the structure, regulation, and measurement of virtual economic activities and their relationship with the real
economy. He received his BS in economics from Mary Washington College
in Fredericksburg, Virginia in 2007 and his MS in economics from Louisiana

Virtual Worlds as Laboratories

15

State University in Baton Rouge in 2012. He has also worked as an economic
research analyst at the US Federal Trade Commission and as an analyst for
SAP Precision Gaming.
RELATED ESSAYS
Social Epigenetics: Incorporating Epigenetic Effects as Social Cause and
Consequence (Sociology), Douglas L. Anderton and Kathleen F. Arcaro
To Flop Is Human: Inventing Better Scientific Approaches to Anticipating
Failure (Methods), Robert Boruch and Alan Ruby
Repeated Cross-Sections in Survey Data (Methods), Henry E. Brady and
Richard Johnston
Ambulatory Assessment: Methods for Studying Everyday Life (Methods),
Tamlin S. Conner and Matthias R. Mehl
Models of Nonlinear Growth (Methods), Patrick Coulombe and James P.
Selig
Quantile Regression Methods (Methods), Bernd Fitzenberger and Ralf
Andreas Wilke
The Evidence-Based Practice Movement (Sociology), Edward W. Gondolf
Meta-Analysis (Methods), Larry V. Hedges and Martyna Citkowicz
The Use of Geophysical Survey in Archaeology (Methods), Timothy J.
Horsley
Network Research Experiments (Methods), Allen L. Linton and Betsy Sinclair
Longitudinal Data Analysis (Methods), Todd D. Little et al.
Structural Equation Modeling and Latent Variable Approaches (Methods),
Alex Liu
Data Mining (Methods), Gregg R. Murray and Anthony Scime
Remote Sensing with Satellite Technology (Archaeology), Sarah Parcak
Quasi-Experiments (Methods), Charles S. Reichard
Digital Methods for Web Research (Methods), Richard Rogers
Modeling Life Course Structure: The Triple Helix (Sociology), Tom Schuller
Content Analysis (Methods), Steven E. Stemler
Person-Centered Analysis (Methods), Alexander von Eye and Wolfgang
Wiedermann
Translational Sociology (Sociology), Elaine Wethington



Virtual Worlds as Laboratories
TRAVIS L. ROSS, EDWARD CASTRONOVA, and ISAAC KNOWLES

Abstract
A virtual world is a persistent space where tens, hundreds, thousands, or even millions of users interact with each other and a mediated environment defined as a
physical space through rules created by designers and enforced by computer code.
Researchers have argued that these characteristics make virtual worlds particularly
well suited for conducting parallel experiments to test macro-level social theory. The
purpose of this essay is to provide an introduction into virtual worlds research. It is
not an exhaustive resource chronicling the history of virtual worlds, but rather an
introduction broken into three sections for those wishing to learn more about the
past, present, and future directions of the topic. First, it explores what researchers
have said about using virtual worlds research and the fields of research where virtual
worlds have been used. In doing so, it focuses on research in video games studies and
complex systems. Second, it examines cutting-edge work in virtual worlds research,
identifying that both academia and the game industry will play a significant role in
the success and direction. Third, it identifies six key issues that scholars using virtual
worlds research will face as they move forward.

INTRODUCTION
Over the last decade, social scientists have expressed interest in using virtual
worlds to study the societies of the real world. The argument follows that
virtual worlds can track user behavior in a database, that virtual worlds can
be experimentally manipulated and run in parallel for controlled experimentation, and that virtual worlds are inhabited by societies of users. These societies are of interest to researchers because they feature project-based teams,
corporations, governments, and economies. In other words, virtual worlds
have the same complex social arrangements that you find in the real world,
and they afford researchers powerful tools for conducting research.
But what is a virtual world? Bartle (2004), who constructed the first virtual
world with Roy Trubshaw, defines one as a persistent online space with physical rules that is shared by many players who can interact with the world and
each other. When he adopted this definition, he was comparing massive multiplayer online games such as Everquest, Ultima Online, Linage, and Multiuser
Emerging Trends in the Social and Behavioral Sciences. Edited by Robert Scott and Stephen Kosslyn.
© 2015 John Wiley & Sons, Inc. ISBN 978-1-118-90077-2.

1

2

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Dungeon to local multiplayer games such as Half-life, Quake, and Mario Kart.
At that time, it was easy to segment environments that were virtual worlds
from those that were not. There were some hybrids, such as Diablo 2, but they
numbered few.
Ask for an example of a virtual world now and the answer might be World of
Warcraft, EVE Online, Second Life, Minecraft, Modern Warfare, Parallel Kingdoms,
or League of Legends. The point being, there are now many online spaces that
are clearly virtual worlds (World of Warcraft, EVE Online, or Second Life) and
there are also many online spaces that straddle the boundary (Modern Warfare, Diablo 3, Minecraft, or League of Legends). Virtual worlds are now more
difficult to delineate from nonvirtual worlds. They contain elements that are
both persistent and nonpersistent, have a few players upward to millions of
players, and have rules that bridge the virtual and the real.
In order to focus on the virtual worlds that have interested social scientists
over the last few decades, it is important to have a working definition that
limits scope. Environments that fall out of this definition are still interesting and share properties with virtual worlds, some more so than others, but
when scholars discuss virtual world experiments, they always possess the
following four characteristics:
Virtual Worlds are Persistent and Multiplayer. They are inhabited by a group
of users and information about those users continues to exist in a
database when those users are not present.
Virtual Worlds Incorporate Space. Whether one-, two-, or three-dimensional
users of virtual worlds interact in a digital representation of a physical
space. Fantasy football does not have a physical space. Second Life does.
Virtual Worlds are Designed. A user can interact with a virtual world
through an interface that is defined by rules created by a designer or
engineer.
The Rules of a Virtual World are Enforced by Computer Code. What separates a
virtual world from a sports league, the television show Survivor, or The
Stanford Prison Experiment is that they instantiated and enforced by a
computer.
Putting all three together into one definition: A virtual world is a persistent
space where tens, hundreds, thousands, or even millions of users interact
with each other and a mediated environment defined as a physical space
through rules created by designers and enforced by computer code.
Each of these characteristics makes a virtual world a valuable tool for studying human society. A persistent world uses a database. The behaviors of
every user in the world can be tracked and stored. Using the appropriate tools

Virtual Worlds as Laboratories

3

for analysis and visualization, a researcher can ask questions about individuals, groups segmented by variables of interest, or the entire population. By
designing the rules and the physical space, a research can craft an environment that is balanced between abstraction and replication of the real world
for the specific question of interest. Finally, computer code allows a virtual
world to be tinkered with and copied. The experimental method requires
running at least two identical environments in parallel while manipulating
only the variable of interest. Virtual worlds are suited for experiments and
scholars have compared them to Petri dishes or neighboring island nations
(Bradley & Froomkin, 2004; Castronova, 2006). In these examples, each world
is constructed with the same rules except for a change in a variable of interest.
It is then populated with a random sample of users and the society is tracked
over time. If behavior in virtual worlds truly parallels behavior in the real
world, then they present a powerful way to study society. It is no wonder that
virtual world research has generated interest in fields such as public health,
economics, communication, complex systems, law, and public policy.
Like the ghosts of A Christmas Carol, this essay explores the past, present,
and future of virtual worlds research. The section titled “Foundational
Research” introduces the past and explores the foundational research for
experimentation with virtual worlds. It shows that researchers working
in complex systems, simulation, and agent-based modeling were actually
the first to consider the use of virtual worlds for social experimentation.
However, it was the video game industry and video game scholars made
it possible to conduct experiments with real human beings. The section
titled “Cutting-Edge Work” focuses on the present. It explores cutting-edge
research in both academia and the game industry, and it introduces studies
and tools that will shape the future of virtual worlds research. The section
titled “Key Issues” focuses on the key issues moving forward and identifies
six issues that researchers crafting virtual worlds for social science research
will have to address moving forward.
FOUNDATIONAL RESEARCH
In the early 2000s (decade), researchers studying games and media identified
that virtual worlds offered a unique opportunity for conducting controlled
experiments on human societies and social groups. They recognized that virtual worlds were designed and instantiated through computer code and that
a scientist could manipulate the code to study social behavior in a series of
parallel experiments. This was the first time that virtual worlds from the
entertainment industry were identified as tools for social scientific research,
but it was not the first time that virtual worlds had been used to study social
behavior. It is the field of complex systems holds that distinction.

4

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Complex systems are nonlinear systems composed of simpler interworking
parts. Ecosystems, beehives, brains, and governments are all complex systems. Each is a collection of individual organisms, bees, neurons, or people
interacting to produce emergent behavior. The origins of complex systems
research can be traced back a few hundred years to Darwin and Adam Smith,
whose theories demonstrate that complexity can arise from a few simple
rules. Much of this early work relied on a top-down approach and used the
scientific paradigm of reductionism to examine the individual components
of the system. This approach, however, was met with limited success because
it is difficult to understand the behavior of a beehive by studying the behavior of a single bee. A bottom-up approach was required, but it was not until
the creation of the modern computer that scientists were able to build virtual
worlds, populate them with autonomous agents, and test hypothesis about
the system as a whole, a technique known as agent-based modeling.
Perhaps the best-known early work in the agent-based modeling comes
from Conway (1970) whose Game of Life spawned a variety of “life-like”
agents in a virtual world with a few simple behavioral rules. Other early
work includes Schelling’s agent-based models of segregation, which show
how a simple rule regarding tolerance to diversity can lead to highly
segregated communities, and Axelrod’s (1987) famous evolutionary tournament designed to test the evolutionary success of cooperative and selfish
agents playing a repeated prisoners’ dilemma. From this groundbreaking
work, an entire field has emerged. It uses virtual worlds to study complex
systems such as food chains, traffic patterns, flocking, local economies, and
information transmission.
Although scholars studying virtual worlds for entertainment rarely mention it, research in the field of complex systems and agent-based modeling
is important, and the two fields are likely to intersect. As Section titled
“Key Issues” of this essay will show, researchers in complex systems have
identified and are currently tackling some of the most important obstacles
for the study of human society using virtual worlds. That said, there is one
important distinction between the two fields. Virtual worlds research proposes using real human societies in virtual worlds to study social behavior,
while agent-based modeling uses computerized agents driven by simple
rules. Agent-based models have their advantages. They can run iterations
of a complex system and examine the behavior of agents across millions of
rounds or generations in a manner of seconds. They also have a weakness.
The predictions of the model are based on simple agents, not real humans.
The foundational research for using virtual worlds with real human beings
as laboratories for social scientific research is largely theoretical and philosophical in nature. Massive multiplayer online virtual worlds have existed
since the late 1970s, but the potential of virtual worlds for social research did

Virtual Worlds as Laboratories

5

not become apparent until Castronova (2001, 2003) demonstrated, through
the use of econometrics, that virtual economies were similar and connected
to the real economy. Afterward, Bradley and Froomkin (2004) picked up
on this work and were the first to identify that virtual worlds, with their
real economies, were well suited to operate as a test bed for law and policy,
especially when the law or policy was too risky or intractable for experimentation in the real world. In 2006, Castronova (2006) performed a case
study on two natural experiments that tested game theoretic coordination
problems in the online role-playing games Dark Age of Camelot and Everquest.
Since then, Castronova’s research has stressed the importance of developing
virtual worlds as a scientific apparatus for conducting experiments on
societies. He has stated many times that policymakers should invest in
virtual worlds as they do in supercolliders, satellites, or other high-tech
research tools (Castronova & Falk, 2009; Castronova, Bell, et al., 2008; 2009;
Ross, Castronova, & Wagner, 2012).
One of the more famous instances where researchers recognized the power
of virtual world research happened by accident. It was triggered by a glitch
known as the Corrupted Blood incident, which acted like a real virus, spreading
through the game World of Warcraft killing thousands of players. Citing the
glitch as an example, Balicer (2007) proposed that data from virtual worlds
could be used to model the spread of infectious disease, and shortly after,
Lofgren and Fefferman (2007) demonstrated that the Corrupted Blood incident shared many of the characteristics of real-world epidemics. Later that
year, the potential of virtual worlds reached a wider scientific community
when Bainbridge (Bainbridge, 2007) published an article in the journal Science detailing the work of scholars thus far and championing virtual worlds
as the future of social science experimentation.
Most of the foundational research regarding the use of synthetic worlds to
study social science has been philosophical in nature; however, there were a
few attempts in the early 2000s (decade) to build virtual worlds for researcher
purposes. In 2005, Castronova led a team of researchers and developers on a
project named Arden, a virtual world based on the works of William Shakespeare. According to Castronova et al. (2008), Arden had two goals, to educate
players by immersing them in the work of William Shakespeare and to provide a platform for cross-server economic experimentation. By late 2006, it
was apparent that Arden would not meet its ambitious long-term goals, so
the project was scaled back and Castronova acknowledge that the failure of
Arden might serve as a warning of the difficulties of virtual world research
(Baker, 2008; Castronova, Cummings, et al., 2009).
In 2008, following his work on Arden, Castronova, Ross, and a team of
researchers began work on a web-based virtual world named Greenland
(Ross, 2009). Once again the goal was macroeconomic experimentation, this

6

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

time specifically focused on the emergence of currency. Construction of the
world was completed in January 2009, and a pretest was conducted with 700
participants and 5 servers. Although the game maintained an active player
base for 2 months, the targeted economic theories were never tested. Once
again the experiment was considered unsuccessful as unexpected expansion
and warfare in the game disrupted economic activity (Ross, 2009; Ross &
Cornell, 2010).
The most successful examples of research involving online societies can be
found in the field of complex systems. In 2006, Salganik, Dodds, and Watts
(2006) constructed parallel instances of a music sharing website where they
tested social learning theory by comparing the influence of song popularity
and song rating on the number of downloads. Although this experiment did
not use a virtual world as defined here, it was groundbreaking because it
provided a successful example of how to test a hypothesis about macro-level
social behavior in parallel online experiments.
Due to the difficulties of virtual world experiments, some researchers have
turned to using the large datasets of existing virtual worlds as a basis for
comparing real-world behavior to virtual behavior. In 2006, Ducheneaut,
Yee, Nickell, and Moore (2006) collected information about the players of
World of Warcraft using autonomous bots and examined how the design of
World of Warcraft motivated the behavior of players and social groups. In
2009, another group of researchers started The Virtual World Observatory,
which is a multiuniversity research group created to study social behavior
using large datasets from popular virtual worlds. The group has published
a considerable amount of research, with much of it based on findings
generated using a four-terabyte dataset from the virtual world Everquest
(Virtual Worlds Observatory, n.d.). In addition to performing research, the
group also tackled some of the major organizational and access problems
that researchers must face when attempting to store and analyze large
amounts of data from virtual worlds (Williams, 2010a).
CUTTING-EDGE WORK
Over the past 5 years, some of the momentum behind using industry grade
virtual worlds as experimental environments has abated, as a number of
projects have demonstrated that building games and virtual worlds for scientific experiments is a considerable challenge. Yet, the momentum continues in
two directions. First, the video game industry is interested in the using social
science theory, experimentation, and analytics for understanding, predicting,
and shaping the behavior of players in online games. Second, academics have
scaled back virtual worlds research. Instead of trying to build industry grade

Virtual Worlds as Laboratories

7

worlds populated by thousands of people, they are now using simpler virtual
worlds to study the dynamics of small groups.
The most powerful tools for testing macro-level social theory now exist in
the game industry. Due to this phenomenon, Castronova, Ross, and Knowels
(2013) argue that over the coming decades the major advances in social
science will be made by game designers tinkering with online societies and
not social scientists in universities. The move toward advanced tools for
behavior tracking, analytics, and visualization in the game industry has been
swift. Over the past 5 years, companies such as Zynga, Bioware, Electronic
Arts, Activision/Blizzard, Riot Games, Valve Software, and CCP Games
have gone from having almost no capabilities for analytics to possessing
the most advanced tools for studying societies ever created. The Game
Developers Conference, the premier forum for discussing the development
of games, now regularly features discussions about the best practices for
studying, creating, and maintaining economies and social institutions in virtual worlds. Many social scientists also been pulled into this work, as Valve
Software, CCP Games, Disney/Playdom, and Riot Games have all recently
hired social scientists to conduct research and design online societies.
While the game industry has scaled up the practice of virtual worlds
research, researchers in academia have scaled back. Currently, the most
successful examples in academia come from researchers using virtual worlds
to study small groups on a short-time scale. For example, Dabbish, Kraut,
and Patton (2012) recently installed World of Warcraft on a private server
and used it to study behavior in groups of 5–25 players. Team members
of The Augur Project at the Carnegie Mellon ETC have used Amazon’s
Mechanical Turk in combination with three of game-like prototypes to study
and build predicative models of player behavior (Augur, 2013). Wisdom and
Goldstone (2010) recently created a small-scale asynchronous virtual world
based around a puzzle game. It allowed players to see information about
what past players had done and was used to test social learning theory. In
complex systems, Mason and Watts (2012) conducted a series of experiments
with simple web-based games to study collaborative problem solving in
groups, and Centola (2010) examined how social networks influence the
exchange of health information by experimenting with parallel interactive
websites. These examples do not exhaust the current research, but they do
serve to demonstrate the direction of virtual worlds research in academia.
Finally, researchers continue to work with large-scale datasets. Szell,
Lambiotte, and Thurner (2010) examined the structure of social networks in
the game Pardus, a virtual world claiming over 100,000 active users. While
Williams and others have begun to explore the link between behavior and
motivation, using surveys linked to behavioral data in virtual worlds such
as League of Legends and Everquest 2 (Virtual Worlds Observatory, n.d.),

8

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Williams’s work has also moved him into the game industry. He recently
stared the company Ninja Metrics to provide social network analysis and
machine learning to game companies (Ninja Metrics, n.d.).
KEY ISSUES
Virtual worlds present an exciting opportunity for researchers looking to
study social systems, but the research will require overcoming some key
issues. This section identifies the main issues facing researchers studying
social systems in virtual worlds. They are the mapping principle, big data,
the attention economy, player synchronization, connecting to mental states, and
complex and chaotic systems.
THE MAPPING PRINCIPLE
Williams (2010b) coined “the mapping principle” as a problem for
researchers wanting to study virtual worlds. In his article, he argues
that using virtual worlds for research is a good idea, but that researchers
wishing to pursue it must understand the similarities and differences
between real and virtual behavior. Virtual worlds are abstractions of the real
world, and thus behavior does not always match one to one. The researchers
who aimed to study the spread of epidemics using the Corrupted Blood
incident in World of Warcraft encountered the mapping principle first hand.
They witnessed infected individuals trying to infect others because it was
funny or dying was of little consequence.
The mapping principle is not entirely a new problem for experimentalists and follows closely to the problem of maintaining ecological or external
validity. When experimental researchers create controlled environments for
the study of real-world decision-making, they make choices about abstraction and how costs and incentives match those of the real world. Maintaining
external validity can be difficult in even simple experiments. The complexity of virtual worlds and the game design decisions that go into them make
constructing virtual worlds for targeted experimentation even more of a challenge (Ross & Cornell, 2010).
BIG DATA
Another problem that researchers studying virtual worlds must face is
storing and analyzing the large, and growing, behavioral datasets of virtual
worlds. Storage space is becoming less expensive and storing terabytes of
data is no longer a major concern. What is of concern for researchers are
the methods for accessing, analyzing, visualizing, and interpreting this

Virtual Worlds as Laboratories

9

data. As the social sciences continue to harness the powers of big data and
computation, it is becoming apparent that scholars in the social sciences and
the humanities must work more closely with researchers, such as computer
scientists, familiar with the techniques and methods required to harness
big data.
ATTENTION ECONOMY
Human beings have limited attention. In our mediated world, attention is
a valuable commodity and there are millions of experiences competing for
this valuable resource, creating what has been termed an attention economy
(Simon, 1996). Researchers who wish to use virtual worlds for studying social
systems will have to compete for attention. Currently, if a scientist needs a
few hundred participants to come into the lab for an hour, they attract them
with extra credit or money. Depending on the question, a scientist using a
virtual world may need more than a few hundred participants and may also
need them for a long period of time. A number of research projects have used
Amazon’s Mechanical Turk to attract a few thousand participants and independent research has validated the results of participants using the service
(Mason & Suri, 2010). Mechanical Turk may present one low-cost method for
attracting players, but researchers wishing to attract players without paying
them must figure out how to create engaging experiences that can attract the
attention of users and maintain ecological validity.
PLAYER SYNCHRONIZATION
The most interesting questions that can be asked using virtual worlds involve
societies interacting over time. This presents a coordination problem for any
researcher asking these questions using a virtual world because players must
all be present in the world at the same time in order to interact with each
other. To make matters worse, many virtual worlds suffer from high attrition
rates and players dropping out or not playing at the same time can create
validity problems for researchers.
One solution is to let players interact asynchronously. Changing the progression of time from continuous to larger discrete units (turns) allows players to enter the virtual world and make strategic decisions at any time during
the turn. There are many successful virtual worlds that use asynchronous
play, but one concern is that it slows the pace of the game and decreases the
number and type of strategic interactions. When a player finishes their turn,
they must wait for all of the other players in the game to react, or for the turn
timer to expire. Some research questions may not be approachable through

10

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

asynchronous play; therefore, researchers must explore the limitations created by the difficulties of synchronization.
CONNECTING BEHAVIOR TO MENTAL STATES
Virtual worlds are good at answering questions about behavior because they
can track it at a very fine time scale. What they are not good at is connecting behavior to the beliefs and desires. Researchers using virtual worlds and
machine learning can make very accurate predictions about the behavior of
individuals, but are challenged when explaining why an individual makes a
certain decision. To overcome this hurdle, researchers have linked self-report
data to behavioral data in order to explore the motivations of players, and
game developers have discussed collecting biometric feedback through controllers (Caplan, Williams, & Yee, 2009; Sottek & Warren, 2013; Williams, Consalvo, Caplan, & Yee, 2009; Williams, Yee, & Caplan, 2008). Still, the connection between behavior and beliefs and desires is limited in virtual worlds and
a problem that researchers must continue to address.
COMPLEX AND CHAOTIC SYSTEMS
Virtual worlds are complex, they have interworking parts that lead to emergent behavior, and are also chaotic, living on the border of order and randomness. In complex and chaotic systems, small decisions early on can lead
to big differences further in time (Miller & Page, 2007). For example, many are
familiar with the metaphor that describes how flapping butterfly wings may
lead to the formation of a hurricane (Lorenz, 1993). It is an extreme example,
but it depicts chaotic systems quite well. As time progresses, societies can
have multiple branches. Small fluctuations can equal big changes. What if
Gore had won the election instead of Bush? If we run the simulation 100 times
with different people in the key roles, would the outcome remain the same
every time? Probably not. Scientists who are studying virtual worlds must
prepare for uncertainty due to early decisions and differences in populations.
They must consider the choices and rules that they design into the game and
understand the motivations of players, and they must run multiple occurrences of the same exact world in parallel. Only then can they understand
when and why a society takes a different path.
CONCLUSION
Over the last decade, game developers have attempted to build hundreds of
virtual worlds and many of them have failed. In order to navigate the difficult of building virtual worlds, the mapping problem, external validity, and

Virtual Worlds as Laboratories

11

the attention economy designers have to carefully combine theory and game
design. Unanticipated design decisions or the decisions of a few individuals
can lead to very different social outcomes in virtual worlds. This means that
researchers who wish to gain the most from virtual worlds must run multiple instances in parallel to establish consistent outcomes and publish the
design, method, and results so that other scientists can replicate the experiment and compare their own results. How will researchers publish the design
and programing decisions made when constructing virtual worlds? In addition, there is little theory available to those who are building simulations of
reality. The best theories of game design, even in serious games, are institutionalized and internalized within a few talented individuals. In order to
realize the potential of virtual world research, there must be a shared knowledgebase of the mapping principle and difficulties encountered during along
the way. Virtual worlds present a powerful opportunity for social scientists,
but the construction and use of these systems may rival in scale and complexity the challenges of building satellites for space exploration or super
colliders for physics.
REFERENCES
Axelrod, R. (1987). The evolution of strategies in the iterated prisoner’s dilemma.
Genetic Algorithms and Simulated Annealing, 3(1), 32–41.
Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science,
317(5837), 472–476. doi:10.1126/science.1146930
Baker, C. (2008 March 24). Trying to design a truly entertaining game can defeat even
a certified genius. WIRED Magazine. Retrieved January 8, 2013, from http://www.
wired.com/gaming/gamingreviews/magazine/16-04/pl_games.
Balicer, R. D. (2007). Modeling infectious diseases dissemination through online
role-playing games. Epidemiology, 18(2), 260–261.
Bartle, R. A. (2004). Designing virtual worlds. Indianapolis: New Riders.
Bradley, C., & Froomkin, M. (2004). Virual worlds, real rules. New York Law School
Law Review, 49(1), 103–146.
Caplan, S., Williams, D., & Yee, N. (2009). Problematic Internet use and psychosocial
well-being among MMO players. Computers in Human Behavior, 25, 1312–1319.
Castronova, E. (2001). Virtual worlds: A first-hand account of market and society
on the cyberian frontier. The Gruter Institute Working Papers on Law, Economics, and
Evolutionary Biology, 2, 1–68.
Castronova, E. (2003). The price of “man” and “woman”: A hedonic pricing model
of avatar attributes in a synthethic world. CESifo Working Paper Series, No. 957.
Retrieved from http://ssrn.com/abstract=415043.
Castronova, E. (2006). On the research value of large games: Natural experiments in Norrath and Camelot. Games and Culture, 1(2), 163–186. doi:10.1177/
1555412006286686

12

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Castronova, E., Bell, M. W., Carlton, M., Cornell, R., Cummings, J. J., Emigh,
W., … , Starks, R. (2008). A test of the law of demand in a virtual world: Exploring the Petri dish approach to social science. CESifo Working Paper No. 2355.
Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1173642.
Castronova, E., Bell, M. W., Cornell, R., Cummings, J. J., Falk, M., Ross, T. L., … ,
Field, A. (2009). Synthetic worlds as experimental instruments. In B. Perron & M.
J. P. Wolf (Eds) The Video Game Theory Reader 2 (272–294). New York, NY: Routledge.
Castronova, E., Cummings, J., Emigh, W., Fatten, M., Mishler, N., Ross, T. L., & Ryan,
W. (2009). Case study: The economics of Arden. Critical Studies in Media Communication, 26(2), 165–179.
Castronova, E., & Falk, M. (2009). Virtual worlds. Games and Culture, 4(4), 396.
Castronova, E., Ross, T. L., Bell, M., Falk, M., Cornell, R., & Haselton, M. (2008). Constructing Arden: Life inside the machine. IEEE MultiMedia, 15(1), 4–8.
Castronova, E., Ross, T. L., & Knowels, I. (2013). Game analytics and human knowledge. In M. S. El-Nasr, A. Drachen & A. Canossa (Eds.), Game analytics: Maximizing
the value of player data. New York, NY: Springer-Verlag.
Centola, D. (2010). The spread of behavior in an online social network experiment.
Science, 329(5996), 1194–1197. doi:10.1126/science.1185231
Conway, J. (1970). The game of life. Scientific American, 223(4), 4.
Dabbish, L., Kraut, R., & Patton, J. (2012). Communication and commitment in an
online game team (879–888). Presented at the Proceedings of the 2012 ACM Annual
Conference on Human Factors in Computing Systems, ACM.
Ducheneaut, N., Yee, N., Nickell, E., & Moore, R. (2006). Building an MMO with mass
appeal: A look at gameplay in world of warcraft. Games and Culture, 1(4), 281–317.
doi:10.1177/1555412006292613
Lofgren, E. T., & Fefferman, N. H. (2007). The untapped potential of virtual game
worlds to shed light on real world epidemics. The Lancet, 7, 625–629.
Lorenz, E. N. (1993). The essence of chaos. Seattle: University of Washington Press.
Mason, W. & Suri, S. (2010). Conducting behavioral research on Amazon’s mechanical turk. SSRN eLibrary. Retrieved from http://ssrn.com/paper=1691163.
Mason, W., & Watts, D. J. (2012). Collaborative learning in networks. Proceedings of
the National Academy of Sciences, 109(3), 764–769.
Miller, J. H., & Page, S. E. (2007). Complex Adaptive Systems: An Introduction to Computational Models of Social Life. Princeton, NJ: Princeton University Press.
Ninja Metrics (n.d.). Ninja metrics creators of the Katana social analytics engine.
Retrieved January 14, 2013, from http://www.ninjametrics.com/.
Project Augur. (2013, January 10). Project Augur. Retrieved January 10, 2013, from
http://www.etc.cmu.edu/projects/augur/.
Ross, T. L. (2009). Constructing a virtual world as a research tool: Lessons learned
from the first iteration in the sevelopment of Greenland. In Proceedings of the 2009
International Conference on Computational Science and Engineering - Volume 04 (pp.
1163–1168). IEEE Computer Society. doi:10.1109/CSE.2009.405
Ross, T. L., Castronova, E., & Wagner, G. G. (2012). Empirical methods in virtual
worlds. In C. N. Silva (Ed.), Online research methods in urban and planning studies:
Design and outcomes. IGI-Global: Hershey, PA.

Virtual Worlds as Laboratories

13

Ross, T. L., & Cornell, R. D. (2010). Towards an experimental methodology of virtual
world research. In Proceedings of the 2010 Second International Conference on Games
and Virtual Worlds for Serious Applications (pp. 143–150). IEEE Computer Society.
doi:10.1109/VS-GAMES.2010.16
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006). Experimental study of inequality
and unpredictability in an artificial cultural market. Science, 311(5762), 854–856.
doi:10.1126/science.1121066
Simon, H. A. (1996). Designing organizations for an information-rich world. International Library of Critical Writings in Economics, 70, 187–202.
Sottek, T. C., & Warren, T. (2013 January 8). Exclusive interview: Valve’s Gabe
Newell on Steam Box, biometrics, and the future of gaming. The VergeRetrieved January 10, 2013, from http://www.theverge.com/2013/1/8/3852144/gabe-newellinterview-steam-box-future-of-gaming.
Szell, M., Lambiotte, R., & Thurner, S. (2010). Multirelational organization of
large-scale social networks in an online world. Proceedings of the National Academy
of Sciences, 107(31), 13636–13641. doi:10.1073/pnas.1004008107
Virtual Worlds Observatory (n.d.). Virtual worlds observatory | virtual worlds
observatory. Retrieved January 14, 2013, from http://129.105.161.80/wp/?
page_id=234.
Williams, D. (2010a). The promises and perils of large-scale data extraction (TBA).
Chicago, IL: MacArthur Foundation.
Williams, D. (2010b). The mapping principle, and a research framework for virtual worlds. Communication Theory, 20(4), 451–470. doi:10.1111/j.1468-2885.2010.
01371.x
Williams, D., Consalvo, M., Caplan, S., & Yee, N. (2009). Looking for gender (LFG):
Gender roles and behaviors among online gamers. Journal of Communication, 59,
700–725.
Williams, D., Yee, N., & Caplan, S. E. (2008). Who plays, how much, and why?
Debunking the stereotypical gamer profile. Journal of Computer-Mediated Communication, 13(4), 993–1018. doi:10.1111/j.1083-6101.2008.00428.x
Wisdom, T. N., & Goldstone, R. L. (2010). Social learning and cumulative innovations in a networked group. In S. K. Chai, J. J. Salerno & P. L. Mabry (Eds.),
Advances in social computing (Vol. 6007/2010, pp. 32–41). Heidelberg/Berlin, Germany: Springer-Verlag.

FURTHER READING
Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science,
317(5837), 472–476. doi:10.1126/science.1146930
Ducheneaut, N. (2010). Massively multiplayer online games as living laboratories:
Opportunities and pitfalls. In W. S. Bainbridge (Ed.), Online worlds: Convergence of
the real and the virtual (pp. 135–145). Berlin: Springer.
Goldstone, R. L., & Gureckis, T. M. (2009). Collective behavior. Topics in Cognitive
Science, 1(3), 412–438.

14

EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES

Ross, T. L., Castronova, E., & Wagner, G. G. (2012). Empirical methods in virtual
worlds. In C. N. Silva (Ed.), Online research methods in urban and planning studies:
design and outcomes. IGI-Global: Hershey, PA.
Williams, D. (2010). The mapping principle, and a research framework for
virtual worlds. Communication Theory, 20(4), 451–470. doi:10.1111/j.1468-2885.
2010.01371.x

TRAVIS L. ROSS SHORT BIOGRAPHY
Travis L. Ross is a PhD candidate in Telecommunications and Cognitive Science at Indiana University. His research focuses on the forces that shape antisocial and prosocial behavior in online societies and the psychology of motivation. He has worked on a number of projects that used virtual worlds or
games to study social behavior and has written extensively about using virtual worlds as laboratories for social science experimentation.
Travis’s Website: www.travislross.com
Travis’s Work: www.motivateplay.com
EDWARD CASTRONOVA SHORT BIOGRAPHY
Edward Castronova is a Professor of Telecommunications and Cognitive Science, Indiana University. Castronova (PhD Economics, Wisconsin, 1991) is
a founder of scholarly online game studies and an expert on the societies
of virtual worlds. Among his academic publications on these topics are two
books: Synthetic Worlds (University of Chicago Press, 2005) and Exodus to
the Virtual World (Palgrave, 2007). Professor Castronova teaches graduate
and undergraduate courses on the design of games, the game industry, and
the management of virtual societies. Outside his academic work, Professor
Castronova makes regular appearances in mainstream media (60 Minutes,
the New York Times, and The Economist), gives keynotes at major conferences (Austin Game Conference, Digital Games Research Association Conference, and Interactive Software Federation of Europe), and consults for
business (McKinsey, Vivendi, and Forrester).
Edward’s Website: http://mypage.iu.edu/∼castro/
ISAAC KNOWLES SHORT BIOGRAPHY
Isaac Knowles is a PhD student in the Department of Telecommunications at
Indiana University. His work focuses on the structure, regulation, and measurement of virtual economic activities and their relationship with the real
economy. He received his BS in economics from Mary Washington College
in Fredericksburg, Virginia in 2007 and his MS in economics from Louisiana

Virtual Worlds as Laboratories

15

State University in Baton Rouge in 2012. He has also worked as an economic
research analyst at the US Federal Trade Commission and as an analyst for
SAP Precision Gaming.
RELATED ESSAYS
Social Epigenetics: Incorporating Epigenetic Effects as Social Cause and
Consequence (Sociology), Douglas L. Anderton and Kathleen F. Arcaro
To Flop Is Human: Inventing Better Scientific Approaches to Anticipating
Failure (Methods), Robert Boruch and Alan Ruby
Repeated Cross-Sections in Survey Data (Methods), Henry E. Brady and
Richard Johnston
Ambulatory Assessment: Methods for Studying Everyday Life (Methods),
Tamlin S. Conner and Matthias R. Mehl
Models of Nonlinear Growth (Methods), Patrick Coulombe and James P.
Selig
Quantile Regression Methods (Methods), Bernd Fitzenberger and Ralf
Andreas Wilke
The Evidence-Based Practice Movement (Sociology), Edward W. Gondolf
Meta-Analysis (Methods), Larry V. Hedges and Martyna Citkowicz
The Use of Geophysical Survey in Archaeology (Methods), Timothy J.
Horsley
Network Research Experiments (Methods), Allen L. Linton and Betsy Sinclair
Longitudinal Data Analysis (Methods), Todd D. Little et al.
Structural Equation Modeling and Latent Variable Approaches (Methods),
Alex Liu
Data Mining (Methods), Gregg R. Murray and Anthony Scime
Remote Sensing with Satellite Technology (Archaeology), Sarah Parcak
Quasi-Experiments (Methods), Charles S. Reichard
Digital Methods for Web Research (Methods), Richard Rogers
Modeling Life Course Structure: The Triple Helix (Sociology), Tom Schuller
Content Analysis (Methods), Steven E. Stemler
Person-Centered Analysis (Methods), Alexander von Eye and Wolfgang
Wiedermann
Translational Sociology (Sociology), Elaine Wethington