-
Title
-
Event Processing as an Executive Enterprise
-
Author
-
Ross, Robbie A.
-
Baldwin, Dare A.
-
Research Area
-
Cognition and Emotions
-
Topic
-
Information Processing
-
Abstract
-
Actual experience as life unfolds tends to be an ebb and flow of dynamic, multimodal sensations, many of which are fleeting. Yet what is encoded, recalled, and talked about tends to be events—units of experience that are conceptualized as having both a beginning and an end. In this essay, we consider processing mechanisms that enable the extraction of event structure from the dynamically unfolding stream of experience. Two proposals emerge from our own and others' recent research on event processing: First, fluent detection of structure within activity streams appears to hinge on knowledge of the predictability relations within those streams. Second, skill at event processing seems to involve harnessing such knowledge of predictability relations to selectively direct attention to information‐rich portions of the activity stream. If this account is correct, individual differences in knowledge and executive skill should influence event‐processing fluency. As well, children's developmental progress in event processing should reveal the telltale impact of increasing knowledge and executive skill. Our hope is that research pursuing these ideas will ultimately make it possible to enhance event‐processing fluency for all, which in turn has the potential to facilitate memory, learning, and social interaction.
-
Identifier
-
etrds0122
-
extracted text
-
Event Processing as an Executive
Enterprise
ROBBIE A. ROSS and DARE A. BALDWIN
Abstract
Actual experience as life unfolds tends to be an ebb and flow of dynamic, multimodal sensations, many of which are fleeting. Yet what is encoded, recalled, and
talked about tends to be events—units of experience that are conceptualized as having both a beginning and an end. In this essay, we consider processing mechanisms
that enable the extraction of event structure from the dynamically unfolding stream
of experience. Two proposals emerge from our own and others’ recent research on
event processing: First, fluent detection of structure within activity streams appears
to hinge on knowledge of the predictability relations within those streams. Second,
skill at event processing seems to involve harnessing such knowledge of predictability relations to selectively direct attention to information-rich portions of the activity
stream. If this account is correct, individual differences in knowledge and executive
skill should influence event-processing fluency. As well, children’s developmental
progress in event processing should reveal the telltale impact of increasing knowledge and executive skill. Our hope is that research pursuing these ideas will ultimately make it possible to enhance event-processing fluency for all, which in turn
has the potential to facilitate memory, learning, and social interaction.
EVENT PROCESSING AS AN EXECUTIVE ENTERPRISE
BANGvooshsploshyowlskitterdripdropdrip.driip..ddrriipp. This rendition
of a genuine recent life occurrence is intended to showcase that experience
as it unfolds across time is dynamic, complex, multimodal, and ephemeral.
As sensory information flows past, we grasp its significance as best we can
and retain whatever possible to guide understanding and future action.
During the course of this ongoing processing, we tend to have the sense
that the stream of information is punctuated by events—coherent units with
identifiable beginnings and endings—that possess inherent structure, such
as sequential, hierarchical, causal, and/or intention/goal structure. In the
example, a feline-enriched personal history might help one to interpret the
dynamic sensory flow as yet another instance of a sensation-seeking cat’s
Emerging Trends in the Social and Behavioral Sciences. Edited by Robert Scott and Stephen Kosslyn.
© 2015 John Wiley & Sons, Inc. ISBN 978-1-118-90077-2.
1
2
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
penchant for knocking over water-filled vases (discrete event 1), with the
inevitable consequences of fright (2), flight (3), and telltale water damage (4).
The particular event structure we derive from the sensory stream of
experience seems to constitute our understanding of that experience.
Increasing evidence indicates that are involved in this ongoing process of
event redescription. These processes appear to include flexible allocation
of attention to information-rich regions within unfolding experience. The
central aims for this chapter are to review such recent insights into the
nature of event-processing mechanisms, and to consider how executive
skills, such as selective attention, seem to guide event processing and may
thus play an integral role in how event processing changes with learning
and development.
As described, unfolding experience tends to be interpreted in terms of
discrete events, despite the fact that change in the world is dynamic and
largely continuous. The event boundaries we identify are not random or
purely idiosyncratic, however. When asked to identify meaningful junctures in a flow of activity, as viewers we display substantial agreement
regarding where events begin and end (Newtson, 1973). When intentional
human action is involved, these boundaries tend to coincide with goal
transitions (Baldwin & Baird, 2001; Zacks, 2004; Zacks & Tversky, 2001).
Identification of segmental structure occurs at multiple levels of scale that
we organize into partonomic hierarchies, with fine-grained, temporally brief
event-elements nested within successively higher order event segments
(Newtson, 1973; Zacks, Tversky & Iyer, 2001). Several sources of evidence
using both behavioral (Hard, Recchia, & Tversky, 2011; Huff, Papenmier, &
Zacks, 2012; Saylor & Baldwin, 2004) and neuroimaging (Zacks et al., 2001,
Zacks, Swallow, Vettel, & MacAvoy, 2006) methods indicate that these event
segmentation processes are automatic aspects of our default processing of
experience.
A new methodology—the dwell-time paradigm (Hard et al., 2011)—shows
particular promise for illuminating mechanisms subserving event processing. In this paradigm, slideshows are constructed by extracting frames at
regular intervals (e.g., 500 ms) from digitized videos of unfolding activity.
Viewers click a computer mouse to advance through these slideshows at their
own pace; the latency between clicks reflects their dwell times to each slide.
Prior work documents systematicity in viewers’ dwell times indicative of
sensitivity to segmental structure within the activity stream (Hard et al., 2011).
Specifically, dwell times are longer to slides coinciding with event boundaries
(as identified by either participants themselves on a subsequent viewing,
or by others) relative to slides depicting within-event content. Dwell-time
patterns also reflect hierarchical organization of segmental structure; that is,
Event Processing as an Executive Enterprise
3
viewers tend to dwell longest on slides depicting higher order event boundaries and progressively more briefly to slides depicting lower level event
boundaries. Moreover, the extent of dwelling on higher order event boundaries predicts recall for the activity stream, demonstrating that segmentation
and/or hierarchical organization processes predict memory for events (Hard
et al., 2011).
The characteristic dwell-time patterns just described emerge for a broad
range of event types, including full-body intentional action (Hard et al., 2011;
Meyer, DeCamp, Hard, Baldwin, & Roy, 2010), fine-grained manual activity
(Sage & Baldwin, 2014), and dynamic facial-emotion displays (e.g., Garrison & Baldwin, 2014). Thus, similar processing strategies seem to guide at
least some aspects of event processing across domains and differences of
scale. Children as young as 2 to 3 years of age display dwell-time patterns
comparable to adults’ (Baldwin & Sage, 2013; Meyer, Baldwin, & Sage, 2011),
at least when children are given relatively familiar, developmentally appropriate event stimuli to view. The fact that children display dwell-time patterns indicative of segmentation and hierarchical organization of segments
both showcases the usefulness of the dwell-time paradigm with children as
well as adults, and points to event processing possessing a similar character,
at least with respect to segmentation phenomena, across a relatively broad
developmental span.
Speaking in general terms, dwell-time patterns appear to index the ongoing modulation of selective attention as activity unfolds across time. But why
does dwelling increase systematically at event boundaries? This is as yet not
entirely clear. Hard and colleagues (2011) have suggested that event boundaries may serve as “conceptual bridges” from one event to the next. If this is
correct, then viewers’ ability to systematically direct attention to boundary
regions within events is an important aspect of fluent event-processing skill.
This proposal—that selective attention to event boundaries is crucial to
fluent event processing—has several implications. First, it suggests that
knowledge or expertise is involved in the fluency of event processing; that
is, without prior knowledge it is unclear how viewers could discern which
portions of the complex, dynamic stream of sensory stimuli are especially
information rich. Second, this proposal suggests that executive function
skill—skill that, among other things, makes it possible to mobilize attention
to selected targets or specific regions within temporal sequences—is central
to fluent event processing. In other words, skill at attention modulation
presumably is required to systematically direct attention to relevant regions
within unfolding sensory streams. Relatedly, we might expect executive
function deficits to impair fluent event processing. Third, the proposal
leads us to expect developmental change in the fluency of event processing,
because knowledge and executive skill both increase substantially during
4
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
childhood. In what follows we discuss available evidence pertinent to
these suggestions, and outline research we are beginning to undertake to
investigate the issues further.
EXPERTISE IN EVENT PROCESSING
The proposal is that viewers spontaneously increase attention to event
boundaries, which are points within unfolding sensory stimuli at which one
segment transitions into the next. An alternative possibility, however, is that
this upregulation of attention occurs simply because event boundaries are
in some inherent way attention attracting. That is, these boundary regions
within unfolding information might naturally draw viewers’ attention
without viewers playing any agentive role in targeting these regions with
selective attention. As we explain below, this alternative account is plausible,
but even at this early phase of investigation the available evidence suggests
that it does not fully capture event-processing phenomena.
Existing evidence suggests that there indeed may be perceptual concomitants of event boundaries that potentially draw viewers’ attention
exogenously. For example, segment boundaries appear to probabilistically
coincide with changes in line-of-regard, the particular object contacted,
directionality of motion, as well as the sheer magnitude of physical motion
change and motion dynamics such as velocity and acceleration parameters
(e.g., Baldwin & Baird, 1999; Hard et al., 2011; Newtson, Enquist, & Bois,
1977; Zacks, 2004; Zacks & Swallow, 2007; Zacks, Swallow, et al., 2006). Any
one, or any combination, of these physical, perceptual occurrences within
unfolding experience might systematically elicit increased attention from
viewers.
Interestingly, however, inherent salience seems not to be the only reason
viewers increase attention to event boundaries. Knowledge appears also
to play a key role. A recent study that married statistical learning with
the dwell-time paradigm provided the first direct evidence to this effect
(Baldwin, Hard, Meyer, & Sage, 2014). The stimuli employed in the research
were novel event sequences specifically designed such that salient perceptual characteristics did not correlate in any way with event boundaries.
Rather, statistical regularities were the only clue to segment boundaries.
Findings from the research revealed that viewers’ dwell times became
reorganized as knowledge of the statistically based segmental structure of
the event sequence grew across time. Once knowledge of the statistics was
gained, viewers displayed the classic segment-related dwell-time patterns
observed in prior dwell-time research: they attended longer to slides
depicting (statistically defined) segment boundaries than to slides depicting
mid-segment content. Before having gained knowledge of the statistics,
Event Processing as an Executive Enterprise
5
viewers’ dwell times lacked this segment-related increase in dwell times.
Strikingly, viewers’ ability to successfully identify intact event segments relative to nonsegment foils in a discrimination task was related to dwell-time
patterns in their prior slide-show viewing: those who displayed a strong
knowledge of segmental structure in the discrimination task displayed
enhanced dwell times at segment boundaries, whereas dwell times were
unrelated to segment boundaries among those who performed poorly on the
segment discrimination task. All in all, these findings clearly showcase that
increasing knowledge of an event sequence, and of its segmental structure
in particular, alters how attention is allocated as viewing of the sequence
unfolds.
Again, one might ask why greater attentional resources appear to be
allocated to regions of a sensory stream where one event ends and the next
begins. Are such regions especially information rich? Zacks and colleagues
(Zacks, Kurby, Eisenberg, & Haroutunian, 2011) have provided some evidence that the culmination points of event segments tend to be junctures
when predictability plummets within the unfolding flow of experience. To
explain: Any number of new activities could follow when an event segment
is completed. For example, when placing a coffee mug onto a table (as at the
end of a taking a sip of coffee segment) it is challenging to predict what will
occur next, as a range of possible event segments could ensue, such as taking
another sip of coffee, taking a bite of food, getting up from the table, pouring more
coffee into a cup, making conversation, and the like. In contrast, just as the next
segment is launched and identified, let us say taking a bite of food, much of
what occurs is highly predictable until the completion of that segment has
been reached. Thus, in information-theoretic terms (e.g., Prokhorov, n.d), the
transition from one segment to the next is, by virtue of its unpredictability,
highly informative. However, a viewer can recognize and selectively attend
to such highly informative junctures within an unfolding sensory stream
only if in possession of at least some prior knowledge of the predictability
structure of the sequence. That is, given some knowledge of the sequence
at hand, segment boundaries become “predictably unpredictable,” and
can be selectively attended to for their heightened information value.
Presumably, this is the basis for the reorganization of attention that occurs
as knowledge about a sequence grows over time, as observed in the study
reported earlier. Consistent with this account, Hard and colleagues (2011)
found that dwell times tend to increase shortly before the end of a segment,
and continue to display elevation for a short period after the next segment
begins; that is, viewers appear to anticipate the occurrence of the predictably
unpredictable, information-rich boundary regions, and continue to attend
at high levels until predictability resumes mid-segment. Such anticipations
were presumably observed in Hard and colleagues’ research because they
6
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
utilized event sequences that were relatively familiar to viewers (e.g., room
cleaning, and the like). To the extent that viewers are radically unfamiliar
with an unfolding sequence, likely they would be unable to selectively
target event boundaries by upregulating attention in anticipation, because
they would not be able to predict when a given event segment was likely to
be completed. This is a prediction that we are examining directly in research
led by Jessica Kosie that is currently under way.
POSSIBLE EXECUTIVE SYSTEM INVOLVEMENT
As described, a key assumption underlying the account offered here for characteristic dwell-time patterns is that viewers systematically (although not
necessarily consciously) direct their attention to information-rich segment
boundaries as events unfold. How might such ongoing attentional modulation be achieved, however? It seems likely that the executive system, which
plays a central role in driving attentional control (e.g., Shallice, 1994), is heavily involved.
The attentional control network is often broadly described as a cognitive
system via which individuals selectively attend to relevant sensory input
and ignore irrelevant information. When observing unfolding events, the
attentional control system may be monitoring the predictability of the event
stream and flexibly modulating the viewer’s attention as predictability
changes. Our existing findings utilizing the dwell-time paradigm seem
to confirm this, as does Zacks and colleagues’ research, described earlier,
showcasing links between adults’ processing of event structure and changes
in event predictability (Zacks et al., 2011). The executive system is likely
upregulating attention to the points in the unfolding sequence that the
viewer expects to be most relevant or information rich and simultaneously
downregulating attention to less relevant or informative parts of the event.
Throughout this complex processing task, the attentional control system
is also incorporating accrued knowledge of the observed predictability
structure of the event into long-term memory, thereby helping to further
enrich event-related knowledge, and facilitating processing of similar event
sequences encountered on future occasions.
On this analysis, deficits in executive function would be expected to be
related to impaired event-processing fluency. Preliminary evidence to this
effect is already available. In a 2006 study, Zacks and colleagues investigated
event segmentation patterns and subsequent memory among adults with
dementia of the Alzheimer’s type, a neurocognitive disorder marked by
impaired memory, attention, and other executive functions (Zacks, Speer,
Vettel & Jacoby, 2006). They found that, in a task in which participants were
asked to watch movies of everyday goal-directed activities and identify
Event Processing as an Executive Enterprise
7
where action boundaries exist, the boundaries identified by aging adults
diagnosed with dementia were markedly discrepant from those identified
by healthy older adults and younger adults, suggesting an impairment in
the ability to fluently identify segmental structure within ongoing events.
In addition, among older adults, the extent to which event segmentation
boundaries agreed with normative boundary selections positively predicted
memory performance.
These findings provide additional evidence that appropriately segmenting ongoing events is crucial for proper encoding and recall. Moreover, they
suggest a rather new conceptualization of the cognitive deficits occurring in
dementia. In particular, they point to difficulties in fluent event processing
arising from executive deficits as central in the cognitive profile of dementia,
and possibly even responsible for the memory deficits typically viewed as
characteristic of the disability. Put another way, memory deficits seen among
older adults with dementia may be due in some significant measure to executive system deficits, rather than to memory problems, per se. That is, deficits
in the attentional control network may undercut effective attentional modulation to the predictability contour of an unfolding event, rendering a poorly
articulated grasp of event structure in those with dementia, and as a result,
poor event encoding.
This interpretation of Zacks and colleagues’ findings predicts other
relations between the executive system and event-processing patterns. For
one, other forms of executive deficit, such as that typically seen in those
with attention-deficit/hyperactivity disorder (ADHD), might also be linked
to event-processing impairments. We are currently undertaking research
to examine this possibility. In particular, we are investigating whether
severity of ADHD symptomatology predicts (i) impairment in identifying
event boundaries in an explicit segmentation task, and (ii) reduction in
implicit dwell-time patterns indicative of sensitivity to the segmental and
hierarchical structure of events. If these predictions are borne out, it would
provide among the first available evidence that fluent event processing is
impaired in ADHD, which in turn may help to account for some of the
learning and memory difficulties known to be associated with ADHD.
DEVELOPMENTS IN EVENT PROCESSING
A classic developmental question is to wonder how newborn infants experience the sensory flow that, as adults, we so effortlessly interpret in terms
of discrete, organized, and meaningful events. Considerable research now
clarifies that newborns demonstrate organized and adaptive perceptual processing (e.g., Maurer & Maurer, 1988), but it is only recently that researchers
began to question whether infants this young experience events as sequences
8
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
comprised of discrete units. As it turns out, basic event segmentation skill
seems to be present in the first year of life (Baldwin, Baird, Saylor, &
Clark, 2001; Hespos, Grossman, & Saylor, 2010; Stahl, Romberg, Roseberry,
Golinkoff, & Hirsh-PasekStahl, 2014), for at least some kinds of simple
human action events. Having said this, it nevertheless also seems obvious
that event processing, including segmentation skill, undergoes considerable
reorganization developmentally, in part because knowledge seems to play
such a central role in shaping it. A 12-month-old infant’s understanding of
a marriage ceremony, or of a presidential bill-signing event, for example,
would have to be very different from that of an educated adult’s, given their
ignorance of the complex social and cultural foundations underlying such
events. Infants might detect fine-grain units within the motion stream with
some degree of fidelity (e.g., grasping a pen, bringing a pen to paper, making
squiggles on the page with a pen) but lack an understanding of the higher
order goals of the actors or what they signify within a larger event context.
Hence, segmentation at higher levels and organization of event segments
according to hierarchical, causal, and intention-goal structure probably
change considerably as knowledge and development progress.
Our working hypothesis is that knowledge change and attentional
reorganization in event processing are deeply interrelated in children’s
developmental progress. As children acquire new information about
predictability relations in the world (knowledge) this will reshape their
attention toward more efficient processing of the sensory flow. Progressively
they will be able to selectively target junctures within unfolding experience
at which relevant new information is predictably to be found (such as event
boundaries, among other possibilities). As such reorganization occurs, event
segments emerge as relatively stable entities in their experience, providing
opportunity for noting similarities across events, which in turn enables
children to construct event categories, learn labels (e.g., verbs) for them,
and draw inferences about causal relationships that event categories enter
into. Buchsbaum and colleagues recently documented portions of this
developmental progression, demonstrating that both adults (Buchsbaum,
Griffiths, Gopnik, & Baldwin, in press) and 3-year-olds (Buchsbaum, Gopnik,
Griffiths, & Shafto, 2011) can use newly gained information (via statistical
learning) about novel event segments to guide their causal inferences, and
conversely, they can also use preexisting knowledge of causal structure to
guide perception of event boundaries.
We are hopeful that the dwell-time paradigm will be a useful technique by
which to further observe developmental change of just this kind as it is under
way. One possible example here might be use of the dwell-time paradigm
to observe developmental progress in conceptual-level knowledge, such as
theory-of-mind understanding. To the extent that infants or young children
Event Processing as an Executive Enterprise
9
either possess (e.g., Baillargeon, Scott, & He, 2010) or lack (e.g., Wellman,
Cross, & Watson, 2001) lack an understanding of mentalistic concepts such
as belief or knowledge, this will directly impact the predictions they have
for how an event sequence will play out, which in turn will likely shape
how they deploy their attention as the event proceeds. An older child or
adult’s understanding that an actor possesses crucial knowledge of a hidden
threat, for example, might lead to enhanced dwelling on threat-relevant junctures within the unfolding stream of activity. In contrast, lacking an ability to
appreciate the actor’s knowledge state should leave infants and younger children oblivious to such threat-relevant junctures, with attendant differences
in their dwell-time patterns. In collaboration with Jessica Kosie and Eric Olofson, research testing these and related predictions is currently under way.
Although knowledge seems to reshape event processing, which in turn
yields developmental progress in event understanding, we suspect that
individual differences in executive function are a key determinant of the
pace of developmental progress in this arena. We (Dare Baldwin and Robbie
Ross) are currently investigating the possibility of such links between
individual differences in children’s developing executive skill and fidelity
of event processing. The preschool period is marked by rapid advances in
executive system development. We predict that children in this age range
will display variability in event segmentation patterns, as measured by
dwell times, that is systematically related to their executive function skill.
Moreover, we suspect that children with relatively poor executive skill will
have difficulty reorganizing event processing in response to changing task
demands. Findings from this research will help clarify whether developing executive skill impacts children’s event-processing fluency, a finding
with potentially wide-ranging implications for understanding individual
differences in learning and cognitive growth.
CONCLUSION
We began with the cacophony of a cat-related catastrophe to illustrate that
life presents itself to us as a multimodal flow of complex, dynamic information that we actively transform, or redescribe, into discrete, organized event
structures. Our own and others’ recent research begins to shed light on the
nature of these redescription processes, and how they emerge in children’s
development. In brief, fluent event processing appears to involve skill at
modulating attention to selectively capture predictable, structurally relevant,
information-rich portions of an unfolding activity stream. Conceptualized
this way, event-processing skill fundamentally depends on knowledge and
executive skill. As knowledge of the world grows, more can be predicted. As
more becomes predictable, it is increasingly possible to selectively project
10
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
where, in the unfolding sensory flow, relevant new information is likely to
be found (the predictably unpredictable). However, these developmental
achievements likely depend critically on skill at flexibly reconfiguring
attention based on knowledge and concurrent task demands, which requires
a high-functioning executive system. Our hope is that gaining increased
understanding of such relations among knowledge, executive skill, and
event-processing fluency will ultimately enable us to enhance such fluency
for all.
REFERENCES
Baillargeon, R., Scott, R. M., & He, Z. (2010). False-belief understanding in infants.
Trends in Cognitive Sciences, 14(3), 110–118.
Baldwin, D. A., & Baird, J. A. (1999). Action analysis: A gateway to intentional inference. In P. Rochat (Ed.), Early social cognition: Understanding others in the first months
of life (pp. 215–240). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Baldwin, D. A., & Baird, J. A. (2001). Discerning intentions in dynamic human action.
Trends in Cognitive Sciences, 5, 171–178.
Baldwin, D. A., Baird, J. A., Saylor, M. M., & Clark, M. A. (2001). Infants parse
dynamic action. Child Development, 72(3), 708–717.
Baldwin, D., Hard, B., Meyer, M., & Sage, K. (2014). Attentional reorganization with
discovery of segmental structure in dynamic human action (Manuscript in preparation).
Eugene, OR: Department of Psychology, University of Oregon.
Baldwin, D., & Sage, K. (2013). Dwelling on action. In M. Rutherford & V. Kuhlmeier
(Eds.), Social perception: Detection and interpretation of animacy, agency, and intention
(pp. 309–330). Cambridge, MA: MIT Press.
Buchsbaum, D., Gopnik, A., Griffiths, T. L., & Shafto, P. (2011). Children’s imitation
of causal action sequences is influenced by statistical and pedagogical evidence.
Cognition, 120(3), 331–340.
Buchsbaum, D., Griffiths, T. L., Gopnik, A., & Baldwin, D. (in press). Learning
from actions and their consequences: Inferring causal variables from continuous
sequences of human action. Cognitive Psychology.
Garrison, J., & Baldwin, D. (2014). Finding emo: Discovering segmental structure in
emotional displays (Manuscript in preparation). Eugene, OR: Department of Psychology, University of Oregon.
Hard, B., Recchia, G., & Tversky, B. (2011). The shape of action. Journal of Experimental
Psychology: General, 140, 586–604.
Hespos, S. J., Grossman, S., & Saylor, M. M. (2010). Infants’ ability to parse continuous
action series: Further evidence. Neural Networks, 23, 1026–1032.
Huff, M., Papenmier, F., & Zacks, J. M. (2012). Visual target detection is impaired at
event boundaries. Visual Cognition, iFirst, 1–17.
Maurer, D., & Maurer, C. (1988). The world of the newborn. New York, NY: Basic Books.
Meyer, M., Baldwin, D. A., & Sage, K. (2011). Assessing young children’s hierarchical
action segmentation. In L. Carlson, C. Hölscher & T. Shipley (Eds.), Proceedings of
Event Processing as an Executive Enterprise
11
the 33rd annual conference of the cognitive science society (pp. 3156–3161). Boston, MA:
Cognitive Science Society.
Meyer, M., DeCamp, P., Hard, B., Baldwin, D., & Roy, D. (2010). Assessing behavioral
and computational approaches to naturalistic action segmentation. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd annual conference of the cognitive
science society (pp. 2710–2715). Portland, OR: Cognitive Science Society.
Newtson, D. (1973). Attribution and the unit of perception of ongoing behavior. Journal of Personality & Social Psychology, 28, 28–38.
Newtson, D., Enquist, G., & Bois, J. (1977). The objective basis of behavior units. Journal of Personality and Social Psychology, 35(12), 847–862.
Prokhorov, Y. V. (n.d). Information. Encyclopedia of Mathematics. Retrieved from
http://www.encyclopediaofmath.org/index.php?title=Information&oldid=
14072.
Sage, K. D., & Baldwin, D. (2014). Finding structure in complex manual sequences
(Manuscript submitted for publication). Eugene, OR: Department of Psychology,
University of Oregon.
Saylor, M., & Baldwin, D. (2004). Action analysis and change blindness: Possible
links. In D. T. Levin (Ed.), Thinking and seeing: Visual metacognition in adults and
children (pp. 37–56). Westport, CT: Greenwood/Praeger.
Shallice, T. (1994). Multiple levels of control processes. Attention and Performance XV,
395–420.
Stahl, A. E., Romberg, A. R., Roseberry, S., Golinkoff, R. N., & Hirsh-Pasek, K. (2014).
Infants segment continuous events using transitional probabilities. Child Development, 85(5), 1821–1826.
Wellman, H. M., Cross, D., & Watson, J. (2001). A meta-analysis of theory of mind
development: The truth about false belief. Child Development, 72, 655–684.
Zacks, J. (2004). Using movement and intentions to understand simple events. Cognitive Science, 28, 979–1008.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z., Ollinger,
J. M., … Raichle, M. E. (2001). Human brain activity time-locked to perceptual
event boundaries. Nature Neuroscience, 4, 651–655.
Zacks, J. M., Kurby, C. A., Eisenberg, M. L., & Haroutunian, N. (2011). Prediction
error associated with the perceptual segmentation of naturalistic events. Journal of
Cognitive Neuroscience, 23(12), 4057–4066.
Zacks, J. M., Speer, N. K., Vettel, J. M., & Jacoby, L. L. (2006). Event understanding
and memory in health aging and dementia of the Alzheimer type. Psychology and
Aging, 21(3), 466–482.
Zacks, J. M., & Swallow, K. M. (2007). Event segmentation. Current Directions in Psychological Science, 16(2), 80–84.
Zacks, J. M., Swallow, K. M., Vettel, J. M., & MacAvoy, M. P. (2006). Visual movement
and the neural correlates of event perception. Brain Research, 1076(1), 15–162.
Zacks, J. M., & Tversky, B. (2001). Event structure in perception and cognition. Psychological Bulletin, 127, 3–21.
Zacks, J. M., Tversky, B., & Iyer, G. (2001). Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130, 29–58.
12
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
ROBBIE A. ROSS SHORT BIOGRAPHY
Robbie A. Ross is a doctoral student in the Psychology Department at the
University of Oregon in Eugene, Oregon. She graduated from Tulane University in 2005 with a bachelor’s degree in psychology and spent several years
working in infancy research. Before joining the UO, she worked in a developmental neuroscience laboratory at Columbia University studying the effects
of perinatal exposures on infant neurodevelopment and cognition. While in
New York, Ross also conducted research at New York University measuring
infants’ understanding of the influence of emotional language on behaviors
in others. Her current research is focused on the role of the executive function
system in learning among preschool and school-aged children. In particular,
she is interested in how executive control is involved in processing dynamic
information streams.
baldwinlab.uoregon.edu
teamduckling.uoregon.edu
psychology.uoregon.edu
DARE A. BALDWIN SHORT BIOGRAPHY
Dare A. Baldwin is Professor of Psychology at the University of Oregon. Her
research focuses on enhancing the human potential for knowledge acquisition. Although in humans knowledge is acquired at a remarkable pace from
earliest infancy on, much remains mysterious about the developmental origins of the underlying mechanisms making this possible. Baldwin seeks to
understand how human infants rapidly and effectively acquire and organize
world knowledge across many domains. Her research has been recognized
by awards such as APA’s Boyd McCandless Award, an APA Distinguished
Scientific Award for Early Career Contribution to Psychology, and a Guggenheim Fellowship. In recent years, Baldwin’s research has been funded by the
National Science Foundation, the Office of Naval Research, and the University of Oregon Fund for Faculty Excellence. A former fellow of the Center
for Advanced Study in the Behavioral Sciences, Baldwin is currently a fellow
of the Association for Psychological Science, and on the executive board of
the Society for Language Development. Baldwin completed undergraduate
work in Psychology at U.C. Berkeley, received a master’s at U.C. Santa Cruz,
and a PhD in Psychology, with a special designation in Cognitive Science, at
Stanford University.
RELATED ESSAYS
Models of Revealed Preference (Economics), Abi Adams and Ian Crawford
Coevolution of Decision-Making and Social Environments (Sociology),
Elizabeth Bruch et al.
Event Processing as an Executive Enterprise
13
Heuristic Decision Making (Political Science), Edward G. Carmines and
Nicholas J. D’Amico
Behavioral Economics (Sociology), Guy Hochman and Dan Ariely
Emotion and Decision Making (Psychology), Jeff R. Huntsinger and Cara Ray
From Individual Rationality to Socially Embedded Self-Regulation (Sociology), Siegwart Lindenberg
Resource Limitations in Visual Cognition (Psychology), Brandon M. Liverence and Steven L. Franconeri
Against Game Theory (Political Science), Gale M. Lucas et al.
-
Event Processing as an Executive
Enterprise
ROBBIE A. ROSS and DARE A. BALDWIN
Abstract
Actual experience as life unfolds tends to be an ebb and flow of dynamic, multimodal sensations, many of which are fleeting. Yet what is encoded, recalled, and
talked about tends to be events—units of experience that are conceptualized as having both a beginning and an end. In this essay, we consider processing mechanisms
that enable the extraction of event structure from the dynamically unfolding stream
of experience. Two proposals emerge from our own and others’ recent research on
event processing: First, fluent detection of structure within activity streams appears
to hinge on knowledge of the predictability relations within those streams. Second,
skill at event processing seems to involve harnessing such knowledge of predictability relations to selectively direct attention to information-rich portions of the activity
stream. If this account is correct, individual differences in knowledge and executive
skill should influence event-processing fluency. As well, children’s developmental
progress in event processing should reveal the telltale impact of increasing knowledge and executive skill. Our hope is that research pursuing these ideas will ultimately make it possible to enhance event-processing fluency for all, which in turn
has the potential to facilitate memory, learning, and social interaction.
EVENT PROCESSING AS AN EXECUTIVE ENTERPRISE
BANGvooshsploshyowlskitterdripdropdrip.driip..ddrriipp. This rendition
of a genuine recent life occurrence is intended to showcase that experience
as it unfolds across time is dynamic, complex, multimodal, and ephemeral.
As sensory information flows past, we grasp its significance as best we can
and retain whatever possible to guide understanding and future action.
During the course of this ongoing processing, we tend to have the sense
that the stream of information is punctuated by events—coherent units with
identifiable beginnings and endings—that possess inherent structure, such
as sequential, hierarchical, causal, and/or intention/goal structure. In the
example, a feline-enriched personal history might help one to interpret the
dynamic sensory flow as yet another instance of a sensation-seeking cat’s
Emerging Trends in the Social and Behavioral Sciences. Edited by Robert Scott and Stephen Kosslyn.
© 2015 John Wiley & Sons, Inc. ISBN 978-1-118-90077-2.
1
2
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
penchant for knocking over water-filled vases (discrete event 1), with the
inevitable consequences of fright (2), flight (3), and telltale water damage (4).
The particular event structure we derive from the sensory stream of
experience seems to constitute our understanding of that experience.
Increasing evidence indicates that are involved in this ongoing process of
event redescription. These processes appear to include flexible allocation
of attention to information-rich regions within unfolding experience. The
central aims for this chapter are to review such recent insights into the
nature of event-processing mechanisms, and to consider how executive
skills, such as selective attention, seem to guide event processing and may
thus play an integral role in how event processing changes with learning
and development.
As described, unfolding experience tends to be interpreted in terms of
discrete events, despite the fact that change in the world is dynamic and
largely continuous. The event boundaries we identify are not random or
purely idiosyncratic, however. When asked to identify meaningful junctures in a flow of activity, as viewers we display substantial agreement
regarding where events begin and end (Newtson, 1973). When intentional
human action is involved, these boundaries tend to coincide with goal
transitions (Baldwin & Baird, 2001; Zacks, 2004; Zacks & Tversky, 2001).
Identification of segmental structure occurs at multiple levels of scale that
we organize into partonomic hierarchies, with fine-grained, temporally brief
event-elements nested within successively higher order event segments
(Newtson, 1973; Zacks, Tversky & Iyer, 2001). Several sources of evidence
using both behavioral (Hard, Recchia, & Tversky, 2011; Huff, Papenmier, &
Zacks, 2012; Saylor & Baldwin, 2004) and neuroimaging (Zacks et al., 2001,
Zacks, Swallow, Vettel, & MacAvoy, 2006) methods indicate that these event
segmentation processes are automatic aspects of our default processing of
experience.
A new methodology—the dwell-time paradigm (Hard et al., 2011)—shows
particular promise for illuminating mechanisms subserving event processing. In this paradigm, slideshows are constructed by extracting frames at
regular intervals (e.g., 500 ms) from digitized videos of unfolding activity.
Viewers click a computer mouse to advance through these slideshows at their
own pace; the latency between clicks reflects their dwell times to each slide.
Prior work documents systematicity in viewers’ dwell times indicative of
sensitivity to segmental structure within the activity stream (Hard et al., 2011).
Specifically, dwell times are longer to slides coinciding with event boundaries
(as identified by either participants themselves on a subsequent viewing,
or by others) relative to slides depicting within-event content. Dwell-time
patterns also reflect hierarchical organization of segmental structure; that is,
Event Processing as an Executive Enterprise
3
viewers tend to dwell longest on slides depicting higher order event boundaries and progressively more briefly to slides depicting lower level event
boundaries. Moreover, the extent of dwelling on higher order event boundaries predicts recall for the activity stream, demonstrating that segmentation
and/or hierarchical organization processes predict memory for events (Hard
et al., 2011).
The characteristic dwell-time patterns just described emerge for a broad
range of event types, including full-body intentional action (Hard et al., 2011;
Meyer, DeCamp, Hard, Baldwin, & Roy, 2010), fine-grained manual activity
(Sage & Baldwin, 2014), and dynamic facial-emotion displays (e.g., Garrison & Baldwin, 2014). Thus, similar processing strategies seem to guide at
least some aspects of event processing across domains and differences of
scale. Children as young as 2 to 3 years of age display dwell-time patterns
comparable to adults’ (Baldwin & Sage, 2013; Meyer, Baldwin, & Sage, 2011),
at least when children are given relatively familiar, developmentally appropriate event stimuli to view. The fact that children display dwell-time patterns indicative of segmentation and hierarchical organization of segments
both showcases the usefulness of the dwell-time paradigm with children as
well as adults, and points to event processing possessing a similar character,
at least with respect to segmentation phenomena, across a relatively broad
developmental span.
Speaking in general terms, dwell-time patterns appear to index the ongoing modulation of selective attention as activity unfolds across time. But why
does dwelling increase systematically at event boundaries? This is as yet not
entirely clear. Hard and colleagues (2011) have suggested that event boundaries may serve as “conceptual bridges” from one event to the next. If this is
correct, then viewers’ ability to systematically direct attention to boundary
regions within events is an important aspect of fluent event-processing skill.
This proposal—that selective attention to event boundaries is crucial to
fluent event processing—has several implications. First, it suggests that
knowledge or expertise is involved in the fluency of event processing; that
is, without prior knowledge it is unclear how viewers could discern which
portions of the complex, dynamic stream of sensory stimuli are especially
information rich. Second, this proposal suggests that executive function
skill—skill that, among other things, makes it possible to mobilize attention
to selected targets or specific regions within temporal sequences—is central
to fluent event processing. In other words, skill at attention modulation
presumably is required to systematically direct attention to relevant regions
within unfolding sensory streams. Relatedly, we might expect executive
function deficits to impair fluent event processing. Third, the proposal
leads us to expect developmental change in the fluency of event processing,
because knowledge and executive skill both increase substantially during
4
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
childhood. In what follows we discuss available evidence pertinent to
these suggestions, and outline research we are beginning to undertake to
investigate the issues further.
EXPERTISE IN EVENT PROCESSING
The proposal is that viewers spontaneously increase attention to event
boundaries, which are points within unfolding sensory stimuli at which one
segment transitions into the next. An alternative possibility, however, is that
this upregulation of attention occurs simply because event boundaries are
in some inherent way attention attracting. That is, these boundary regions
within unfolding information might naturally draw viewers’ attention
without viewers playing any agentive role in targeting these regions with
selective attention. As we explain below, this alternative account is plausible,
but even at this early phase of investigation the available evidence suggests
that it does not fully capture event-processing phenomena.
Existing evidence suggests that there indeed may be perceptual concomitants of event boundaries that potentially draw viewers’ attention
exogenously. For example, segment boundaries appear to probabilistically
coincide with changes in line-of-regard, the particular object contacted,
directionality of motion, as well as the sheer magnitude of physical motion
change and motion dynamics such as velocity and acceleration parameters
(e.g., Baldwin & Baird, 1999; Hard et al., 2011; Newtson, Enquist, & Bois,
1977; Zacks, 2004; Zacks & Swallow, 2007; Zacks, Swallow, et al., 2006). Any
one, or any combination, of these physical, perceptual occurrences within
unfolding experience might systematically elicit increased attention from
viewers.
Interestingly, however, inherent salience seems not to be the only reason
viewers increase attention to event boundaries. Knowledge appears also
to play a key role. A recent study that married statistical learning with
the dwell-time paradigm provided the first direct evidence to this effect
(Baldwin, Hard, Meyer, & Sage, 2014). The stimuli employed in the research
were novel event sequences specifically designed such that salient perceptual characteristics did not correlate in any way with event boundaries.
Rather, statistical regularities were the only clue to segment boundaries.
Findings from the research revealed that viewers’ dwell times became
reorganized as knowledge of the statistically based segmental structure of
the event sequence grew across time. Once knowledge of the statistics was
gained, viewers displayed the classic segment-related dwell-time patterns
observed in prior dwell-time research: they attended longer to slides
depicting (statistically defined) segment boundaries than to slides depicting
mid-segment content. Before having gained knowledge of the statistics,
Event Processing as an Executive Enterprise
5
viewers’ dwell times lacked this segment-related increase in dwell times.
Strikingly, viewers’ ability to successfully identify intact event segments relative to nonsegment foils in a discrimination task was related to dwell-time
patterns in their prior slide-show viewing: those who displayed a strong
knowledge of segmental structure in the discrimination task displayed
enhanced dwell times at segment boundaries, whereas dwell times were
unrelated to segment boundaries among those who performed poorly on the
segment discrimination task. All in all, these findings clearly showcase that
increasing knowledge of an event sequence, and of its segmental structure
in particular, alters how attention is allocated as viewing of the sequence
unfolds.
Again, one might ask why greater attentional resources appear to be
allocated to regions of a sensory stream where one event ends and the next
begins. Are such regions especially information rich? Zacks and colleagues
(Zacks, Kurby, Eisenberg, & Haroutunian, 2011) have provided some evidence that the culmination points of event segments tend to be junctures
when predictability plummets within the unfolding flow of experience. To
explain: Any number of new activities could follow when an event segment
is completed. For example, when placing a coffee mug onto a table (as at the
end of a taking a sip of coffee segment) it is challenging to predict what will
occur next, as a range of possible event segments could ensue, such as taking
another sip of coffee, taking a bite of food, getting up from the table, pouring more
coffee into a cup, making conversation, and the like. In contrast, just as the next
segment is launched and identified, let us say taking a bite of food, much of
what occurs is highly predictable until the completion of that segment has
been reached. Thus, in information-theoretic terms (e.g., Prokhorov, n.d), the
transition from one segment to the next is, by virtue of its unpredictability,
highly informative. However, a viewer can recognize and selectively attend
to such highly informative junctures within an unfolding sensory stream
only if in possession of at least some prior knowledge of the predictability
structure of the sequence. That is, given some knowledge of the sequence
at hand, segment boundaries become “predictably unpredictable,” and
can be selectively attended to for their heightened information value.
Presumably, this is the basis for the reorganization of attention that occurs
as knowledge about a sequence grows over time, as observed in the study
reported earlier. Consistent with this account, Hard and colleagues (2011)
found that dwell times tend to increase shortly before the end of a segment,
and continue to display elevation for a short period after the next segment
begins; that is, viewers appear to anticipate the occurrence of the predictably
unpredictable, information-rich boundary regions, and continue to attend
at high levels until predictability resumes mid-segment. Such anticipations
were presumably observed in Hard and colleagues’ research because they
6
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
utilized event sequences that were relatively familiar to viewers (e.g., room
cleaning, and the like). To the extent that viewers are radically unfamiliar
with an unfolding sequence, likely they would be unable to selectively
target event boundaries by upregulating attention in anticipation, because
they would not be able to predict when a given event segment was likely to
be completed. This is a prediction that we are examining directly in research
led by Jessica Kosie that is currently under way.
POSSIBLE EXECUTIVE SYSTEM INVOLVEMENT
As described, a key assumption underlying the account offered here for characteristic dwell-time patterns is that viewers systematically (although not
necessarily consciously) direct their attention to information-rich segment
boundaries as events unfold. How might such ongoing attentional modulation be achieved, however? It seems likely that the executive system, which
plays a central role in driving attentional control (e.g., Shallice, 1994), is heavily involved.
The attentional control network is often broadly described as a cognitive
system via which individuals selectively attend to relevant sensory input
and ignore irrelevant information. When observing unfolding events, the
attentional control system may be monitoring the predictability of the event
stream and flexibly modulating the viewer’s attention as predictability
changes. Our existing findings utilizing the dwell-time paradigm seem
to confirm this, as does Zacks and colleagues’ research, described earlier,
showcasing links between adults’ processing of event structure and changes
in event predictability (Zacks et al., 2011). The executive system is likely
upregulating attention to the points in the unfolding sequence that the
viewer expects to be most relevant or information rich and simultaneously
downregulating attention to less relevant or informative parts of the event.
Throughout this complex processing task, the attentional control system
is also incorporating accrued knowledge of the observed predictability
structure of the event into long-term memory, thereby helping to further
enrich event-related knowledge, and facilitating processing of similar event
sequences encountered on future occasions.
On this analysis, deficits in executive function would be expected to be
related to impaired event-processing fluency. Preliminary evidence to this
effect is already available. In a 2006 study, Zacks and colleagues investigated
event segmentation patterns and subsequent memory among adults with
dementia of the Alzheimer’s type, a neurocognitive disorder marked by
impaired memory, attention, and other executive functions (Zacks, Speer,
Vettel & Jacoby, 2006). They found that, in a task in which participants were
asked to watch movies of everyday goal-directed activities and identify
Event Processing as an Executive Enterprise
7
where action boundaries exist, the boundaries identified by aging adults
diagnosed with dementia were markedly discrepant from those identified
by healthy older adults and younger adults, suggesting an impairment in
the ability to fluently identify segmental structure within ongoing events.
In addition, among older adults, the extent to which event segmentation
boundaries agreed with normative boundary selections positively predicted
memory performance.
These findings provide additional evidence that appropriately segmenting ongoing events is crucial for proper encoding and recall. Moreover, they
suggest a rather new conceptualization of the cognitive deficits occurring in
dementia. In particular, they point to difficulties in fluent event processing
arising from executive deficits as central in the cognitive profile of dementia,
and possibly even responsible for the memory deficits typically viewed as
characteristic of the disability. Put another way, memory deficits seen among
older adults with dementia may be due in some significant measure to executive system deficits, rather than to memory problems, per se. That is, deficits
in the attentional control network may undercut effective attentional modulation to the predictability contour of an unfolding event, rendering a poorly
articulated grasp of event structure in those with dementia, and as a result,
poor event encoding.
This interpretation of Zacks and colleagues’ findings predicts other
relations between the executive system and event-processing patterns. For
one, other forms of executive deficit, such as that typically seen in those
with attention-deficit/hyperactivity disorder (ADHD), might also be linked
to event-processing impairments. We are currently undertaking research
to examine this possibility. In particular, we are investigating whether
severity of ADHD symptomatology predicts (i) impairment in identifying
event boundaries in an explicit segmentation task, and (ii) reduction in
implicit dwell-time patterns indicative of sensitivity to the segmental and
hierarchical structure of events. If these predictions are borne out, it would
provide among the first available evidence that fluent event processing is
impaired in ADHD, which in turn may help to account for some of the
learning and memory difficulties known to be associated with ADHD.
DEVELOPMENTS IN EVENT PROCESSING
A classic developmental question is to wonder how newborn infants experience the sensory flow that, as adults, we so effortlessly interpret in terms
of discrete, organized, and meaningful events. Considerable research now
clarifies that newborns demonstrate organized and adaptive perceptual processing (e.g., Maurer & Maurer, 1988), but it is only recently that researchers
began to question whether infants this young experience events as sequences
8
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
comprised of discrete units. As it turns out, basic event segmentation skill
seems to be present in the first year of life (Baldwin, Baird, Saylor, &
Clark, 2001; Hespos, Grossman, & Saylor, 2010; Stahl, Romberg, Roseberry,
Golinkoff, & Hirsh-PasekStahl, 2014), for at least some kinds of simple
human action events. Having said this, it nevertheless also seems obvious
that event processing, including segmentation skill, undergoes considerable
reorganization developmentally, in part because knowledge seems to play
such a central role in shaping it. A 12-month-old infant’s understanding of
a marriage ceremony, or of a presidential bill-signing event, for example,
would have to be very different from that of an educated adult’s, given their
ignorance of the complex social and cultural foundations underlying such
events. Infants might detect fine-grain units within the motion stream with
some degree of fidelity (e.g., grasping a pen, bringing a pen to paper, making
squiggles on the page with a pen) but lack an understanding of the higher
order goals of the actors or what they signify within a larger event context.
Hence, segmentation at higher levels and organization of event segments
according to hierarchical, causal, and intention-goal structure probably
change considerably as knowledge and development progress.
Our working hypothesis is that knowledge change and attentional
reorganization in event processing are deeply interrelated in children’s
developmental progress. As children acquire new information about
predictability relations in the world (knowledge) this will reshape their
attention toward more efficient processing of the sensory flow. Progressively
they will be able to selectively target junctures within unfolding experience
at which relevant new information is predictably to be found (such as event
boundaries, among other possibilities). As such reorganization occurs, event
segments emerge as relatively stable entities in their experience, providing
opportunity for noting similarities across events, which in turn enables
children to construct event categories, learn labels (e.g., verbs) for them,
and draw inferences about causal relationships that event categories enter
into. Buchsbaum and colleagues recently documented portions of this
developmental progression, demonstrating that both adults (Buchsbaum,
Griffiths, Gopnik, & Baldwin, in press) and 3-year-olds (Buchsbaum, Gopnik,
Griffiths, & Shafto, 2011) can use newly gained information (via statistical
learning) about novel event segments to guide their causal inferences, and
conversely, they can also use preexisting knowledge of causal structure to
guide perception of event boundaries.
We are hopeful that the dwell-time paradigm will be a useful technique by
which to further observe developmental change of just this kind as it is under
way. One possible example here might be use of the dwell-time paradigm
to observe developmental progress in conceptual-level knowledge, such as
theory-of-mind understanding. To the extent that infants or young children
Event Processing as an Executive Enterprise
9
either possess (e.g., Baillargeon, Scott, & He, 2010) or lack (e.g., Wellman,
Cross, & Watson, 2001) lack an understanding of mentalistic concepts such
as belief or knowledge, this will directly impact the predictions they have
for how an event sequence will play out, which in turn will likely shape
how they deploy their attention as the event proceeds. An older child or
adult’s understanding that an actor possesses crucial knowledge of a hidden
threat, for example, might lead to enhanced dwelling on threat-relevant junctures within the unfolding stream of activity. In contrast, lacking an ability to
appreciate the actor’s knowledge state should leave infants and younger children oblivious to such threat-relevant junctures, with attendant differences
in their dwell-time patterns. In collaboration with Jessica Kosie and Eric Olofson, research testing these and related predictions is currently under way.
Although knowledge seems to reshape event processing, which in turn
yields developmental progress in event understanding, we suspect that
individual differences in executive function are a key determinant of the
pace of developmental progress in this arena. We (Dare Baldwin and Robbie
Ross) are currently investigating the possibility of such links between
individual differences in children’s developing executive skill and fidelity
of event processing. The preschool period is marked by rapid advances in
executive system development. We predict that children in this age range
will display variability in event segmentation patterns, as measured by
dwell times, that is systematically related to their executive function skill.
Moreover, we suspect that children with relatively poor executive skill will
have difficulty reorganizing event processing in response to changing task
demands. Findings from this research will help clarify whether developing executive skill impacts children’s event-processing fluency, a finding
with potentially wide-ranging implications for understanding individual
differences in learning and cognitive growth.
CONCLUSION
We began with the cacophony of a cat-related catastrophe to illustrate that
life presents itself to us as a multimodal flow of complex, dynamic information that we actively transform, or redescribe, into discrete, organized event
structures. Our own and others’ recent research begins to shed light on the
nature of these redescription processes, and how they emerge in children’s
development. In brief, fluent event processing appears to involve skill at
modulating attention to selectively capture predictable, structurally relevant,
information-rich portions of an unfolding activity stream. Conceptualized
this way, event-processing skill fundamentally depends on knowledge and
executive skill. As knowledge of the world grows, more can be predicted. As
more becomes predictable, it is increasingly possible to selectively project
10
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
where, in the unfolding sensory flow, relevant new information is likely to
be found (the predictably unpredictable). However, these developmental
achievements likely depend critically on skill at flexibly reconfiguring
attention based on knowledge and concurrent task demands, which requires
a high-functioning executive system. Our hope is that gaining increased
understanding of such relations among knowledge, executive skill, and
event-processing fluency will ultimately enable us to enhance such fluency
for all.
REFERENCES
Baillargeon, R., Scott, R. M., & He, Z. (2010). False-belief understanding in infants.
Trends in Cognitive Sciences, 14(3), 110–118.
Baldwin, D. A., & Baird, J. A. (1999). Action analysis: A gateway to intentional inference. In P. Rochat (Ed.), Early social cognition: Understanding others in the first months
of life (pp. 215–240). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Baldwin, D. A., & Baird, J. A. (2001). Discerning intentions in dynamic human action.
Trends in Cognitive Sciences, 5, 171–178.
Baldwin, D. A., Baird, J. A., Saylor, M. M., & Clark, M. A. (2001). Infants parse
dynamic action. Child Development, 72(3), 708–717.
Baldwin, D., Hard, B., Meyer, M., & Sage, K. (2014). Attentional reorganization with
discovery of segmental structure in dynamic human action (Manuscript in preparation).
Eugene, OR: Department of Psychology, University of Oregon.
Baldwin, D., & Sage, K. (2013). Dwelling on action. In M. Rutherford & V. Kuhlmeier
(Eds.), Social perception: Detection and interpretation of animacy, agency, and intention
(pp. 309–330). Cambridge, MA: MIT Press.
Buchsbaum, D., Gopnik, A., Griffiths, T. L., & Shafto, P. (2011). Children’s imitation
of causal action sequences is influenced by statistical and pedagogical evidence.
Cognition, 120(3), 331–340.
Buchsbaum, D., Griffiths, T. L., Gopnik, A., & Baldwin, D. (in press). Learning
from actions and their consequences: Inferring causal variables from continuous
sequences of human action. Cognitive Psychology.
Garrison, J., & Baldwin, D. (2014). Finding emo: Discovering segmental structure in
emotional displays (Manuscript in preparation). Eugene, OR: Department of Psychology, University of Oregon.
Hard, B., Recchia, G., & Tversky, B. (2011). The shape of action. Journal of Experimental
Psychology: General, 140, 586–604.
Hespos, S. J., Grossman, S., & Saylor, M. M. (2010). Infants’ ability to parse continuous
action series: Further evidence. Neural Networks, 23, 1026–1032.
Huff, M., Papenmier, F., & Zacks, J. M. (2012). Visual target detection is impaired at
event boundaries. Visual Cognition, iFirst, 1–17.
Maurer, D., & Maurer, C. (1988). The world of the newborn. New York, NY: Basic Books.
Meyer, M., Baldwin, D. A., & Sage, K. (2011). Assessing young children’s hierarchical
action segmentation. In L. Carlson, C. Hölscher & T. Shipley (Eds.), Proceedings of
Event Processing as an Executive Enterprise
11
the 33rd annual conference of the cognitive science society (pp. 3156–3161). Boston, MA:
Cognitive Science Society.
Meyer, M., DeCamp, P., Hard, B., Baldwin, D., & Roy, D. (2010). Assessing behavioral
and computational approaches to naturalistic action segmentation. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd annual conference of the cognitive
science society (pp. 2710–2715). Portland, OR: Cognitive Science Society.
Newtson, D. (1973). Attribution and the unit of perception of ongoing behavior. Journal of Personality & Social Psychology, 28, 28–38.
Newtson, D., Enquist, G., & Bois, J. (1977). The objective basis of behavior units. Journal of Personality and Social Psychology, 35(12), 847–862.
Prokhorov, Y. V. (n.d). Information. Encyclopedia of Mathematics. Retrieved from
http://www.encyclopediaofmath.org/index.php?title=Information&oldid=
14072.
Sage, K. D., & Baldwin, D. (2014). Finding structure in complex manual sequences
(Manuscript submitted for publication). Eugene, OR: Department of Psychology,
University of Oregon.
Saylor, M., & Baldwin, D. (2004). Action analysis and change blindness: Possible
links. In D. T. Levin (Ed.), Thinking and seeing: Visual metacognition in adults and
children (pp. 37–56). Westport, CT: Greenwood/Praeger.
Shallice, T. (1994). Multiple levels of control processes. Attention and Performance XV,
395–420.
Stahl, A. E., Romberg, A. R., Roseberry, S., Golinkoff, R. N., & Hirsh-Pasek, K. (2014).
Infants segment continuous events using transitional probabilities. Child Development, 85(5), 1821–1826.
Wellman, H. M., Cross, D., & Watson, J. (2001). A meta-analysis of theory of mind
development: The truth about false belief. Child Development, 72, 655–684.
Zacks, J. (2004). Using movement and intentions to understand simple events. Cognitive Science, 28, 979–1008.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z., Ollinger,
J. M., … Raichle, M. E. (2001). Human brain activity time-locked to perceptual
event boundaries. Nature Neuroscience, 4, 651–655.
Zacks, J. M., Kurby, C. A., Eisenberg, M. L., & Haroutunian, N. (2011). Prediction
error associated with the perceptual segmentation of naturalistic events. Journal of
Cognitive Neuroscience, 23(12), 4057–4066.
Zacks, J. M., Speer, N. K., Vettel, J. M., & Jacoby, L. L. (2006). Event understanding
and memory in health aging and dementia of the Alzheimer type. Psychology and
Aging, 21(3), 466–482.
Zacks, J. M., & Swallow, K. M. (2007). Event segmentation. Current Directions in Psychological Science, 16(2), 80–84.
Zacks, J. M., Swallow, K. M., Vettel, J. M., & MacAvoy, M. P. (2006). Visual movement
and the neural correlates of event perception. Brain Research, 1076(1), 15–162.
Zacks, J. M., & Tversky, B. (2001). Event structure in perception and cognition. Psychological Bulletin, 127, 3–21.
Zacks, J. M., Tversky, B., & Iyer, G. (2001). Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130, 29–58.
12
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
ROBBIE A. ROSS SHORT BIOGRAPHY
Robbie A. Ross is a doctoral student in the Psychology Department at the
University of Oregon in Eugene, Oregon. She graduated from Tulane University in 2005 with a bachelor’s degree in psychology and spent several years
working in infancy research. Before joining the UO, she worked in a developmental neuroscience laboratory at Columbia University studying the effects
of perinatal exposures on infant neurodevelopment and cognition. While in
New York, Ross also conducted research at New York University measuring
infants’ understanding of the influence of emotional language on behaviors
in others. Her current research is focused on the role of the executive function
system in learning among preschool and school-aged children. In particular,
she is interested in how executive control is involved in processing dynamic
information streams.
baldwinlab.uoregon.edu
teamduckling.uoregon.edu
psychology.uoregon.edu
DARE A. BALDWIN SHORT BIOGRAPHY
Dare A. Baldwin is Professor of Psychology at the University of Oregon. Her
research focuses on enhancing the human potential for knowledge acquisition. Although in humans knowledge is acquired at a remarkable pace from
earliest infancy on, much remains mysterious about the developmental origins of the underlying mechanisms making this possible. Baldwin seeks to
understand how human infants rapidly and effectively acquire and organize
world knowledge across many domains. Her research has been recognized
by awards such as APA’s Boyd McCandless Award, an APA Distinguished
Scientific Award for Early Career Contribution to Psychology, and a Guggenheim Fellowship. In recent years, Baldwin’s research has been funded by the
National Science Foundation, the Office of Naval Research, and the University of Oregon Fund for Faculty Excellence. A former fellow of the Center
for Advanced Study in the Behavioral Sciences, Baldwin is currently a fellow
of the Association for Psychological Science, and on the executive board of
the Society for Language Development. Baldwin completed undergraduate
work in Psychology at U.C. Berkeley, received a master’s at U.C. Santa Cruz,
and a PhD in Psychology, with a special designation in Cognitive Science, at
Stanford University.
RELATED ESSAYS
Models of Revealed Preference (Economics), Abi Adams and Ian Crawford
Coevolution of Decision-Making and Social Environments (Sociology),
Elizabeth Bruch et al.
Event Processing as an Executive Enterprise
13
Heuristic Decision Making (Political Science), Edward G. Carmines and
Nicholas J. D’Amico
Behavioral Economics (Sociology), Guy Hochman and Dan Ariely
Emotion and Decision Making (Psychology), Jeff R. Huntsinger and Cara Ray
From Individual Rationality to Socially Embedded Self-Regulation (Sociology), Siegwart Lindenberg
Resource Limitations in Visual Cognition (Psychology), Brandon M. Liverence and Steven L. Franconeri
Against Game Theory (Political Science), Gale M. Lucas et al.
Event Processing as an Executive
Enterprise
ROBBIE A. ROSS and DARE A. BALDWIN
Abstract
Actual experience as life unfolds tends to be an ebb and flow of dynamic, multimodal sensations, many of which are fleeting. Yet what is encoded, recalled, and
talked about tends to be events—units of experience that are conceptualized as having both a beginning and an end. In this essay, we consider processing mechanisms
that enable the extraction of event structure from the dynamically unfolding stream
of experience. Two proposals emerge from our own and others’ recent research on
event processing: First, fluent detection of structure within activity streams appears
to hinge on knowledge of the predictability relations within those streams. Second,
skill at event processing seems to involve harnessing such knowledge of predictability relations to selectively direct attention to information-rich portions of the activity
stream. If this account is correct, individual differences in knowledge and executive
skill should influence event-processing fluency. As well, children’s developmental
progress in event processing should reveal the telltale impact of increasing knowledge and executive skill. Our hope is that research pursuing these ideas will ultimately make it possible to enhance event-processing fluency for all, which in turn
has the potential to facilitate memory, learning, and social interaction.
EVENT PROCESSING AS AN EXECUTIVE ENTERPRISE
BANGvooshsploshyowlskitterdripdropdrip.driip..ddrriipp. This rendition
of a genuine recent life occurrence is intended to showcase that experience
as it unfolds across time is dynamic, complex, multimodal, and ephemeral.
As sensory information flows past, we grasp its significance as best we can
and retain whatever possible to guide understanding and future action.
During the course of this ongoing processing, we tend to have the sense
that the stream of information is punctuated by events—coherent units with
identifiable beginnings and endings—that possess inherent structure, such
as sequential, hierarchical, causal, and/or intention/goal structure. In the
example, a feline-enriched personal history might help one to interpret the
dynamic sensory flow as yet another instance of a sensation-seeking cat’s
Emerging Trends in the Social and Behavioral Sciences. Edited by Robert Scott and Stephen Kosslyn.
© 2015 John Wiley & Sons, Inc. ISBN 978-1-118-90077-2.
1
2
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
penchant for knocking over water-filled vases (discrete event 1), with the
inevitable consequences of fright (2), flight (3), and telltale water damage (4).
The particular event structure we derive from the sensory stream of
experience seems to constitute our understanding of that experience.
Increasing evidence indicates that are involved in this ongoing process of
event redescription. These processes appear to include flexible allocation
of attention to information-rich regions within unfolding experience. The
central aims for this chapter are to review such recent insights into the
nature of event-processing mechanisms, and to consider how executive
skills, such as selective attention, seem to guide event processing and may
thus play an integral role in how event processing changes with learning
and development.
As described, unfolding experience tends to be interpreted in terms of
discrete events, despite the fact that change in the world is dynamic and
largely continuous. The event boundaries we identify are not random or
purely idiosyncratic, however. When asked to identify meaningful junctures in a flow of activity, as viewers we display substantial agreement
regarding where events begin and end (Newtson, 1973). When intentional
human action is involved, these boundaries tend to coincide with goal
transitions (Baldwin & Baird, 2001; Zacks, 2004; Zacks & Tversky, 2001).
Identification of segmental structure occurs at multiple levels of scale that
we organize into partonomic hierarchies, with fine-grained, temporally brief
event-elements nested within successively higher order event segments
(Newtson, 1973; Zacks, Tversky & Iyer, 2001). Several sources of evidence
using both behavioral (Hard, Recchia, & Tversky, 2011; Huff, Papenmier, &
Zacks, 2012; Saylor & Baldwin, 2004) and neuroimaging (Zacks et al., 2001,
Zacks, Swallow, Vettel, & MacAvoy, 2006) methods indicate that these event
segmentation processes are automatic aspects of our default processing of
experience.
A new methodology—the dwell-time paradigm (Hard et al., 2011)—shows
particular promise for illuminating mechanisms subserving event processing. In this paradigm, slideshows are constructed by extracting frames at
regular intervals (e.g., 500 ms) from digitized videos of unfolding activity.
Viewers click a computer mouse to advance through these slideshows at their
own pace; the latency between clicks reflects their dwell times to each slide.
Prior work documents systematicity in viewers’ dwell times indicative of
sensitivity to segmental structure within the activity stream (Hard et al., 2011).
Specifically, dwell times are longer to slides coinciding with event boundaries
(as identified by either participants themselves on a subsequent viewing,
or by others) relative to slides depicting within-event content. Dwell-time
patterns also reflect hierarchical organization of segmental structure; that is,
Event Processing as an Executive Enterprise
3
viewers tend to dwell longest on slides depicting higher order event boundaries and progressively more briefly to slides depicting lower level event
boundaries. Moreover, the extent of dwelling on higher order event boundaries predicts recall for the activity stream, demonstrating that segmentation
and/or hierarchical organization processes predict memory for events (Hard
et al., 2011).
The characteristic dwell-time patterns just described emerge for a broad
range of event types, including full-body intentional action (Hard et al., 2011;
Meyer, DeCamp, Hard, Baldwin, & Roy, 2010), fine-grained manual activity
(Sage & Baldwin, 2014), and dynamic facial-emotion displays (e.g., Garrison & Baldwin, 2014). Thus, similar processing strategies seem to guide at
least some aspects of event processing across domains and differences of
scale. Children as young as 2 to 3 years of age display dwell-time patterns
comparable to adults’ (Baldwin & Sage, 2013; Meyer, Baldwin, & Sage, 2011),
at least when children are given relatively familiar, developmentally appropriate event stimuli to view. The fact that children display dwell-time patterns indicative of segmentation and hierarchical organization of segments
both showcases the usefulness of the dwell-time paradigm with children as
well as adults, and points to event processing possessing a similar character,
at least with respect to segmentation phenomena, across a relatively broad
developmental span.
Speaking in general terms, dwell-time patterns appear to index the ongoing modulation of selective attention as activity unfolds across time. But why
does dwelling increase systematically at event boundaries? This is as yet not
entirely clear. Hard and colleagues (2011) have suggested that event boundaries may serve as “conceptual bridges” from one event to the next. If this is
correct, then viewers’ ability to systematically direct attention to boundary
regions within events is an important aspect of fluent event-processing skill.
This proposal—that selective attention to event boundaries is crucial to
fluent event processing—has several implications. First, it suggests that
knowledge or expertise is involved in the fluency of event processing; that
is, without prior knowledge it is unclear how viewers could discern which
portions of the complex, dynamic stream of sensory stimuli are especially
information rich. Second, this proposal suggests that executive function
skill—skill that, among other things, makes it possible to mobilize attention
to selected targets or specific regions within temporal sequences—is central
to fluent event processing. In other words, skill at attention modulation
presumably is required to systematically direct attention to relevant regions
within unfolding sensory streams. Relatedly, we might expect executive
function deficits to impair fluent event processing. Third, the proposal
leads us to expect developmental change in the fluency of event processing,
because knowledge and executive skill both increase substantially during
4
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
childhood. In what follows we discuss available evidence pertinent to
these suggestions, and outline research we are beginning to undertake to
investigate the issues further.
EXPERTISE IN EVENT PROCESSING
The proposal is that viewers spontaneously increase attention to event
boundaries, which are points within unfolding sensory stimuli at which one
segment transitions into the next. An alternative possibility, however, is that
this upregulation of attention occurs simply because event boundaries are
in some inherent way attention attracting. That is, these boundary regions
within unfolding information might naturally draw viewers’ attention
without viewers playing any agentive role in targeting these regions with
selective attention. As we explain below, this alternative account is plausible,
but even at this early phase of investigation the available evidence suggests
that it does not fully capture event-processing phenomena.
Existing evidence suggests that there indeed may be perceptual concomitants of event boundaries that potentially draw viewers’ attention
exogenously. For example, segment boundaries appear to probabilistically
coincide with changes in line-of-regard, the particular object contacted,
directionality of motion, as well as the sheer magnitude of physical motion
change and motion dynamics such as velocity and acceleration parameters
(e.g., Baldwin & Baird, 1999; Hard et al., 2011; Newtson, Enquist, & Bois,
1977; Zacks, 2004; Zacks & Swallow, 2007; Zacks, Swallow, et al., 2006). Any
one, or any combination, of these physical, perceptual occurrences within
unfolding experience might systematically elicit increased attention from
viewers.
Interestingly, however, inherent salience seems not to be the only reason
viewers increase attention to event boundaries. Knowledge appears also
to play a key role. A recent study that married statistical learning with
the dwell-time paradigm provided the first direct evidence to this effect
(Baldwin, Hard, Meyer, & Sage, 2014). The stimuli employed in the research
were novel event sequences specifically designed such that salient perceptual characteristics did not correlate in any way with event boundaries.
Rather, statistical regularities were the only clue to segment boundaries.
Findings from the research revealed that viewers’ dwell times became
reorganized as knowledge of the statistically based segmental structure of
the event sequence grew across time. Once knowledge of the statistics was
gained, viewers displayed the classic segment-related dwell-time patterns
observed in prior dwell-time research: they attended longer to slides
depicting (statistically defined) segment boundaries than to slides depicting
mid-segment content. Before having gained knowledge of the statistics,
Event Processing as an Executive Enterprise
5
viewers’ dwell times lacked this segment-related increase in dwell times.
Strikingly, viewers’ ability to successfully identify intact event segments relative to nonsegment foils in a discrimination task was related to dwell-time
patterns in their prior slide-show viewing: those who displayed a strong
knowledge of segmental structure in the discrimination task displayed
enhanced dwell times at segment boundaries, whereas dwell times were
unrelated to segment boundaries among those who performed poorly on the
segment discrimination task. All in all, these findings clearly showcase that
increasing knowledge of an event sequence, and of its segmental structure
in particular, alters how attention is allocated as viewing of the sequence
unfolds.
Again, one might ask why greater attentional resources appear to be
allocated to regions of a sensory stream where one event ends and the next
begins. Are such regions especially information rich? Zacks and colleagues
(Zacks, Kurby, Eisenberg, & Haroutunian, 2011) have provided some evidence that the culmination points of event segments tend to be junctures
when predictability plummets within the unfolding flow of experience. To
explain: Any number of new activities could follow when an event segment
is completed. For example, when placing a coffee mug onto a table (as at the
end of a taking a sip of coffee segment) it is challenging to predict what will
occur next, as a range of possible event segments could ensue, such as taking
another sip of coffee, taking a bite of food, getting up from the table, pouring more
coffee into a cup, making conversation, and the like. In contrast, just as the next
segment is launched and identified, let us say taking a bite of food, much of
what occurs is highly predictable until the completion of that segment has
been reached. Thus, in information-theoretic terms (e.g., Prokhorov, n.d), the
transition from one segment to the next is, by virtue of its unpredictability,
highly informative. However, a viewer can recognize and selectively attend
to such highly informative junctures within an unfolding sensory stream
only if in possession of at least some prior knowledge of the predictability
structure of the sequence. That is, given some knowledge of the sequence
at hand, segment boundaries become “predictably unpredictable,” and
can be selectively attended to for their heightened information value.
Presumably, this is the basis for the reorganization of attention that occurs
as knowledge about a sequence grows over time, as observed in the study
reported earlier. Consistent with this account, Hard and colleagues (2011)
found that dwell times tend to increase shortly before the end of a segment,
and continue to display elevation for a short period after the next segment
begins; that is, viewers appear to anticipate the occurrence of the predictably
unpredictable, information-rich boundary regions, and continue to attend
at high levels until predictability resumes mid-segment. Such anticipations
were presumably observed in Hard and colleagues’ research because they
6
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
utilized event sequences that were relatively familiar to viewers (e.g., room
cleaning, and the like). To the extent that viewers are radically unfamiliar
with an unfolding sequence, likely they would be unable to selectively
target event boundaries by upregulating attention in anticipation, because
they would not be able to predict when a given event segment was likely to
be completed. This is a prediction that we are examining directly in research
led by Jessica Kosie that is currently under way.
POSSIBLE EXECUTIVE SYSTEM INVOLVEMENT
As described, a key assumption underlying the account offered here for characteristic dwell-time patterns is that viewers systematically (although not
necessarily consciously) direct their attention to information-rich segment
boundaries as events unfold. How might such ongoing attentional modulation be achieved, however? It seems likely that the executive system, which
plays a central role in driving attentional control (e.g., Shallice, 1994), is heavily involved.
The attentional control network is often broadly described as a cognitive
system via which individuals selectively attend to relevant sensory input
and ignore irrelevant information. When observing unfolding events, the
attentional control system may be monitoring the predictability of the event
stream and flexibly modulating the viewer’s attention as predictability
changes. Our existing findings utilizing the dwell-time paradigm seem
to confirm this, as does Zacks and colleagues’ research, described earlier,
showcasing links between adults’ processing of event structure and changes
in event predictability (Zacks et al., 2011). The executive system is likely
upregulating attention to the points in the unfolding sequence that the
viewer expects to be most relevant or information rich and simultaneously
downregulating attention to less relevant or informative parts of the event.
Throughout this complex processing task, the attentional control system
is also incorporating accrued knowledge of the observed predictability
structure of the event into long-term memory, thereby helping to further
enrich event-related knowledge, and facilitating processing of similar event
sequences encountered on future occasions.
On this analysis, deficits in executive function would be expected to be
related to impaired event-processing fluency. Preliminary evidence to this
effect is already available. In a 2006 study, Zacks and colleagues investigated
event segmentation patterns and subsequent memory among adults with
dementia of the Alzheimer’s type, a neurocognitive disorder marked by
impaired memory, attention, and other executive functions (Zacks, Speer,
Vettel & Jacoby, 2006). They found that, in a task in which participants were
asked to watch movies of everyday goal-directed activities and identify
Event Processing as an Executive Enterprise
7
where action boundaries exist, the boundaries identified by aging adults
diagnosed with dementia were markedly discrepant from those identified
by healthy older adults and younger adults, suggesting an impairment in
the ability to fluently identify segmental structure within ongoing events.
In addition, among older adults, the extent to which event segmentation
boundaries agreed with normative boundary selections positively predicted
memory performance.
These findings provide additional evidence that appropriately segmenting ongoing events is crucial for proper encoding and recall. Moreover, they
suggest a rather new conceptualization of the cognitive deficits occurring in
dementia. In particular, they point to difficulties in fluent event processing
arising from executive deficits as central in the cognitive profile of dementia,
and possibly even responsible for the memory deficits typically viewed as
characteristic of the disability. Put another way, memory deficits seen among
older adults with dementia may be due in some significant measure to executive system deficits, rather than to memory problems, per se. That is, deficits
in the attentional control network may undercut effective attentional modulation to the predictability contour of an unfolding event, rendering a poorly
articulated grasp of event structure in those with dementia, and as a result,
poor event encoding.
This interpretation of Zacks and colleagues’ findings predicts other
relations between the executive system and event-processing patterns. For
one, other forms of executive deficit, such as that typically seen in those
with attention-deficit/hyperactivity disorder (ADHD), might also be linked
to event-processing impairments. We are currently undertaking research
to examine this possibility. In particular, we are investigating whether
severity of ADHD symptomatology predicts (i) impairment in identifying
event boundaries in an explicit segmentation task, and (ii) reduction in
implicit dwell-time patterns indicative of sensitivity to the segmental and
hierarchical structure of events. If these predictions are borne out, it would
provide among the first available evidence that fluent event processing is
impaired in ADHD, which in turn may help to account for some of the
learning and memory difficulties known to be associated with ADHD.
DEVELOPMENTS IN EVENT PROCESSING
A classic developmental question is to wonder how newborn infants experience the sensory flow that, as adults, we so effortlessly interpret in terms
of discrete, organized, and meaningful events. Considerable research now
clarifies that newborns demonstrate organized and adaptive perceptual processing (e.g., Maurer & Maurer, 1988), but it is only recently that researchers
began to question whether infants this young experience events as sequences
8
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
comprised of discrete units. As it turns out, basic event segmentation skill
seems to be present in the first year of life (Baldwin, Baird, Saylor, &
Clark, 2001; Hespos, Grossman, & Saylor, 2010; Stahl, Romberg, Roseberry,
Golinkoff, & Hirsh-PasekStahl, 2014), for at least some kinds of simple
human action events. Having said this, it nevertheless also seems obvious
that event processing, including segmentation skill, undergoes considerable
reorganization developmentally, in part because knowledge seems to play
such a central role in shaping it. A 12-month-old infant’s understanding of
a marriage ceremony, or of a presidential bill-signing event, for example,
would have to be very different from that of an educated adult’s, given their
ignorance of the complex social and cultural foundations underlying such
events. Infants might detect fine-grain units within the motion stream with
some degree of fidelity (e.g., grasping a pen, bringing a pen to paper, making
squiggles on the page with a pen) but lack an understanding of the higher
order goals of the actors or what they signify within a larger event context.
Hence, segmentation at higher levels and organization of event segments
according to hierarchical, causal, and intention-goal structure probably
change considerably as knowledge and development progress.
Our working hypothesis is that knowledge change and attentional
reorganization in event processing are deeply interrelated in children’s
developmental progress. As children acquire new information about
predictability relations in the world (knowledge) this will reshape their
attention toward more efficient processing of the sensory flow. Progressively
they will be able to selectively target junctures within unfolding experience
at which relevant new information is predictably to be found (such as event
boundaries, among other possibilities). As such reorganization occurs, event
segments emerge as relatively stable entities in their experience, providing
opportunity for noting similarities across events, which in turn enables
children to construct event categories, learn labels (e.g., verbs) for them,
and draw inferences about causal relationships that event categories enter
into. Buchsbaum and colleagues recently documented portions of this
developmental progression, demonstrating that both adults (Buchsbaum,
Griffiths, Gopnik, & Baldwin, in press) and 3-year-olds (Buchsbaum, Gopnik,
Griffiths, & Shafto, 2011) can use newly gained information (via statistical
learning) about novel event segments to guide their causal inferences, and
conversely, they can also use preexisting knowledge of causal structure to
guide perception of event boundaries.
We are hopeful that the dwell-time paradigm will be a useful technique by
which to further observe developmental change of just this kind as it is under
way. One possible example here might be use of the dwell-time paradigm
to observe developmental progress in conceptual-level knowledge, such as
theory-of-mind understanding. To the extent that infants or young children
Event Processing as an Executive Enterprise
9
either possess (e.g., Baillargeon, Scott, & He, 2010) or lack (e.g., Wellman,
Cross, & Watson, 2001) lack an understanding of mentalistic concepts such
as belief or knowledge, this will directly impact the predictions they have
for how an event sequence will play out, which in turn will likely shape
how they deploy their attention as the event proceeds. An older child or
adult’s understanding that an actor possesses crucial knowledge of a hidden
threat, for example, might lead to enhanced dwelling on threat-relevant junctures within the unfolding stream of activity. In contrast, lacking an ability to
appreciate the actor’s knowledge state should leave infants and younger children oblivious to such threat-relevant junctures, with attendant differences
in their dwell-time patterns. In collaboration with Jessica Kosie and Eric Olofson, research testing these and related predictions is currently under way.
Although knowledge seems to reshape event processing, which in turn
yields developmental progress in event understanding, we suspect that
individual differences in executive function are a key determinant of the
pace of developmental progress in this arena. We (Dare Baldwin and Robbie
Ross) are currently investigating the possibility of such links between
individual differences in children’s developing executive skill and fidelity
of event processing. The preschool period is marked by rapid advances in
executive system development. We predict that children in this age range
will display variability in event segmentation patterns, as measured by
dwell times, that is systematically related to their executive function skill.
Moreover, we suspect that children with relatively poor executive skill will
have difficulty reorganizing event processing in response to changing task
demands. Findings from this research will help clarify whether developing executive skill impacts children’s event-processing fluency, a finding
with potentially wide-ranging implications for understanding individual
differences in learning and cognitive growth.
CONCLUSION
We began with the cacophony of a cat-related catastrophe to illustrate that
life presents itself to us as a multimodal flow of complex, dynamic information that we actively transform, or redescribe, into discrete, organized event
structures. Our own and others’ recent research begins to shed light on the
nature of these redescription processes, and how they emerge in children’s
development. In brief, fluent event processing appears to involve skill at
modulating attention to selectively capture predictable, structurally relevant,
information-rich portions of an unfolding activity stream. Conceptualized
this way, event-processing skill fundamentally depends on knowledge and
executive skill. As knowledge of the world grows, more can be predicted. As
more becomes predictable, it is increasingly possible to selectively project
10
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
where, in the unfolding sensory flow, relevant new information is likely to
be found (the predictably unpredictable). However, these developmental
achievements likely depend critically on skill at flexibly reconfiguring
attention based on knowledge and concurrent task demands, which requires
a high-functioning executive system. Our hope is that gaining increased
understanding of such relations among knowledge, executive skill, and
event-processing fluency will ultimately enable us to enhance such fluency
for all.
REFERENCES
Baillargeon, R., Scott, R. M., & He, Z. (2010). False-belief understanding in infants.
Trends in Cognitive Sciences, 14(3), 110–118.
Baldwin, D. A., & Baird, J. A. (1999). Action analysis: A gateway to intentional inference. In P. Rochat (Ed.), Early social cognition: Understanding others in the first months
of life (pp. 215–240). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Baldwin, D. A., & Baird, J. A. (2001). Discerning intentions in dynamic human action.
Trends in Cognitive Sciences, 5, 171–178.
Baldwin, D. A., Baird, J. A., Saylor, M. M., & Clark, M. A. (2001). Infants parse
dynamic action. Child Development, 72(3), 708–717.
Baldwin, D., Hard, B., Meyer, M., & Sage, K. (2014). Attentional reorganization with
discovery of segmental structure in dynamic human action (Manuscript in preparation).
Eugene, OR: Department of Psychology, University of Oregon.
Baldwin, D., & Sage, K. (2013). Dwelling on action. In M. Rutherford & V. Kuhlmeier
(Eds.), Social perception: Detection and interpretation of animacy, agency, and intention
(pp. 309–330). Cambridge, MA: MIT Press.
Buchsbaum, D., Gopnik, A., Griffiths, T. L., & Shafto, P. (2011). Children’s imitation
of causal action sequences is influenced by statistical and pedagogical evidence.
Cognition, 120(3), 331–340.
Buchsbaum, D., Griffiths, T. L., Gopnik, A., & Baldwin, D. (in press). Learning
from actions and their consequences: Inferring causal variables from continuous
sequences of human action. Cognitive Psychology.
Garrison, J., & Baldwin, D. (2014). Finding emo: Discovering segmental structure in
emotional displays (Manuscript in preparation). Eugene, OR: Department of Psychology, University of Oregon.
Hard, B., Recchia, G., & Tversky, B. (2011). The shape of action. Journal of Experimental
Psychology: General, 140, 586–604.
Hespos, S. J., Grossman, S., & Saylor, M. M. (2010). Infants’ ability to parse continuous
action series: Further evidence. Neural Networks, 23, 1026–1032.
Huff, M., Papenmier, F., & Zacks, J. M. (2012). Visual target detection is impaired at
event boundaries. Visual Cognition, iFirst, 1–17.
Maurer, D., & Maurer, C. (1988). The world of the newborn. New York, NY: Basic Books.
Meyer, M., Baldwin, D. A., & Sage, K. (2011). Assessing young children’s hierarchical
action segmentation. In L. Carlson, C. Hölscher & T. Shipley (Eds.), Proceedings of
Event Processing as an Executive Enterprise
11
the 33rd annual conference of the cognitive science society (pp. 3156–3161). Boston, MA:
Cognitive Science Society.
Meyer, M., DeCamp, P., Hard, B., Baldwin, D., & Roy, D. (2010). Assessing behavioral
and computational approaches to naturalistic action segmentation. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd annual conference of the cognitive
science society (pp. 2710–2715). Portland, OR: Cognitive Science Society.
Newtson, D. (1973). Attribution and the unit of perception of ongoing behavior. Journal of Personality & Social Psychology, 28, 28–38.
Newtson, D., Enquist, G., & Bois, J. (1977). The objective basis of behavior units. Journal of Personality and Social Psychology, 35(12), 847–862.
Prokhorov, Y. V. (n.d). Information. Encyclopedia of Mathematics. Retrieved from
http://www.encyclopediaofmath.org/index.php?title=Information&oldid=
14072.
Sage, K. D., & Baldwin, D. (2014). Finding structure in complex manual sequences
(Manuscript submitted for publication). Eugene, OR: Department of Psychology,
University of Oregon.
Saylor, M., & Baldwin, D. (2004). Action analysis and change blindness: Possible
links. In D. T. Levin (Ed.), Thinking and seeing: Visual metacognition in adults and
children (pp. 37–56). Westport, CT: Greenwood/Praeger.
Shallice, T. (1994). Multiple levels of control processes. Attention and Performance XV,
395–420.
Stahl, A. E., Romberg, A. R., Roseberry, S., Golinkoff, R. N., & Hirsh-Pasek, K. (2014).
Infants segment continuous events using transitional probabilities. Child Development, 85(5), 1821–1826.
Wellman, H. M., Cross, D., & Watson, J. (2001). A meta-analysis of theory of mind
development: The truth about false belief. Child Development, 72, 655–684.
Zacks, J. (2004). Using movement and intentions to understand simple events. Cognitive Science, 28, 979–1008.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z., Ollinger,
J. M., … Raichle, M. E. (2001). Human brain activity time-locked to perceptual
event boundaries. Nature Neuroscience, 4, 651–655.
Zacks, J. M., Kurby, C. A., Eisenberg, M. L., & Haroutunian, N. (2011). Prediction
error associated with the perceptual segmentation of naturalistic events. Journal of
Cognitive Neuroscience, 23(12), 4057–4066.
Zacks, J. M., Speer, N. K., Vettel, J. M., & Jacoby, L. L. (2006). Event understanding
and memory in health aging and dementia of the Alzheimer type. Psychology and
Aging, 21(3), 466–482.
Zacks, J. M., & Swallow, K. M. (2007). Event segmentation. Current Directions in Psychological Science, 16(2), 80–84.
Zacks, J. M., Swallow, K. M., Vettel, J. M., & MacAvoy, M. P. (2006). Visual movement
and the neural correlates of event perception. Brain Research, 1076(1), 15–162.
Zacks, J. M., & Tversky, B. (2001). Event structure in perception and cognition. Psychological Bulletin, 127, 3–21.
Zacks, J. M., Tversky, B., & Iyer, G. (2001). Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130, 29–58.
12
EMERGING TRENDS IN THE SOCIAL AND BEHAVIORAL SCIENCES
ROBBIE A. ROSS SHORT BIOGRAPHY
Robbie A. Ross is a doctoral student in the Psychology Department at the
University of Oregon in Eugene, Oregon. She graduated from Tulane University in 2005 with a bachelor’s degree in psychology and spent several years
working in infancy research. Before joining the UO, she worked in a developmental neuroscience laboratory at Columbia University studying the effects
of perinatal exposures on infant neurodevelopment and cognition. While in
New York, Ross also conducted research at New York University measuring
infants’ understanding of the influence of emotional language on behaviors
in others. Her current research is focused on the role of the executive function
system in learning among preschool and school-aged children. In particular,
she is interested in how executive control is involved in processing dynamic
information streams.
baldwinlab.uoregon.edu
teamduckling.uoregon.edu
psychology.uoregon.edu
DARE A. BALDWIN SHORT BIOGRAPHY
Dare A. Baldwin is Professor of Psychology at the University of Oregon. Her
research focuses on enhancing the human potential for knowledge acquisition. Although in humans knowledge is acquired at a remarkable pace from
earliest infancy on, much remains mysterious about the developmental origins of the underlying mechanisms making this possible. Baldwin seeks to
understand how human infants rapidly and effectively acquire and organize
world knowledge across many domains. Her research has been recognized
by awards such as APA’s Boyd McCandless Award, an APA Distinguished
Scientific Award for Early Career Contribution to Psychology, and a Guggenheim Fellowship. In recent years, Baldwin’s research has been funded by the
National Science Foundation, the Office of Naval Research, and the University of Oregon Fund for Faculty Excellence. A former fellow of the Center
for Advanced Study in the Behavioral Sciences, Baldwin is currently a fellow
of the Association for Psychological Science, and on the executive board of
the Society for Language Development. Baldwin completed undergraduate
work in Psychology at U.C. Berkeley, received a master’s at U.C. Santa Cruz,
and a PhD in Psychology, with a special designation in Cognitive Science, at
Stanford University.
RELATED ESSAYS
Models of Revealed Preference (Economics), Abi Adams and Ian Crawford
Coevolution of Decision-Making and Social Environments (Sociology),
Elizabeth Bruch et al.
Event Processing as an Executive Enterprise
13
Heuristic Decision Making (Political Science), Edward G. Carmines and
Nicholas J. D’Amico
Behavioral Economics (Sociology), Guy Hochman and Dan Ariely
Emotion and Decision Making (Psychology), Jeff R. Huntsinger and Cara Ray
From Individual Rationality to Socially Embedded Self-Regulation (Sociology), Siegwart Lindenberg
Resource Limitations in Visual Cognition (Psychology), Brandon M. Liverence and Steven L. Franconeri
Against Game Theory (Political Science), Gale M. Lucas et al.
