EARLY CHILDHOOD DISABILITIES PAPER INSTRUCTIONS After reading Chapter 14 of the O’Brien and Beattie textbook, you will research the concepts covered in the chapter utilizing scholarly journal articles. After reading the assigned chapter and scholarly journal articles, you will write a 2-3-page summary (excluding the title page, abstract, and reference page), in current APA format, addressing the prompts below. The paper should incorporate at least 6 citations utilizing at least 2 references (the textbook and a scholarly journal article) to support your assertions. Include a correctly formatted title page, abstract, and reference page. Organize the paper utilizing appropriate headings.


• What do in-service teachers need to know about the concept of early childhood special education development of the preschool child who had not previously received infant- toddler early interventions services.

• Why is a family centered approach to providing early intervention in the natural environment important?

• Describe the concept of embedded intervention that is used to provide early intervention/each childhood special education?

• What role does research and evidence-based practice play in early childhood interventions?

• What do pre-service teachers need to know about providing early intervention to students with special needs?


Stuart W. Volkow Alex C. Howland, PhD

The world of work is rapidly changing. Now, more than ever, the need for continuous workforce training

is needed. While there are many benefits to social and experiential offerings of face-to-face training,

distance learning is typically more practical in today’s society. Unfortunately, current distance-learning

technologies lack the immersion necessary for learning 21st-century skills. Virtual reality and

augmented reality (i.e., mixed realities) can be more effective for training and learning than traditional

flat-screen media.

THE FUTURE OF WORK AND THE OPPORTUNITY OF MIXED REALITIES TO IMPROVE PERFORMANCE Robots, automation, and artificial intelligence are rapidly changing the face of the American workforce. As more and more jobs are filled by machines, experts agree that the education marketplace will need to change to keep up with the growing and widespread need for worker retrain- ing (Pew Research Center Internet & Technology, 2017). While there are benefits to the social and experiential learning experiences that can be offered in a face-to-face setting, distance learning is often an affordable and flexible way to provide that training.

Unfortunately, most eLearning and webinar platforms only offer participants a passive experience (e.g., watch- ing videos, listening to a lecture). With corporate edu- cation, including employee orientation, onboarding, and skill building, passive learning is the norm, consisting largely of sitting down and consuming pre-packaged con- tent in bulk that’s presented formally by an educator (Hinchcliffe, 2017). Such offerings do not help develop the vital skills needed in today’s increasingly global and dis- tributed economy, such as teaming, communication, lead- ership, and cultural intelligence. They also do not immerse learners into the context of the learning and provide the ability for learners to practice in a safe environment. As a result, many learners develop feelings of isolation, dis-

connectedness, and frustration, often associated with poor retention rates and low return on investment (Willging & Johnson, 2009).

Mixed-reality technologies (i.e., virtual reality and aug- mented reality) provide solutions to these problems by allowing people to come together in an active simulated environment that allows them to see and interact with fel- low participants and the simulated environment, regard- less of geographic location. Such technologies have the po- tential to dramatically transform education, training, and human performance. The aim of this article is to provide an overview of mixed realities (MR), to discuss theories as- sociated with how the technologies can provide value for performance, and to provide specific examples of effective early-use cases.

Introduction to Mixed Realities Well told, any story can be immersive. From spoken word to literature, film, and television, imagination works to transport us. The new immersive platforms and media transcend these by adding an element commonly referred to as presence. Mixed realities offer increased immersion though features including head tracking, hand tracking, eye tracking, haptics, and 360 imagery. When combined, these add visual and kinesthetic cues that complete the il- lusion of being somewhere else, completely immersed in real or imagined environments. Virtual reality (VR) refers

Performance Improvement, vol. 57, no. 4, April 2018 © 2018 International Society for Performance Improvement

Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/pfi.21777 29

to immersive experiences which are completely separated from the real world. Augmented reality (AR) refers to ex- periences that overlay or are otherwise integrated into the real environment.

VR and AR technologies usually require tethered or wirelessly connected head-mounted display systems (HMDs—e.g., Oculus Rift and HTC Vive) with special- ized optics, hand controllers, and haptic accessories. In ad- dition to HMDs, almost any connected screen can serve as an augmented-reality window, which may be the most common modality for MR experiences for some years to come. The advent of numerous professional and consumer HMDs and development tools from Google, Apple, Mi- crosoft, Unity, Worldviz, and other technology providers are opening new opportunities for education, training, and development. The term mixed reality (MR) is being used by Microsoft (Hololens) and others to refer to a con- tinuum of experiences inclusive of both VR and AR.

The Value of MR for Learning and Improving Human Performance: Presence and Embodiment Chief learning officers, curriculum designers, trainers, and educators can take advantage of mixed-reality (MR) tools to create cost-efficient immersive experiences that engage learners in new ways and, in some cases, may replace or enhance face-to-face courses and collaborations. MR can accommodate visual and kinesthetic learning styles in ways traditional methods often cannot. Social, cultural, and collaborative aspects of learning that are diminished or eliminated with traditional eLearning platforms can be supplemented or replaced to good effect. While empiri- cal research into MR is still in its infancy, institutions are sharing results of their applied research and initial explo- rations. Some are shared in this article. Two examples of the value mixed realities offer to learning and performance are social presence and embodied cognition.

“I hear and I forget, I see and I remember, I do and I understand.” (Confucius) Social presence in virtual worlds has been shown to cre- ate a sense of realism and immersion that enhances learn- ing beyond face-to-face or traditional online interactions (Biocca & Harms, 2002; Robb, Lampotang, & Wendling, 2015). Simulations, serious games, role playing, and vir- tual worlds have become cost effective to create and pub- lish using distributed platforms for PCs, mobile devices, and head-mounted displays. Social VR is a term that is being used to describe mixed-reality experiences whereby

Social presence in virtual worlds has been shown to create a sense of realism and immersion, enhancing learning beyond face-to-face or traditional online interactions.

geographically remote users can interact and collaborate in virtual spaces.

Embodied cognition is another explanation for why MR is so effective as a learning modality. Embodied cognition is an emerging cognitive science that postulates that learn- ing, cognition, language, memory, and even mathematics (e.g., Lakoff & Nunez, 2000) are inexorably linked to our physicality, kinesis, and even the morphology of the brain and nervous system (McNerney, 2011; Wilson & Golonka, 2013). Our experience of the physical world is not separate from learning and cognition but rather involves essential parts of cognition, reflected in linguistic structures and the circuitry of the brain. Abstractions such as mathematics are metaphors understood through embodied experiences that include movement and gesturing (Lakoff & Johnson, 1999; Lakoff & Nunez, 2000; McNerney, 2011). For exam- ple, phrases such as “getting my arms around a problem” or “you’re on top of this” embody abstractions. Pointing, pinching, and even counting on fingers, are examples of gestural embodiment. This may be why learn by doing is such a powerful axiom.

MR technologies can bring embodiment to abstract and difficult-to-simulate learning scenarios. We posit that this embodiment extends the concept of experiential ed- ucation as proposed by educational philosopher and re- former John Dewey (1997). He stated, “Education should derive its materials from present experience and should enable the learner to cope with the problems of the present and the future” (p. 51). Dewey proposed an experiential continuum, whereby teachers are there to set up activities and experiences that lead to growth and where learning is doing. Mixed reality can bring together audio, visual, kinesthetic, and tactile modalities, thereby catering to al- most all learning styles.

Presence A sense of presence is a major differentiator between VR, AR, and other mediums. Presence is a shorthand for telepresence and is the illusion or perception of be- ing teleported somewhere beyond your physical, real

30 www.ispi.org • DOI: 10.1002/pfi • APRIL 2018


Component Example

Spatial/Physical A visual, kinesthetic, proprioceptive, and/or vestibular illusion of being in a physical space from a room or a building to a mountaintop or a molecule. May allow the perception of moving through the space by walking, driving, running, floating, or flying. Sometimes used to describe telepresence such as in the operation of an excavator, robot, or drone. Also called first and second order presence (Riva, Waterworth, & Murray, 2014).

Sensory Achieved with haptics (i.e., tactile sensation and control), the illusion can extend to touch, vibration, texture, and pressure. Spatialized sound adds the auditory dimension with 3D binaural cues.

Psychological Psychological presence is akin to the feeling of being immersed in the narrative of a movie enough to suspend disbelief and accept the story-world as real.

Social Extends the illusion to the convincing presence of other people in the virtual world presented as realistic representations or avatars.

location. Books, movies, and games achieve this to some extent through imagination. MR technologies dramati- cally intensify the illusion. The works of Nonny De La Pena (Hunger in Los Angeles, Project Syria, Gone Gitmo) and Gabo Arora (Clouds Over Sidra, The Last Goodbye) are critically acclaimed examples of embodied journalism. Ms. La Pena’s immersive documentary and journalistic works have been instrumental in furthering the idea that MR technologies are empathy engines.

Even with relatively low-fidelity representations, the response-as-if-real or RAIR can be intense (e.g., De La Pena et al., 2010; Alsever, 2015; Watercutter, 2017). Stan- ford University’s Virtual Human Interaction Lab is fo- cused on VR technology’s application of embodied cog- nition theory and its impact on empathy and behavior. Stanford’s Empathy at Scale project is exploring how these technologies can be used to teach and measure empathy. Their research has verified the impact of virtual experi- ences on perception and behavior (Bailey, Bailenson, & Casasanto, 2016).

The International Society for Presence Research (2000) has defined presence more precisely by breaking it down into several components (as shown in Table 1). Variants of presence can be effectively leveraged to give learners immersive experiences difficult or impossible to provide through conventional means. Such immersion leads to in- creased learning engagement and retention.

Social Presence and the Use of Avatars Researchers have demonstrated how avatars in VR or AR situations can take advantage of experiential learn- ing, embodied cognition, and immersive learning envi- ronments to augment or replace face-to-face situations and to improve student engagement. Kahn (2015) has found that participants feel better with higher motivation,

score higher on retention, and are better able to transfer knowledge by using avatars for embodiment. She ob- served that students often merge their real selves with their avatars to great effect. The avatars are expressions of per- sonality that make real connections. Dr. Kahn has used avatars in virtual worlds to teach history and educational psychology. The Virtual Experience Research Group at the University of Florida has shown that avatars can be effec- tive in training medical students and professionals in di- agnostic and surgical procedures, replacing expensive and hard-to-schedule real people (Lok & Chuah, 2014). Urol- ogy residents at the University of Miami Miller School of Medicine have been successfully taught core clinical communications skills using patient avatars in a virtual environment (Kava, Andrade, Marcovich, Idress, & Ruiz, 2017).

The commercial mass deployment of social VR plat- forms may make the demand for this kind of learning ex- perience grow rapidly as users become comfortable with the modality. At Facebook’s F8 Annual Developers Sum- mit, Facebook CEO Mark Zuckerberg announced the beta release of Spaces, its social VR application. Spaces uses avatars to interact in virtual environments. Indicative of its commitment to AR and VR, Facebook has a head of social VR, Rachel Franklin. According to Franklin (2017), “VR has the promise to be the most powerful social platform.” Franklin defends the use of avatars and asserts that in a short time users easily customize their avatars to represent them well and become comfortable with using them as a proxy for natural interactions so they can “be themselves” and experience a powerful sense of presence in VR, or as she describes it, “an extension of who you are.” Such social presence allows communities of learners to come together, regardless of geographical location, to share, network, and learn. The social VR platform, VirBELA, allows users to

Performance Improvement • Volume 57 • Number 4 • DOI: 10.1002/pfi 31

connect and communicate across devices: iOS/Android, Mac/PC, and fully immersive VR. Universities such as the University of California and the Stanford Graduate School of Business use the technology for distance learning with students around the world.

Learning Games and Serious Games Another benefit to MR technologies is that they can house simulations and serious games. The term serious games has been applied to games that teach, motivate, and result in organized actions. Simulation, role-playing, and strat- egy games use clever game mechanics to drive learners through progressive levels to accomplish learning objec- tives. Gaming satisfies the do in the learning ladder in ways that other methods cannot. Gamification of curricula with points and badges and teams is a superficial level with little use of game mechanics. In Reality Is Broken, futur- ist and game designer Jane McGonigal (2011) pushed the idea that adding a gameful element can make almost any- thing more engaging and memorable. Social games can bring people together in new ways for common causes. In VR users can design their own avatars, earn badges for performance in games or simulations, and appear in leaderboards.

Researchers out of the University of California–San Diego explored the opportunity for developing and assess- ing the global leadership and business acumen of MBA students around the world by engaging them in a business simulation in VR (see Figure 1). Thirty students from 10 different countries participated on teams in a VR automo- bile simulation where no two people were from the same country or university. The research (Howland, Rembisz, Wang-Jones, Heise, & Brown, 2015) highlighted that al- though the students were participating through avatars, the medium was effective in assessing leadership skills, cultural intelligence, and business acumen. Each team was observed by an organizational psychologist. The psychol- ogists believed that they obtained nearly as much infor- mation from observing VR behavior as they do when they conduct behavioral assessments in live face-to-face simu- lations. Impression management of students quickly went away. Students were so immersed that some teams were screaming at one another, while others came together for group avatar hugs with joy.

The research also showed the foundation of future op- portunities for assessment in VR. Server logs were used to observe how high-performance teams’ communica- tion patterns differed from low-performing teams, and heatmaps were generated to see where students spent their time in the environment (i.e., what information they were or were not accessing to drive their decisions). One lim-

Social VR provides a new platform for bringing everyone to one central virtual headquarters. No one has any advantage over others for being in the city or the headquarters or on-site instead of logging in virtually because everyone is virtual.

itation was the lack of facial expressions, something that may be addressed with improvements of facial and eye tracking in head mounted displays. The YouTube video of Mark Zuckerberg’s VR demo at the Oculus Connect con- ference (2016) provides an example of what is coming with facial expressions on avatars.

Opportunities for Mixed Realities to Improve Performance There are an incredible number of opportunities to lever- age MR technologies to improve performance. The follow- ing text provides a handful of examples of what is possible and what is already being done in practice today. The hope is that the examples cultivate creativity in the readership as to how they may use MR to improve performance in their respective industries and areas of expertise.

Example 1: Social VR for Improved Communications of Distributed Workforces Although there are many benefits to having a distributed workforce, many companies do not realize its full ben- efit because they struggle to build a culture with high- performing distributed employees and teams. Working remotely can be a lonely experience. Collaboration is typ- ically limited to transactional conversations and lacks the relational aspects face-to-face teams experience. Small groups of employees are all siloed from each other, as most of their communication is on webinars and conference calls that are isolated from one another. To compensate, companies spend large amounts of money to occasionally bring employees together.

Social VR provides a new platform to bringing ev- eryone to one central virtual headquarters. No one has advantage over others by being in the city or at the

32 www.ispi.org • DOI: 10.1002/pfi • APRIL 2018



headquarters or on-site instead of logging in virtually, be- cause everyone is virtual. The spatialized environment and voice allows for fluidity in supporting diverse-size meet- ings, from one on one, to small groups, to hundreds of par- ticipants. Through embodied cognition, VR has the op- portunity to foster deep relationships among distributed employees. Learners quickly get the feeling of actually be- ing in the same room with one another, even more so than by looking at each other’s faces through a webcam. Quotes from our own survey research have included the following two:

“Very interesting experience—I can see using some- thing like this over Readytalk or other conferencing platforms. Feels more real.”

“I was skeptical before to start but was really very surprised by the virtual environment. You really have the feeling to be in the same room and work as a team.”

Imagine running scrum meetings in VR, going to a vir- tual ropes course for team development, having an open- door policy in virtual offices, or hanging out in a VR cafe

Performance Improvement • Volume 57 • Number 4 • DOI: 10.1002/pfi 33


with employees around the world for informal network- ing. Being represented by avatars has been demonstrated to reduce the unfortunately negative aspects of diver- sity from a race, weight, height, and culture standpoint and to change how people interact. Based on avatar representations, participants engage in more self- disclosure and can become more confident. The impact of an avatar on its user has been dubbed the Proteus Effect (Yee & Bailenson, 2007).

Some companies are already taking this to an extreme and are running completely cloud-based businesses in VR. eXp Realty is a cloud-based residential realty company that holds its board meetings, all-hands leadership meet- ings, training, and realtor transactions in a virtual head- quarters. The company is saving a fortune on brick-and- mortar costs, is inclusive of people across geographies, has a small environmental footprint, is growing like crazy, and has a united culture with its realtors around the United States and Canada. Because the company saves consid- erable money by not having a brick-and-mortar head- quarters, it can pass some of the savings back to real es- tate agents—a competitive advantage that has allowed the company to grow from under 1,000 agents to nearly 5,000 in about 12 months. The environment not only allows for instructor-led training but also supports communities of learners. Mastermind groups have formed organically, peer-to-peer coaching occurs, and the 3D environment is used for teams to break into groups to work among them- selves. There is no need to send emails back and forth to try to set up a meeting days or weeks in advance. Team members can just walk their avatars over to each other to ask about their availability or even meet immediately. A picture of the campus is provided in Figure 4.

Example 2: AR for Field-Based Knowledge Support Highly technical field-engineering and maintenance work, especially when health, critical infrastructure, and transportation are involved, requires both high levels of skill and access to voluminous reference information. AR is being successfully applied to provide heads-up visual overlays that reduce training time and deliver just-in-time reference information to workers when and where they need it (Abraham & Annunziata, 2017). Aircraft main- tenance is an exemplary use case. By augmenting skills with visual overlays and guidance from remote experts, training times for engineers could be reduced by one to two years (Deal, 2017). Similar cases can be made for dangerous construction work (Le, Pedro, Lim, Park, H., Park, C., & Kim, 2015), safety training (Quang et al., 2015) and manufacturing (Magid & Marco, 2017).

Example 3: VR for Competency-Based Training Spun out of Stanford University, Strivr is a company that applies VR, mostly in the form of 360 video experiences, to sports and industrial training. NFL athletes and coaches have reported dramatic gains in performances by being able to perform mental reps of plays in the 360 video re- plays (Gaudiosi, 2015). The benefits seem to be more than anecdotal (Casale, 2017).

Example 4: VR for Innovation Research on ideation has found that creativity is fos- tered when people and teams are pulled out of their nor- mal environments, allowing them to break out of their typical mental models. With VR, there is no need for

34 www.ispi.org • DOI: 10.1002/pfi • APRIL 2018


expensive outings to immerse employees in such alterna- tive environments—or universes, for that matter. In an instant, employees can go just about anywhere. Innova- tion is also more likely to occur when silos are integrated, helping bring diverse perspectives and ideas together. So- cial VR will allow employees across geographical, market, and cultural silos to come together without the expense of travel to brainstorm and explore new business and product ideas. VR will also allow innovators to work with things they might not be able to do in the real world. For exam- ple, San Diego startup Nanome (http://nanome.ai/) allows users to interact and build molecular models in VR that are obviously too small to see with the naked eye.

Example 5: MR for Marketing and Sales VR will have an impact in marketing and sales training, as well as sales execution. For example, a medical-device company could train its globally distributed sales force how the device works through AR or VR. The sales folks could even demo the product with the customer in this fashion. The emotional experience of interacting with the product is anticipated to lead to increased sales over a typical presentation with images, videos, or slides. Three- hundred-and-sixty-degree (360o) cameras are starting to be used heavily in the real estate (e.g., Christoffer, 2016), travel (Levere, 2017), and automotive sectors (Gaudiosi, 2016). Home buyers can tour homes without driving from house to house, travelers can get a sense of a vacation lo- cation before making a purchase, and car buyers can get a sense of what it’s like to take an Audi out on the race track.

Example 6: VR for Risk Management Safety training and risk mitigation are a lot more impact- ful and effective through experience. High-risk industries, including the military, oil and gas, and mining have been front runners in leveraging VR simulations for risk reduc- tion. Many risks cannot be experienced in the real world because of cost and safety constraints. VR reduces (or removes) these constraints. Employees will have very real experiences in VR that evoke emotions that encode learning into long-term memory and create ah-ha mo- ments. The application can be applied in areas such as diversity and inclusion training (e.g., taking the experi- ence of another to gain empathy), to practicing emer- gency procedures (e.g., what to do in the event of a fire), to helping in preventative design measures (e.g., explore the experience and safety features of a mine or Navy vessel before it is even constructed). Leading de- velopers in these areas include companies such as eon Reality, Cubic Corporation, Workplace Technologies Re- search Institute, and Luminous, although there are many others. These firms often specialize in MR development and also have expertise in such areas as instructional design, cognitive science, and industrial/organizational psychology.

CONCLUSION The potential for MR to provide performance improve- ments is immense. Taking advantage of these technolo- gies will take some time, iteration, and experimentation. Hardware will need continual improvement, and price

Performance Improvement • Volume 57 • Number 4 • DOI: 10.1002/pfi 35http://nanome.ai/

points need to come down to increase accessibility. Many will likely wait for mass-market adoption before explor- ing these technologies, but early adopters may reap the most benefits. It is not too early to make smart invest- ments to use MR for projects that add value and pro- vide a clear return on investment; many companies are already doing so. This article has provided an overview of MR technologies, has touched on theory as to how the technologies add value, and has offered several examples of how MR can be and is being used for performance improvements. These are many examples and opportu- nities beyond those provided here. Hopefully, the exam- ples help ignite creativity and discussions to push into the unknown in an attempt to improve performance across many industries and professions with exciting emerging technologies.


Abraham, M., & Annunziata, M. (2017, March 13). Augmented reality is already improving worker performance. Retrieved from https://hbr.org/2017/03/augmented-reality-is-already- improving-worker-performance

Alsever, J. (2015, November). Is virtual reality the ultimate empathy engine? Retrieved from https://www.wired.com/ brandlab/2015/11/is-virtual-reality-the-ultimate-empathy- machine/

Bailey, J.O., Bailenson, J.N., & Casasanto, D. (2016). When does virtual embodiment change our minds? Presence: Teleoperators and Virtual Environments. 25(2). Retrieved from https://vhil .stanford.edu/mm/2016/10/bailey-presence-change-our-minds .pdf

Biocca, F., & Harms, F. (2002). Defining and measuring social presence: Contribution to the Networked Minds Theory and Measure. Proceedings of the 5th International Workshop on Presence. Retrieved from http://astro.temple.edu/˜lombard/ ISPR/Proceedings/2002/Biocca%20and%20Harms.pdf

Casale, M. (2017). STRIVR training demonstrates faster and more accurate learning compared to traditional study methods. Retrieved from https://strivrlabs.com/wp-content/uploads/ 2017/04/STRIVRtech-Report_final_long.pdf

De La Pena, N., Weil, P., Llobera, J., Pomes, J.L., Spanlang, B., Freidman, D., Sanchez-Vives, M.V., & Slater, M., (2010). Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence, 19, 291–301. Retrieved from http://www.mitpressjournals.org/doi/pdf/ 10.1162/PRES_a_00005

Deal, K. (2017, February 24). Commercial aviation MRO using augmented reality, virtual reality to bridge skills shortage gap.

Retrieved from http://www.intelligent-aerospace.com/articles/ 2017/02/commercial-aviation-mro-using-augmented-reality- to-bridge-skills-shortage-gap.html

Dewey, J. (1997). Experience and education. Indianapolis, IN: Kappa Delta Pi.

Facebook. (2016, October 12). Oculus Connect 3 opening keynote: Mark Zuckerberg. [Video File]. Available from https://www.youtube.com/watch?v=NLnk2cCirQM

Franklin, R. (2017, May 1). Vision summit keynote. Retrieved from https://youtu.be/ODXMhaNIF5E?t=52m3s

Gaudiosi, J. (2015, August 10). Here’s why NFL teams are training in virtual reality. Retrieved from http://fortune.com/2015/08/10/strivr-virtual-reality-nfl/

Gaudiosi, J. (2016, January 8). Audi drives virtual reality showroom with HTC Vivie. Retrieved from http://fortune.com/2016/01/08/audi-showroom-uses-vr/

Howland, A. C., Rembisz, R., Wang-Jones, T. S., Heise, S. R., & Brown, S. (2015). Developing a virtual assessment center. Consulting Psychology Journal: Practice and Research, 67(2), 110–126.

International Society for Presence Research. (2000). The concept of presence: explication statement. Retrieved from https://ispr.info/

Kahn, S.A. (2015). Embodiment and learning: When avatars smile. Sunnyvale, CA: London Socrates LLC.

Kava, B.R., Andrade, A.D., Marcovich, R., Idress, T., & Ruiz, J.G. (2017). Communication skills assessment using human avatars: Piloting a virtual world objective structured clinical examination. Urology Practice Journal, 4, 76–84. Retrieved from: http://www.urologypracticejournal.com/article/S2352- 0779(16)00025-X/pdf

Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its challenge to western thought. New York, NY: Basic Books.

Lakoff, G., & Nunez, R. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. New York, NY: Basic Books.

Le, Q.T., Pedro, A., Lim, C.R., Park, H.T., Park, C.S., & Kim, H.K. (2015). A framework for using mobile based virtual reality and augmented reality for experiential construction safety education. International Journal of Engineering Education, 31, 713–725.

Levere, J.L. (2017, February 12). Before you take a trip: How about a virtual “test drive”? Retrieved from: https://www .nytimes.com/2017/02/12/business/media/travel-virtual- reality.html?mcubz=3

36 www.ispi.org • DOI: 10.1002/pfi • APRIL 2018https://hbr.org/2017/03/augmented-reality-is-already-improving-worker-performancehttps://hbr.org/2017/03/augmented-reality-is-already-improving-worker-performancehttps://www.wired.com/brandlab/2015/11/is-virtual-reality-the-ultimate-empathy-machine/https://www.wired.com/brandlab/2015/11/is-virtual-reality-the-ultimate-empathy-machine/https://www.wired.com/brandlab/2015/11/is-virtual-reality-the-ultimate-empathy-machine/https://vhil.stanford.edu/mm/2016/10/bailey-presence-change-our-minds.pdfhttps://vhil.stanford.edu/mm/2016/10/bailey-presence-change-our-minds.pdfhttps://vhil.stanford.edu/mm/2016/10/bailey-presence-change-our-minds.pdfhttp://astro.temple.edu/~lombard/ISPR/Proceedings/2002/Biocca%20and%20Harms.pdfhttp://astro.temple.edu/~lombard/ISPR/Proceedings/2002/Biocca%20and%20Harms.pdfhttps://strivrlabs.com/wp-content/uploads/2017/04/STRIVRtech-Report_final_long.pdfhttps://strivrlabs.com/wp-content/uploads/2017/04/STRIVRtech-Report_final_long.pdfhttp://www.mitpressjournals.org/doi/pdf/10.1162/PRES_a_00005http://www.mitpressjournals.org/doi/pdf/10.1162/PRES_a_00005http://www.intelligent-aerospace.com/articles/2017/02/commercial-aviation-mro-using-augmented-reality-to-bridge-skills-shortage-gap.htmlhttp://www.intelligent-aerospace.com/articles/2017/02/commercial-aviation-mro-using-augmented-reality-to-bridge-skills-shortage-gap.htmlhttp://www.intelligent-aerospace.com/articles/2017/02/commercial-aviation-mro-using-augmented-reality-to-bridge-skills-shortage-gap.htmlhttps://www.youtube.com/watch?v=NLnk2cCirQMhttps://youtu.be/ODXMhaNIF5E?t=52m3shttp://fortune.com/2015/08/10/strivr-virtual-reality-nfl/http://fortune.com/2016/01/08/audi-showroom-uses-vr/https://ispr.info/http://www.urologypracticejournal.com/article/S2352-0779(16)00025-X/pdfhttp://www.urologypracticejournal.com/article/S2352-0779(16)00025-X/pdfhttps://www.nytimes.com/2017/02/12/business/media/travel-virtual-reality.html?mcubz=3https://www.nytimes.com/2017/02/12/business/media/travel-virtual-reality.html?mcubz=3https://www.nytimes.com/2017/02/12/business/media/travel-virtual-reality.html?mcubz=3

Lok, B., & Chuah, J.H. (2014). Mixed-reality humans for team training. IEEE Computer Graphics and Applications, 32, 72–75. https://doi.org/10.1109/MCG.2014.57

McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York, NY: Penguin Books.

McNerney, S. (2011). A brief guide to embodied cognition: Why you are not your brain. Retrieved from https://blogs .scientificamerican.com/guest-blog/a-brief-guide-to- embodied-cognition-why-you-are-not-your-brain/

Pew Research Center Internet & Technology. (2017). The future of jobs and jobs training. Retrieved from http://www.pewin ternet.org/2017/05/03/the-future-of-jobs-and-jobs-training/

Riva, G., Waterworth, J., & Murray, D. (2014). Interacting with presence: HCI and the sense of presence in computer mediated environments. Warsaw/Berline: De Gruyter Open Ltd.

Robb, A., Cordar, A., Lampotang, S., White, C., Wendling, A., & Lok, B. (2015). Teaming up with virtual humans: How other

people change our perceptions of and behavior with virtual teammates. IEEE Transactions on Visualization and Computer Graphics, 21, 511–519.

Watercutter, A. (2017, April 20). The incredible urgent power of remembering the Holocaust in VR. Retrieved from https://www.wired.com/2017/04/vr-holocaust-history-prese rvation/

Willging, P.A., & Johnson, S.D. (2009). Factors that influence students’ decisions to drop out of online courses. J. Asynchronous Learning Networks, 13, 115–127.

Wilson, A.D., & Golonka, S. (2013). Embodied cognition is not what you think it is. Frontiers in Psychology, 4(58). https://doi.org/10.3389/fpsyg.2013.00058. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3569617/

Yee, N., & Bailenson, J.N. (2007). The Proteus effect: Self transformations in virtual reality. Human Communication Research, 33(3), 271–290. Retrieved from https://vhil.stanford.edu/mm/2007/yee-proteus-effect.pdf

STUART W. VOLKOW is a visiting scholar at The Arthur C. Clarke Center for Human Imagination at UCSD, and an immersive media producer. He’s producing a VR slate with Deepak Chopra based on Chopra’s best- seller You Are the Universe. After earning his undergraduate physiology degree, he pursued neuroscience as a clinical researcher in pain and stress disease. He completed a fellowship at the American Film Institute going on to a career in film, TV, and interactive media. He pioneered Internet TV, and led a machine- learning company out of the University of Illinois, focused on music recommendation. He is an instructor at UCLA Extension, and Office of Custom Program, and is on the faculty of the Merage U.S.-Israel Busi- ness Bridge Program. He served the Defense Advanced Research Projects Agency designing a knowledge management system, and the Office of the Director of National Intelligence focusing on the role of mixed reality and social media in radicalization. He may be reached at svolkow@gmail.com

ALEX C. HOWLAN, PhD, is an entrepreneur and organizational psychologist. He leads a cross-disciplinary team to create a social virtual reality platform to improve the way people learn and collaborate while geographically apart. To date the VirBELA technology is used for distance education by universities, to help Navy SEALs transition from military to civilian careers, to replace brick-and-mortar offices, and in healthcare applications. His research and innovations have received financial support from the Graduate Management Admissions Council, the National Science Foundation, the National Institute of Health, the Laura Arrillaga-Andreessen Foundation, and the American Psychological Association. He is also an assis- tant faculty member at Alliant International University in the School of Management and Leadership. He may be reached at ahowland@virbela.com

Performance Improvement • Volume 57 • Number 4 • DOI: 10.1002/pfi 37https://doi.org/10.1109/MCG.2014.57https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/http://www.pewinternet.org/2017/05/03/the-future-of-jobs-and-jobs-training/http://www.pewinternet.org/2017/05/03/the-future-of-jobs-and-jobs-training/https://www.wired.com/2017/04/vr-holocaust-history-preservation/https://www.wired.com/2017/04/vr-holocaust-history-preservation/https://doi.org/10.3389/fpsyg.2013.00058https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3569617/https://vhil.stanford.edu/mm/2007/yee-proteus-effect.pdf


Directions: Use this form to complete your reviews of the educational apps that you select. In addition, students will present and demonstrate some of the reviewed apps to the class. You should be prepared to demonstrate and talk about the educational app.

App Review


Description: (3-5 sentences)

Publisher: (Name and APP store URL)

Features (animation, narration, video):

Grade levels:

Subject areas:

Skills (Common Core Standards, Social skills, Curriculum objectives, what could you teach with this):

Lesson Use (highlight during lesson presentation): (4-5 sentences)

How would you incorporate or infuse this in a lesson?

Delivery format (classwide, small groups, individual)

What would students do?

What would teacher do?

Assessments: (Does the app record student performance)

Strengths: (3 sentences)

Weaknesses: (2-3 Sentences)

Recommendations: (1-2 sentences)

Overall impression: (Consider depth of content, ease of use, time to set up, etc)

Review this App and its features for following UDL Guidelines (CAST, 2011). Not every guideline needs to be addressed but hopefully you can find examples for at least 5 out 9 of the guidelines.

UDL Guidelines

Provide Multiple Means of Representation

UDL 1 Provide options for perception

UDL 2 Provide options for language and symbols

UDL 3 Provide options for comprehension

Principle II Provide Multiple Means for Action and Expression

UDL 4: Provide options for physical actions

UDL 5: Provide options for expressive skills and fluency

UDL 6: Provide options for executive functions

Principle III Provide Multiple Means for Engagement

UDL 7: Provide options for recruiting interest

UDL 8: Provide options for sustaining effort and persistence

UDL 9: Provide options for self-regulation

Assistive Technology Considerations Template

Subject AreaSample TaskAssistive Technology Tools, Accommodations or ModificationsLinks to Resource Vendors
Example :MathUse coins and bills to make change and solve math word problems involving money. (Arizona Department of Education, 2015)Talking Calculator andCoin-U-latorTalking calculators and money calculators provide tactile, auditory, and visual feedback and can help students with disabilities perform math calculation assignments. The Coin-U-Lator actually has buttons for coins and bills.Math APPs are electronic games that can provide additional independent practice on an iPad, phone, or computer.https://www.enablemart.com/coin-u-lator-accessorieshttp://www.attainmentcompany.com/talking-calculatorhttps://www.commonsensemedia.org/learning-ratings/reviews?sort=field_reference_review_lr%3Afield_total_learning_rating&order=desc&gclid=COXVoKWB68gCFROSfgodpqsKFA
Oral CommunicationDevelopment

© 2015. Grand Canyon University. All Rights Reserved.

Case Study: Jacob

Student: Jacob

Age: 9.9

Grade: 4th

Jacob is an active boy who enjoys coming to school and participates in sports after school. Jacob does well in most classes but has difficulty with new vocabulary words. This difficulty with new vocabulary is not only seen during reading class but also during content area classes such as science and social studies. Jacob’s mother indicated that he has always had difficulty understanding new vocabulary but that it has not affected his comprehension until this year. Jacob is willing to adopt new strategies that will assist him in learning and remembering new vocabulary words and also help him better understand the story as a whole.

Adapted from Paulsen, K. & The IRIS Center. (2004). Comprehension and vocabulary: Grades 3-5.

© 2015. Grand Canyon University. All Rights ReservedEffective Date: June 2015

articles summery

Article 3: “School within a School”: Examining Implementation Barriers in Spanish/English Transitional Bilingual Education Program

Published in Bilingual Research Journal

– This journal focuses on discussing theory and practice in bilingual education. Further, it discusses language minortity children in the U.S.


– The researcher primarily observed and interviewed teachers who participated in one elementary school’s Spanish/English transitional bilingual education program. The researcher sought to answer what barriers were present while implementation was taking place


– Administrators sought to physical and psychologically separate the bilingual education program from the general education program at the school

– Physical separation occurred as the bilingual program was located in area of the school that made the bilingual students and teachers feel as though they were not a part of the school. This also separated the students during lunch and other school activities

– It was apparent that the bilingual program was being provided with many resources, teachers felt that the bilingual program had more resources than necessary

– Teachers all had different ideas about the requirements for the bilingual education program and ESL teaching methods. This led to misinterpretation in the way the program was taught by each individual teacher

– More dialogue and sharing of knowledge was necessary between teachers, administrators, and staff to make the program successful


– A 6 month qualitative research case study of one midwestern elementary school’s bilingual program

– 21 teachers, staff, and administrators participated

– Ethnography was used

– 16 observations took place of participants with 8 observations of school events in addition to interviews

– Artifacts like notes school-to-home communications, and articles related to the school from local newspapers were collected

Theoretical Orientations

– Critical pedagogy was used to evaluate this transitional bilingual education program. This states that teaching is a political act that influences students and aids in releasing them from oppression. Freire was a major proponent of critical pedagogy

articles summery

Language Arts Journal of Michigan Volume 19 Issue 1 Relevance Article 9


Making Meaningful Connections: The Role of Multicultural Literature in the Lived Experiences of Students Gina Louise DeBlase Wayne State University

Follow this and additional works at: https://scholarworks.gvsu.edu/lajm

This Article is brought to you for free and open access by ScholarWorks@GVSU. It has been accepted for inclusion in Language Arts Journal of Michigan by an authorized editor of ScholarWorks@GVSU. For more information, please contact scholarworks@gvsu.edu.

Recommended Citation DeBlase, Gina Louise (2003) “Making Meaningful Connections: The Role of Multicultural Literature in the Lived Experiences of Students,” Language Arts Journal of Michigan: Vol. 19: Iss. 1, Article 9. Available at: https://doi.org/10.9707/2168-149X.1279https://scholarworks.gvsu.edu/lajm?utm_source=scholarworks.gvsu.edu%2Flajm%2Fvol19%2Fiss1%2F9&utm_medium=PDF&utm_campaign=PDFCoverPageshttps://scholarworks.gvsu.edu/lajm/vol19?utm_source=scholarworks.gvsu.edu%2Flajm%2Fvol19%2Fiss1%2F9&utm_medium=PDF&utm_campaign=PDFCoverPageshttps://scholarworks.gvsu.edu/lajm/vol19/iss1?utm_source=scholarworks.gvsu.edu%2Flajm%2Fvol19%2Fiss1%2F9&utm_medium=PDF&utm_campaign=PDFCoverPageshttps://scholarworks.gvsu.edu/lajm/vol19/iss1/9?utm_source=scholarworks.gvsu.edu%2Flajm%2Fvol19%2Fiss1%2F9&utm_medium=PDF&utm_campaign=PDFCoverPageshttps://scholarworks.gvsu.edu/lajm?utm_source=scholarworks.gvsu.edu%2Flajm%2Fvol19%2Fiss1%2F9&utm_medium=PDF&utm_campaign=PDFCoverPageshttps://doi.org/10.9707/2168-149X.1279mailto:scholarworks@gvsu.edu








Sitting in the back of an eighth grade

English classroom on an early winter day, I listened

while the teacher and students talked about the

Native American short story, “The Medicine Bag.”

In the story, a Sioux great-grandfather carries on

tradition by passing down the family medicine bag to

his great-grandson. Because this eighth grade class

was part of a K-8 Native American magnet school,

the teacher’s selection of this story seemed like a

good choice for connecting literature to the lives of

students. Multicultural literature has the potential to

help connect the world of the classroom with lives

our students live outside of the classroom. However,

in order for this potential to be realized, classroom

discussions of this literature must create space for

students to connect the literature to their own lives as

a primary means for creating significance and

understanding. For Native American adolescents, the search

for identity is often caught up in conflicting cultural

values between Native American and white society.

Mindful of the tensions this disparity causes in the lives of adolescents, Mary, the classroom teacher,

sought out multicultural literature, such as “The

Medicine Bag,” in an effort to make literature relevant to the lives of her students. She viewed

literature as an opportunity for students to “bring

their own lives” to the texts and to see their lives “in

the context ofa community [i.e., the classroom

community].” As she told me in an interview, she

wanted her students to use literature as a tool for

“express[ing] how they felt.”


Research supports Mary’s perspective and has documented how the use of multicultural

literature can help develop students’ abilities to use

multicultural perspectives and knowledge to think

about literature, society (Miller & McCaskill, 1993;

Rios 1996; Rogers & Soter, 1997), and their own

lives. The diverse perspectives found in the voices

of cultural groups who have been excluded from

literary study can offer “alternative vantage points on

the world” (Greene, 1992). In this view, literature by

and about Latinos/a, African Americans, Native

Americans and Asians can provide teachers with

opportunities for meeting many goals of

multicultural education, where voices interact and

students reflect, think critieally, increase cultural

awareness, decrease ethnocentrism, and develop a

global perspective.

However, as I learned from observing and

listening to the students in this particular classroom,

in order for multicultural literature to engage

students in critical thinking and provide

opportunities to connect classroom life to the lives of

our students, it is not sufficient to simply introduce

multicultural literature into the classroom. We need

to think more deeply and critically about the ways

students’ cultural, social, and community identities

and experiences interanimate with the reading and

teaching of multicultural literature in language arts

classrooms to create meaning. Because students

bring their own experiences and perspectives to the

reading of multicultural literature, using it in the language arts classroom requires attention to not

only what we teach when we select multicultural literature but also to how we teach it. What questions

do we raise for discussion? Do we engage and validate our students’ experiences and prior

knowledge to help shape meaning? Do we find

space for multiple interpretations and understandings

of multicultural literature?

Kristy, a student in this particular eighth

grade classroom, provides an excellent case study for

thinking about these questions and issues. As part of

a research project, I had the opportunity of meeting

with Kristy several times to talk about her

experiences in this English class. As Kristy narrates


her thinking around these experiences, she provides

an important perspective of the multicultural

literature she read in her eighth grade English class

as she connected it to her life.

Kristy At the time of this study, Kristy was a

fourteen-year old Native American girl a member

of the Mohawk tribe which is part of the Iroquois

Nation. Tall, with long dark hair and braces she

dressed simply in blue jeans, tee-shirts, and sneakers.

In her eighth grade English class, she was usually

very quiet and seldom contributed to discussions.

She was, as she told me teachers in her school

expected her to be – “nice, polite, and smart.”

Kristy thought of herself as a poet and, in

fact, she was very proud that a poem she wrote about

her grandmother had recently been accepted for

publication in a New York State student anthology.

She wrote her poem, “Ak Sot Ha, My Grandmother,”

in alternating stanzas of English and Mohawk,

perhaps as representative of her sense of living in

two worlds. Much of Kristy’s time outside of school

was spent at the Long House, and she was taking

lessons to learn to speak and write the Mohawk language. Several years ago, Kristy and her friend

planted a traditional Native garden incorporating, as

she explained it to me, “the old ways” of planting and sowing and together they were learning the traditional Mohawk meaning behind each of the plants.

Clearly, Kristy’s life experiences as a young

Mohawk woman were very significant and a primary social foundation to her sense of self. Because of this, I believed Kristy’s perceptions about the Native

American literature she read in her English class

would be particularly relevant and revealing for what

they might suggest about the role of multicultural

literature in the lives of students of color. In one of

two interviews I had with Kristy, I asked her what

she thought about the Native American stories she

read in English class.

Kristy: Sometimes [they] seem kind

of fake and sometimes real. Fifty

percent. Sometimes I feel they’re making fun of us. They don’t know

any-thing. They’re just talking stuff to

make us look bad. . .. They be

talking stuff they don’t even know

about and it just makes … me feel

bad… , And I feel angry because

they don’t know anything about it

and there’s stuff that wasn’t Native

that I learned. Like there’s always

two sides [to every story].

Kristy challenged the validity of the texts she

read in English class. Her sense was that sometimes

the stories were inauthentic saying that they “seem

kind of fake” and that “they don’t know anything

about it.” Further, these stories left her feeling “bad”

and “angry.” However, my analysis ofthe teaching

of multicultural literature in this classroom suggests

that Kristy’s sense of the stories she read was

directly related to the enacted pedagogy around the

reading and teaching of the literature.

Below is a segment of the discussion of

“The Medicine Bag.” In this classroom excerpt,

students were taking turns reading the story aloud.

Mary would stop the reading at certain points to talk

about what they had just read. This segment begins

with Mary introducing the story, prior to the beginning of the reading.

Mary: How many people in here are learning the Native American traditions? Who are you learning them from?

Kristy: My friends.

Amanda (Mohawk female): My pastor.

Mary: That’s right. You go to aNative American

church. Different cultures are probably so

integrated, we lose culture. Culture is what we eat,

how we dress, our values, the music we listen to.

The Native Americans had to retrieve their culture

because of it being systematically taken away. It’s similar to Black Americans.

Damien (African American male): Kristy, what kind

of music do you listen to?

[Kristy does not answer Damien.}


Paul (African American male): Beating drums. Mary: You have tapes [of Native American music] don’t you? Kristy: Yes. Mary: What’s the stereotype of Native American men? Amanda: TalL Mean. Mary: How does he wear his hair? Amanda; Braids.

Mary: At this age, you get embarrassed about how

parent and family members look and what they wear.

[This is in reference to the boy in the story who is

embarrassed by his grandfather sNative American dress.]

[Mary asks a student to begin to read the story aloud

and the student does so. Paul interrupts the reading

to ask Kristy a question.]

Paul: Do you know what that means Kristy? [Referring to the words “Hau, Takoza” in the story.] Mary: No. Kristy knows Mohawk. This is Sioux. Continue reading Kristy.

[Kristy reads.}

Mary: So do Native Americans show affection in public? [referring to scene in the story where Mom

moves to embrace Grandpa and remembers that public displays of affection are unseemly to the

Sioux]. No. Is that true now? No. Now, Native Americans like to show affection very publicly.

[Mary calls on other students take turns reading


Mary: What might fainting be a foreshadowing of? [referring to scene in story when grandfather faints]

Paul: He’s going to die.

Mary: Be aware that when something like this

happens, it usually signals something else is going to

happen…. Does anyone spend time with their

grandparents? [Many hands go up.] What a

wonderful gift! Why? Tina (European-American female): They’re wiser.


Kristy: You learn about the family. Mary: They pass on traditions, names. Grandparents accept and love you unconditionally. You don’t have to make sure that you do the appropriate thing. Jared (African-American male): They just spoil you. Mary: Jared, continue reading. I’m giving points for reading.

[Jared reads.]

Mary: Note what it says about the oldest male child. Even though it’s a matriarchal culture, this was

something that girls couldn’t participate in. Tina: What if there were only girls? Mary: I don’t know. I’ll have to ask the Native

American teacher upstairs. Are there any customs in your families that you’re uncomfortable with?

[No response.}

Mary: My grandmother prayed the rosary. We weren’t comfortable but we did it to honor her. Sometimes there’s tension. We’d say the rosary with her but not on our own. Although she thought we did.

[Mary reads the rest ofthe story.}

Mary: What do you think of passing customs down only to men? Paul: That’s genderizing (sic) or something.

[Bell rings for the end ofthe period.}

Mary: We’ll talk about that tomorrow.

In this excerpt of classroom discussion, there

were simultaneously several different purposes for

the classroom talk. From the beginning, Mary

attempted to acknowledge some students’ expertise

and personal knowledge by asking who in the class

was learning about Native American traditions. Kristy and Amanda responded that they were

learning from friends and the pastor at church,

respectively. However, rather than asking the girls to elaborate on what they knew, Mary turned to a

definition of culture based on popular conceptions of

culture as food, dress, and music. She then opened

space for a potentially significant and meaningful

discussion when she remarked on the similar ways

both the Native culture and the African American

culture were both systematically “taken away.”

Damien, an African-American boy, perhaps in an

effort to take up the connection that Mary had made

between these two cultures, asked Kristy what kind

of music she listens to. Kristy, though, did not answer to Damien. Similarly, later on, Paul, another

African-American boy, asked Kristy if she knew the

meaning of two of the Sioux words found in the

story. Mary spoke for Kristy when she responded

that Kristy knew the Mohawk language and not

Sioux. Mary then moved on to make a

generalization about Native displays of public

affection and any potential for Kristy, Damien, Paul,

and the other students to engage with multicultural literature in a dialogue that might help them to make

meaningful connections across their own experiences

was lost.

This same missed opportunity occurred

again around their conversation about the role of

grandparents in their lives. Further, at the end of the

conversation, Mary introduced the issue of gender bias in the story and Tina posed an interesting dilemma (“What if there were no girls?”) around that issue which, again, was not taken up during this class session or on the following day.

Given the context for discussing

multicultural literature in this class, it is possible to construe Kristy’s boredom, referred to earlier, as

caught up within the anger she said she felt about the

misrepresentation of Native peoples in the literature

and her inability to articulate what she knew and felt

about Native culture. Stereotypical representations

of Native peoples, such as the description of Native

men that Mary pointed out to the class, were, at

times, unwittingly reproduced in the teacher’s own

talk. For example, during a review of vocabulary

words for a spelling test, Mary used the term “Indian

giver” to help clarify the meaning of the word

“rescind.” So, while Mary did attempt to use the literature to help students think more deeply about

their own experiences and to offer a social critique of

racism, these goals for literature were not elaborated

or consistent throughout her teaching.

Consequently, Kristy was never able to

articulate her own critical and resistant reading of

these texts, based in her own experiences. Recall

that in my conversation with Kristy, she said, “I feel

angry because they don’t know anything about it and

there’s stuff that wasn’t Native that I learned. Like

there’s always two sides [to every story].” The point

Kristy felt so strongly about was that she had come

to understand that the culturally dominant versions

of history she had been taught were not, necessarily,

the only versions or even the correct versions. Here

is the rest of my conversation from my first

interview with Kristy:

Interviewer: Two sides to every story?

Kristy: Yeah! I used to like to read about Abraham Lincoln. But when I heard about what he did to

some of the Native American people, I kind of hated

him then. Like they don’t tell you what he did.

They just tell you the good stuff really.

Interviewer: So how did you find out about what he

did? Kristy: I found out when we were protesting in Albany. We were protesting there and I was listening very closely. There were a lot of people there and it was noisy. And they were saying that they killed thirty-two Native Americans [during the time of Lincoln]> And that half of them didn’t even do

anything. And at the end when they died, they

holded hands together and sang.

Through her experiences outside of school,

she had learned there are cultural and historical

stories about Native people left out of classroom

texts. At fourteen years old, Kristy’s identity as a

young Mohawk woman has begun being shaped by

an evolving awareness of the ways in which her

identity is caught up in the larger sociopolitical

structure. Kristy sensed the missing voices and

perspectives (the other side of the story) in much of


this literature and it angered her and made her “feel bad.”

As a consequence, the teaching of multicultural literature (in this instance, specifically Native American literature), rather than creating

space for multiple meanings and the opportunity to

listen to and learn from students’ experiences,

reinforced the sense of polarized and oppositional

communities school community was set apart from

Kristy’s Native American community. In other

words, there existed a disconnect for Kristy between

her experiences in school and the significance of her

lived experiences outside of school. Ironically, this result was the exact opposite of the teacher’s

intentions for this literature. Recall that Mary said

she taught this literature because of the opportunities

it held for “students to bring their own lives” to the

text and to see their lives “in the context of

community. “

In fact, students’ experiences outside of the

classroom directly impacted meanings they made

from representations in the literature as well as

classroom talk around the literature and the ways in

which they took up and, on occasion, resisted these

representations. For example, Kristy was deeply

involved with her Native way of life outside of

school. She had attended a protest rally in Albany

where she learned that Abraham Lincoln, someone

she used to admire, had been responsible for killing

Native peoples. She told me that her experiences

had led her to believe that the Native American stories she read in English class were “kind of fake and sometimes real. They’re just talking stuff to

make us look bad. And I feel angry because they don’t know nothing about it.” Yet, because there was no space for the articulation of diverse experiences in

the classroom, she had little to say during class. Mary interpreted Kristy’s silence as

reticence. In an interview, when I asked her to

describe Kristy her assessment is very revealing:

Mary: She is very poor. She has a learning disability.

She’s just a typical Native American girl. Very

reticent. But I thought she wrote really


good papers. She really would not articulate verbally. Kristy isn’t very bright.

Implications for classroom practice The teacher in this study understood the

importance of incorporating multicultural literature

into her curriculum as a way to make literature

relevant to the lives of her students. However, she

struggled with how to enable classroom discussion

about the literature in ways that allowed students to

bring their own experiences and understandings into

the exploration of meaning. As a result, she was not

able to achieve the positive goals she had for teaching the literature. In fact, as Kristy’s story

demonstrates, the inadvertent silencing of her voice

and the voices of others contributed to Kristy’s sense

of frustration. This suggests the need to develop specific strategies around the teaching of

multicultural literature that move students toward

connections in their own lives. Such strategies

involve allowing students to take charge of their own

learning and leaving space for them to ask the

questions and entertain the possible responses about

the literature they are studying. In this way, students

“own” the understandings they make about texts in a

way they do not when discussion focuses on the

teacher’s questions designed to elicit specific and

factual information. When students ask the

questions and clarify their understanding, then meaning becomes student-generated while still allowing them to practice their text analysis skills. In the classroom, this can be accomplished in several

ways: 1. Form reader response circles where reading

is acknowledged as a social activity. In such a circle, students can pick one passage they find the most interesting or important,

explain why, and invite others to respond

(Daniels, 2002)

2. Form literature study circles. These are

different from response circles in that the

former is an expression of individual views

and the latter is focused more on

collaborative construction of meaning


around the texts (Cherland, 1994; Daniels,


3. Ask students to respond in writing to a

reading. Some prompts might include what

they liked, what they didn’t like, what

confused them, and what they thought about

particular sections of the text.

Ifwe are better able to understand the many

dimensions of this relationship between multicultural

literature and students’ outside lives, we are better

able to create the democratic classroom we envision

– a place for the articulation of lived experiences and

perceptions of those experiences both in and out of


Works Cited Cherland, Meredith. Private Practices: Girls

Reading Fiction and Constructing Identity.

Bristol, PA: Taylor & Francis, Inc. 1994.

Daniels, Harvey. Literature Circles: Voice and

Choice in Book Clubs and Reading

Groups.2nd ed. Portland, MA: Stenhouse

Publishers, 2002.

Greene, Maxine. “The Passions of Pluralism:

Multiculturalism and the Expanding

Community,” Educational Researcher, 22.1

(1992): 13-18. Miller, Suzanne, and Barbara McCaskill, eds.

Multicultural Literature and Literacies:

Making Space jor Difference. Albany, NY:

State University ofNew York Press 1993.

Rios, Francisco, ed. Teacher Thinking in Cultural

Contexts. Albany, NY: State University of

New York Press, 1996.

Rogers Theresa. and Anna O. Soter, eds. Reading

across cultures: Teaching Literature in a

Diverse Society. New York: Teachers

College Press, 1997.


  • Language Arts Journal of Michigan
    • 2003
  • Making Meaningful Connections: The Role of Multicultural Literature in the Lived Experiences of Students
    • Gina Louise DeBlase
      • Recommended Citation
  • Making Meaningful Connections: The Role of Multicultural Literature in the Lived Experiences of Students

articles summery


Journal of Teacher Education 2017, Vol. 68(3) 262 –279 © 2017 American Association of Colleges for Teacher Education Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0022487117696280 journals.sagepub.com/home/jte


Well-resourced classrooms are beneficial for students and points of pride for schools, but simply having the right “stuff” is insufficient for ensuring learning. Students need time and instruction related to using those resources in pro- ductive ways. Similarly, when it comes to professional development (PD), teachers need the trifecta: adequate tan- gible resources, time to try out and refine instructional tech- niques, and professional content and pedagogical knowledge (Shulman, 1986). However, many teachers with whom we have worked lament that this ideal is far from their reality, echoing Shulman’s conclusion: “There are lots of ideas but no time to figure out how to use them and no materials [e.g., books, lesson or unit plans] to implement”; and “[We] often don’t get to see how the PD will work for us.” In addition, in elementary schools, the PD time and resources available are typically spent on tested content areas (i.e., math and lan- guage arts), leaving little time for PD on untested subjects, such as social studies. In fact, when we asked the partici- pants in this study about PD related to social studies, we were met largely with blank stares—it simply did not exist in their experiences.

For some time, this narrow curricular view may also have been due to the adoption of new language arts and math stan- dards (National Governors Association Center for Best Practices & Council of Chief State School Officers [NGACBP & CCSS], 2010). However, in 2013, leading organizations in science and social studies also released standards documents (National Council for the Social Studies [NCSS], 2013; NGSS Lead States, 2013) that have since been widely

adopted. According to the inservice teachers with whom we work, these standards have simply been handed to teachers, who perhaps received a day of PD on them. Many teachers have been given little support to learn about, teach, and assess four new sets of standards released in the last 6 years—each of which merits ongoing, focused PD.

It is not feasible to focus on one set of standards at a time when all need to be implemented. Engaging in token amounts of PD in each subject area would likely also be ineffective. One approach that could mitigate the drawbacks of these approaches would be to address areas of overlap in the stan- dards, for example, between the social studies content and lit- eracy skills needed to access that content. We cross-walked the social studies and language arts standards documents and saw a unique opportunity “double-dip.” The Common Core State Standards for English Language Arts (CCSS; NGACBP & CCSS, 2010) emphasize the use of literacy skills for the pur- pose of informational text comprehension and creation, par- ticularly in the content areas, whereas The C3 Framework for Social Studies State Standards (C3; NCSS, 2013) focuses on the inquiry arc that can be used to comprehend social studies concepts from a variety of sources, including informational

696280 JTEXXX10.1177/0022487117696280Journal of Teacher EducationBrugar and Roberts research-article2017

1The University of Oklahoma, Norman, USA 2Wayne State University, Detroit, MI, USA

Corresponding Author: Kristy A. Brugar, University of Oklahoma, Jeannine Rainbolt College of Education, 820 Van Vleet Oval #114, Norman, OK 73019, USA. Email: kristy.a.brugar@ou.edu

Seeing Is Believing: Promoting Visual Literacy in Elementary Social Studies

Kristy A. Brugar1 and Kathryn L. Roberts2

Abstract This study addresses the following questions: Does professional development (PD) designed to meet third-, fourth-, and fifth- grade teachers’ pedagogical and content needs influence how teachers teach and engage with graphical devices found in social studies texts? If so, what effect does that instruction and engagement have on students’ comprehension of those devices and social studies reading materials that contain them? We worked with teachers and students in a context-embedded PD series that emphasized literacy skills specific to standards that address accessing and sharing information (Common Core State Standards for English Language Arts) as social studies specialists (e.g., historians, geographers, economists; C3 Framework), with a particular focus on the graphical devices commonly found in social studies instructional materials (i.e., captioned images, maps, tables, timelines). Using teacher interviews, curricular materials, field notes, and student pre- and post-assessments as data sources, we explain the impact of this intervention on teaching and learning.

Keywords practice-based teacher education, teacher knowledge, instructional practices, elementary education, standardshttps://us.sagepub.com/en-us/journals-permissionshttps://journals.sagepub.com/home/jtemailto:kristy.a.brugar@ou.eduhttp://crossmark.crossref.org/dialog/?doi=10.1177%2F0022487117696280&domain=pdf&date_stamp=2017-03-09

Brugar and Roberts 263

texts, and to create new knowledge. The purpose of this article is to explore the impact of a PD model focused on these areas of overlap, closely examining its impact on both teachers’ instruction and students’ learning.

Theoretical Framework

This research is grounded in two interconnected theoretical frameworks: pedagogical content knowledge (PCK; Shulman, 1986) and situated learning (Lave & Wenger, 1991). PCK includes teachers’ understandings of specific content, the ways in which content can best be represented and shared to make it comprehensible to others, and knowl- edge of the most common conceptions and misconceptions that students bring to the content learning (Shulman, 1986). Situated learning posits that we learn best when we have opportunities to practice new skills in authentic contexts, progressing from newcomer to expert through authentic activities within a community of practice (Lave & Wenger, 1991). Together, these theories served to guide our thinking about what teachers need to successfully teach literacy skills in the context of social studies instruction (PCK), and how we might to support them as their PCK, related to particular content and skills, moves from novice to expert.

In the design stage of this study, we began by collecting published, peer-reviewed research related to what students need to know and be able to do to be successful, independent social studies learners. We also interviewed all participating teachers and observed them teaching social studies and lit- eracy lessons to determine aspects of PCK including (a) stu- dents’ understandings and misconceptions, (b) teachers’ understanding of what discipline-specific skills and knowl- edge are needed, and (c) teachers’ disciplinary-specific peda- gogical understandings. Triangulating these points situated our own learning and that of our participants within a par- ticular context and community, and allowed us to identify particular content upon which to focus. With this informa- tion, we were able to design PD to address PCK, specifically centered on teaching students to read the most common graphical elements (i.e., captioned images, maps, tables, timelines) found in social studies resources, an area in which students had several misconceptions and about which teach- ers had very little PCK. To examine the effectiveness of our approach, we asked the following in relation to this particu- lar context:

Research Question 1: Does PD designed to meet third-, fourth-, and fifth-grade teachers’ pedagogical and content needs influence how teachers teach and engage with graphical devices found in social studies texts? If so, in what ways? Research Question 2: If so, what effect does that instruc- tion and engagement have on students’ comprehension of those devices and social studies reading materials that contain them?

Literature Review

Building on the PCK and situated learning frameworks, we draw on three areas of educational research: (a) visual liter- acy and graphical devices, (b) learning of literacy and social studies, and (c) PD. The PD implemented in this study was designed with pedagogy related to visual literacy and graphi- cal device comprehension (content) as a central focus. Furthermore, we worked collaboratively with teachers to design PD grounded in their particular context (e.g., school community, grade, classroom curricula).

Visual Literacy/Graphical Devices

For the purposes of this study, we use graphics and graphical devices interchangeably to describe visual presentations of content with or without textual adjunct, such as captioned photographs, flowcharts, graphs, illustrations/drawings, maps, and tables. These types of devices are prevalent as pri- mary sources (e.g., Lewis and Clark’s maps of North America, photographs of the Dust Bowl), and nearly univer- sal in the text and trade books that serve as secondary sources for students’ learning of social studies (Fingeret, 2012). It is not surprising that the research community recommends that students learn to comprehend both written and visual repre- sentations (e.g., Staley, 2006). Although there is a good deal of research that examines students’ understanding of graphi- cal devices under particular conditions (e.g., Barton, 2002; Pozzer-Ardenghi & Roth, 2004) and their responses to par- ticular instruction or support (e.g., Gersmehl & Gersmehl, 2007; Masterman & Rogers, 2002), there is little research that examines teachers’ preparedness to teach with and about them. So, although we know that it is possible for students to learn more from integrating graphics and print, research does not provide much guidance as to the best ways to prepare teachers to help students do so.

Learning of Literacy and Social Studies

The skills and knowledge needed to comprehend text vary by context, text type, purpose for reading, interest, background knowledge, and topic, among other factors (e.g., Duke, 2005; RAND Reading Study Group, 2002), all of which fluctuate with content area. The skills and knowledge needed to read a social studies textbook, atlas, or historical document—the types of texts common in social studies instruction—differ from the skills needed to read the narrative fiction texts with which elementary students tend to be bombarded (e.g., Duke, 2000; Jeong, Gaffney, & Choi, 2010). It stands to reason that instruction should differ, as well. To be successful, social studies learners must be provided with opportunities to learn appropriate, domain-specific literacy skills (Bain, 2006; Moje, 2007; VanSledright, 2005).

It is well established that readers’ understandings of text- based differences, such as text structures or content-specific

264 Journal of Teacher Education 68(3)

vocabulary, can affect comprehension (e.g., Meyer & Poon, 2001; Williams, Stafford, Lauer, Hall, & Pollini, 2009). However, variations in the literacy skills required for con- tent-area learning go well beyond the written text. For exam- ple, many social studies texts include maps designed to show geographic relationships between events. To effectively comprehend these, readers must understand when and how to access information from the map and successfully inte- grate it with information from the running text. Similarly, readers must be able to use captions to interpret images, tables to explore relationships between data, and timelines to contextualize events. Not only do these graphics help readers to understand the written text, they also provide information that cannot be gained from the written text, alone (Fingeret, 2012). In addition, the ability to accurately interpret these graphical elements of informational text uniquely contrib- utes to overall understanding of the full text (Roberts, Norman, & Cocco, 2015). Therefore, successful social stud- ies learning, in addition to general skills (e.g., decoding and encoding words) and habits of mind (e.g., acknowledging varied perspectives, questioning sources), requires that stu- dents are able to access and share information via a variety of modalities, many of which are distinct to the field.

Professional Development

There is a large body of research on what is likely to lead to effective PD. First, PD should reflect student and teacher needs, which is best accomplished by involving teachers in its planning, development, and implementation (e.g., Hawley & Valli, 2001; Lieberman & Grolnick, 1996). This entails that we understand PD not as something given or done to teachers, but rather something created and implemented with teachers. In addition, PD should be part of a larger plan for collaborative, school-wide change (e.g., Hawley & Valli, 2001), providing an environment in which everyone can sup- port each other in working toward a common goal. Ball and Cohen (1999) note the importance of sustained PD opportu- nities throughout teachers’ careers, a call echoed by others (e.g., Bailey, 2010; Feiman-Nemser, 2001). Furthermore, PD should also be relevant to daily classroom life and embedded across the school day (e.g., Darling-Hammond, 1998; Garet, Porter, Desimone, Birman, & Yoon, 2001). Without such opportunities, teachers are less likely to adapt their teaching to meet ever-changing learning needs of their students and standards of practice.


With this research in mind, we (two researchers) began to design a mixed-methods study to examine a job-embedded PD initiative in which we worked collaboratively with teach- ers to improve their PCK and students’ understanding of social studies informational text. We identified a consistent theme in the analysis of preintervention interviews with the

teacher participants in this study, which was the need to help teachers and students better understand the graphical devices most frequently found in social studies trade and textbooks (i.e., captions, maps, tables, and timelines) as a means to help them better understand social studies content and build infor- mational text reading skills. The teachers understood graph- ics in some ways. They talked around the concept of visual literacy from describing the posters on classroom walls to students drawing pictures across the curriculum, and were able to discuss what graphical devices they used in their classrooms. To this point, Ms. Ruler (all names are pseud- onyms) gave as examples such as “using graphs [and] map- ping.” However, they had more difficulty describing how they were used instructionally or to aid in student compre- hension; thus, these facets of graphical comprehension became the focus of our PD work with the teachers.

Theories of situated learning (e.g., Lave & Wenger, 1991) assert that learning is most effective when it is a part of authentic, context-embedded activity in which one can apply new knowledge to an existing context. Therefore, we posi- tioned ourselves within participating teachers’ classrooms, aligning the content of our PD and model lessons to class- room rhythms and social studies curricula, as well as the coi- dentified pedagogical practices associated with graphics.

Intervention Design

Essential to our intervention model was time for teachers and researchers to collaboratively identify needs, learn from each other through observation and discussion, and put that learn- ing into practice through teaching followed by reflection. Also essential was time frame—the PD in which we engaged focused on one topic (graphical comprehension in social studies texts) over an extended period of time (8 weeks), allowing teachers the time to learn, observe, implement, reflect, question, and implement again (Figure 1).

Working with grade-level teams, we delivered five 20- to 30-min PD sessions. In the first, we discussed content-area and graphical literacy in the context of student assessment results (described below). In each of the next four sessions, we focused on a particular graphical device, its role in textual understanding, and instructional pedagogy. For example, when we conducted the PD on captioned images with fourth- grade teachers, Ms. Rollins and Ms. Ruler, we started with sharing some of the responses their students gave to caption items on the Visual Literacy Assessment (VLA, described below; Brugar, Roberts, & McGuire, 2013; Duke, Roberts, & Norman, 2011), and worked together to identify patterns in student’s understandings and misunderstandings. Then, we defined canonical captions as illustrations or photographs accompanied by words, phrases, or sentences, near (but not within) the graphic, but separate from the running text (Roberts et al., 2013). With this definition in mind, we intro- duced visual thinking strategies (VTS; Yenawine, 2013). VTS entails presenting students with an image and asking

Brugar and Roberts 265

what they see happening within it, and what they saw that made them believe so. We then talked coplanned a social studies lesson using VTS in which their students would cre- ate captions for historical photographs of the Michigan lum- ber industry.

A few days later, we modeled the lesson in one fourth- grade classroom while both teachers observed, after which we debriefed the lesson together. Next, after approximately a week of independent implementation, we observed the teach- ers as they taught about and with the device and debriefed those lessons with each teacher, individually. Teachers then continued to teach with the devices between sessions (cumu- latively, as they were introduced), which were about 1.5 to 2 weeks apart, depending on individual teachers’ schedules. This structure was used for all three grade levels.


This study took place in an ethnically diverse suburban school with approximately 400 students in Grades K-5 (Table 1). All teachers who taught Grades 3 (n = 2), 4 (n = 2), and 5 (n = 3) participated in the study, as well as their collective 173 stu- dents, including 41 focal students (11 third-, 12 fourth-, and 18 fifth graders). Six students per classroom were selected as

focal students, using a random number generator app (one third grader was unable to complete the study). All classes dedicated most of their days to math and language arts instruc- tion. Teachers reported teaching two to four social studies les- sons per week, with the exception of the three fifth-grade teachers, who reported daily instruction (Table 2).

We chose to work with Grades 3 to 5 because demands for independent reading of informational texts, including social studies texts, increase dramatically in the upper elementary grades (Chall & Jacobs, 2003), coinciding with an increase in more consistent social studies instruction (Fitchett & Heafner, 2010). Also, one of our measures required independent read- ing, which would likely have been invalid or extremely frus- trating to complete for any student reading below grade level in a lower grade, resulting in sampling bias.

Data Sources and Analysis

Using mixed methods allowed us to describe processes and outcomes, and investigate possible explanations for each through data triangulation of qualitative and quantitative

Table 1. School Demographics.

Adams elementary

Student population 444 (K-5) White 63% African American 23% Multiracial 4% Hispanic 8% American Indian n/a Native Hawaiian n/a Asian 2% % third graders passing

State Reading Test 59%

% free and reduced lunch 72%

Source. Michigan Department of Education (n.d.).

Table 2. Teacher Demographics.

Teachera Grade Years

teaching Literacy time (all daily) Social studies time

Allen 3 <1 Paul 3 <1 3 hr 30 min/week Rollins 4 26 2 hr 45 min 40 min/dayb

Ruler 4 22 2 hr 45 min 30-45 min,b sometimes 60 min every other day

Gaines 5 14 3 hr (includes social studies and science)

30 min/day

Henderson 5 10 3 hr 30 min every other day

Miller 5 12 3 hr 40-45 min/day

aAll names are pseudonyms. bAlternated teaching science and social studies units during these time frames.

Prior to PD PDa Post PD

A) Interviews (teachers) B) Pre-Assessments (students) -Visual Literacy Assessment (focal) – Passages (all)

1) Direct instruction about graphical literacy or one graphical device 2) Model teaching lesson (researchers) with observation (teachers) 3) De-brief lesson 4) Model teaching (teachers) with observation (researchers) 5) De-brief

A) Post-Assess- ments (students) -Visual Literacy Assessment (focal) – Passages (all)

Figure 1. Overview of job-embedded PD phases. Note. PD = professional development. aRepeated four times.

266 Journal of Teacher Education 68(3)

data (Johnson & Onwuegbuzie, 2004). Data sources and analyses are discussed together here, as analyses of some data influenced decisions related to intervention design and, in turn, the other types of data collected.

Teacher measures. Prior to the beginning of the PD series, all participants engaged in individual, semistructured inter- views, which lasted approximately 20 min. Teachers were asked to reflect on their past PD experiences and the kinds of PD in which they would like to participate, as well as social studies and literacy (including visual literacy) content knowl- edge and classroom instruction. Interview transcripts were subsequently analyzed using open coding to identify catego- ries of participants’ self-identified needs. Detailed results of these analyses are not reported here as they informed the PD design, but do not speak directly to the research questions.

During the 8-week intervention, each researcher spent approximately two full days per week at the research site. We observed five of our seven teachers teaching a social studies lesson prior to the start of the intervention, observed all teachers teaching several lessons during the intervention, and all teachers teaching one lesson postintervention (pre- and postintervention lessons were analyzed for this article). The two fourth-grade teachers in the study were not observed teaching a social studies lesson preintervention because they alternated teaching science with social studies, and were teaching science during the preintervention period. However, we do include all other data points for these two teachers as excluding them would create an artificial rendering of the social studies instruction in this setting, and including them provides important insight into the teaching and learning that occurred at the school.

During each observation, we took descriptive field notes (Bogdan & Biklen, 2006) to document teachers’ instruction

and students’ responses and interactions. Also, we docu- mented each teacher’s use of instructional materials, particu- larly graphical devices (i.e., detailed descriptions of all materials; detailed notes on how they were introduced, dis- cussed, and used). Finally, we photographed or collected and copied all lesson materials.

To analyze the field notes, we used an interpretivist approach (Miles & Huberman, 1994). We first highlighted all references to graphical content (e.g., text illustrations, tables created by students or teachers, maps). Then, using artifacts from the lessons (e.g., student work, photographs of work on the board, textbook pages) as a means of contextu- alization, we worked collaboratively, discussing and ulti- mately agreeing upon scores for each reference based on teachers’ and children’s degree of interaction with the graph- ical content, using a five-point (0-4) scale (Table 3). (Children’s independent interactions were also scored, but not analyzed for the purposes of this article.) The scale is somewhat cumulative; higher numbers on the scale indicate higher level interaction with graphics. For example, if a teacher was scored as a Level 4 for a lesson (creates opportu- nities for students to use graphical devices in a way that helps them to construct or convey meaning), graphics must have been present in the lesson (Level 1) and referenced (Level 2), and may also have been a point of direct instruction (Level 3). Thus, we ultimately labeled each observation at the les- son level, with the highest level of interaction observed. Coded observations were then analyzed within teacher, across time points to determine whether teachers had increased their pedagogical attention to graphical elements of text and facilitated higher levels of student engagement.

Student measures. After identifying content-area literacy skills and graphical comprehension skills as areas of need

Table 3. Teacher Observation Codebook With Examples.

Teacher instruction

Category Definition Example

No evidence (0) There is no evidence of the teacher’s use of graphical devices within the lesson.


Decorative (1) Graphical devices are present, but not a part of instruction. They may serve decorative purposes.

Graphics are present, but not mentioned, in a book the teacher is reading aloud

Reference (2) The teacher references a graphical device, but does not engage in instruction or use of the graphic to create or convey meaning.

Teacher: If you draw pictures, label it so I know what that is. Words and pictures!

Teach (3) The teacher explicitly instructs about a graphical device.

Teacher: What did we make here? Gestures to the table

Student: a comparison Teacher: yes, a graph, or chart, or table. What

is it called? The title is “Everyday Life” so it is called Everyday Life.

Construct or convey meaning (4)

The teacher creates opportunities for students to use graphical devices within the/a lesson in a way that helps them to construct or convey meaning.

Students create table to share information related to the food and homes of Native American tribes

Brugar and Roberts 267

based on initial teacher interviews, all participating students were given one of two versions of a pre- and postassessment we refer to as “passages” (randomly assigned and counterbal- anced within classroom). Each passage included approxi- mately 200 words, timelines, maps, captioned pictures, and tables (see example, Appendix A). We selected social studies topics that do not appear in the state curriculum until middle school, increasing the likelihood that students would need to attempt to gain information from the articles, not by relying on prior knowledge. Prior to giving the students the passage, we administered a Vocabulary Recognition Task (VRT; adapted from Stahl, 2008) to gauge students’ prior knowledge of the content (see example, Appendix B) and to use as a variable in analyses, if necessary. The results of the VRT indicated that students did not have significant prior knowledge on these top- ics, thus, the measure was not used beyond this point in the study. After or while reading a passage (at their discretion), students responded to 12 questions, eight of which are included in the analyses for this study. Each of the eight questions focused on content from the written text or graphics, in equal proportion (Appendix A). Written text, graphics, and compos- ite scores were computed to provide insight as to the contribu- tion of each type of question to composite scores and relative to each other. Means for each score type were also compared pre- and postintervention using nonparametric statistics.

In addition, 41 randomly selected focal students (five to six per classroom) were assessed one-on-one using the VLA (Brugar et al., 2013; Duke et al., 2011) both pre- and postint- ervention. This assessment consists of a series of open-ended questions in which children identify, interpret, and interact with four types of graphical devices commonly found in social studies trade and textbooks (i.e., captioned images, maps, tables, and timelines). Maps tasks were not included in the original assessment design, but were added for the pur- poses of this study to more accurately reflect the graphical content of social studies texts designed for students in Grades 3 to 5. At the time this study was conducted, there were no published assessments of visual literacy for this age group available for purposes of conducting concurrent validity. The authors of the original measure minimized threats by draw- ing on previous research and pilot work to design the tasks. In addition, they used authentic children’s texts to increase ecological validity, as well as age-appropriate language in

questions and follow-up prompts. Finally, they designed multiple tasks for each concept. These multiple tasks pro- vided an opportunity to triangulate the data and lessen the possibility of one item disproportionately affecting the results (Duke et al., 2013). These procedures were replicated in the creation of the maps tasks.

During the assessment, students were shown a variety of authentic texts and asked specifically about individual graphical devices. For each of the four devices assessed, stu- dents were first shown an array of examples of the device and asked what they were called and what they were for. Then, students were shown several individual instantiations of the devices and asked to extract information (e.g., “What country is directly east of Pakistan?”) or perform a task (e.g., determine the distance between two locations on a map). Responses were subsequently assigned short descriptive codes, each of which was assigned a score (0, 1, or 2) indicat- ing the level of understanding it indicated (no demonstrated understanding, partial, or full). Two trained coders scored all responses using and refining an existing codebook for cap- tions, tables, and timelines. Then, researchers created codes for maps to be included in the existing codebook based on the first 25.00% (n = 10) of the transcripts, and used the codebook to code the remaining responses. Raters were in agreement on 92.31% of all level-of-understanding codes; all disagreements were discussed and resolved prior to statisti- cal analysis. Scores were recorded for each of the four devices, as well as for the total score and ability to name the devices (recorded separately). Finally, within and across grade levels, we calculated percentage scores (points earned/ points possible) for the naming of all devices, use of each device separately, and composite scores for using all devices. We then compared pre- and postintervention mean scores in each category using a (nonparametric) Wilcoxon signed- ranks test (Table 4). The results of these analyses informed our decisions as to the particular content to address in each PD session and subsequent model lesson.

Teachers Teaching and Learning: Intervention Lessons

In the following sections, we profile lessons for each graphi- cal device addressed during the intervention. For each device,

Table 4. Comparison of Visual Literacy Assessment (VLA) Scores.

Graphical comprehension category Preintervention M (SD) Postintervention M (SD) Change Z-score Significance (two-tailed) Effect size

Captions 65.00 (17.03) 72.64 (20.56) +7.64 2.78 .005 .31 Maps 60.85 (22.66) 65.82 (20.19) +4.97 1.49 .136 .16 Tables 64.15 (26.46) 70.00 (22.80) +5.85 1.92 .055 .21 Timelines 72.26 (24.92) 81.10 (25.02) +5.84 2.65 .008 .29 Total 65.56 (19.11) 72.39 (18.32) +6.83 3.73 .000 .41 Total naming 47.07 (18.74) 54.63 (21.22) +7.56 2.98 .003 .33

268 Journal of Teacher Education 68(3)

we first describe a lesson that we modeled for a grade-level team, and then profile one teacher at that grade level’s early attempts at instruction implementing PD content and model lesson pedagogy. Although for each device, we only describe lessons at one grade level, we taught model lessons for each device at each grade level and observed all teachers doing the same; teachers also taught lessons using the devices when we were not present. As a point of context, we also open each section with the preintervention VLA assessment results that informed our decisions regarding the focus of our instruction (reported across grades unless there were significant between-grade differences). Although the passage results generally indicated a need to address graphical content, we found the results of the VLA more useful in our planning as they highlighted particular patterns of misconception.

Captioned images. Very few children were able to accurately name (9.76%, n = 4) or describe the purpose (24.39%, n = 10) of captions. When asked to create canonical captions for two images, 22.73% of third graders’, 33.33% of fourth graders’, and 30.55% of fifth graders’ captions were canoni- cal (defined here as a phrase or sentence(s) that “describe[s], comment[s] on, or provide[s] additional information related to the graphic”; Duke et al., 2013, p. 16). The next two tasks required children to recognize and select the best of four pro- vided captions for an identified image in a trade book. With the exception of two fourth graders, all students were suc- cessful on at least one trial; 73.17% (n = 30) were successful on both.

Although children were generally aware that captions should relate closely to images, they were unclear or incon- sistent regarding the purposes of captions and canonical phrasing (e.g., C3, Dimension 2, Geographic Representations; CCSS, Reading Informational Text Standard 7, Integration of Knowledge and Ideas). Consequently, we designed a les- son in which we guided students through the process of cap- tioning historical photographs of Michigan’s lumber industry, the current topic of study in the class.

First, we engaged Ms. Ruler’s students (all names are pseudonyms) with a lumber camp image using VTS (i.e., Yenawine, 2013). We then explained that authors or photog- raphers may include a caption to communicate their ideas, and gave direct instruction on the purposes of captions and their common characteristics. Next, we guided students through drafting a caption for the image, using the descrip- tions they had created in combination with what they thought the photographer might have wanted us to know, and then revised according to the criteria for content and length. Finally, we gave students time to engage in this process in pairs and independently. We ended by sharing a few of the newly captioned images and reviewing the purpose and structure of captions.

Following the model lesson and debrief, we, and Ms. Ruler, observed a lesson in which Mrs. Rollins displayed historical photographs associated with the daily life of lumberjacks and

engaged in the VTS process with her students, asking them to make observations, provide evidence, and think more about the photographs. During this lesson, Mrs. Rollins went beyond “call and response” types of interactions. Rather, she asked students to explain their answers, provide further details asso- ciated with each photograph, and record their understandings in the form of captions. When we debriefed this lesson, both fourth-grade teachers noticed that, although the students had a good grasp of length and content, they were having a difficult time distinguishing between conversational and the “book” language typically used in captions. Accordingly, they identi- fied this as a topic for a subsequent lesson.

Maps. Most students, 80.48% (n = 33), were able to identify maps by name. When students were asked to describe the purpose(s) of four maps taken from social studies texts, third graders had quite a bit of difficulty doing so (18.18%, n = 2), whereas fourth (50.00%, n = 6) and fifth graders (77.78%, n = 14) fared much better. Next, we asked children to deter- mine the purpose of a map of South America with a promi- nently displayed, color-coded key titled “environments.” Just over a third (39.02%, n = 16) of students responded with an accurate response (e.g., “to tell you what the environment is in different parts”); of those who did not, 39.02% (n = 16) were able to describe the purpose of the key when asked explicitly. When students were asked to “point to a desert” (presumably by using the key) on a map of South America, they were largely successful across all three grades (third grade, 90.90%, n = 10; fourth grade, 75.00%, n = 9; fifth grade, 100%, n = 18).

Next, children were given a map that included a scale and asked to explain how they would determine the distance (in miles) between two cities. We then identified and explained the purpose of the scale, provided students with a ruler and string, and asked them to try to determine the distance. In the full sample, 36.50% (n = 15) were able to describe a feasible process; 21.95% (n = 9) were able to make a reasonable attempt at implementing that process. In addition, we gave students a map that featured a compass rose (though we did not point it out) and asked them to identify “the country directly east of Pakistan.” The majority of students across all three grades (61.00%, n = 25) inaccurately identified “Bangladesh” or other countries to the east, but not directly east of Pakistan. Finally, students were shown a compass rose and asked what it was called and to describe its purpose. Many students were able to name the compass rose (80.49%, n = 33), whereas slightly fewer were able to describe its pur- pose (68.29%, n = 28).

In both fourth-grade classrooms, students frequently com- pleted Daily Geo (Johnson, 2004) worksheets. We included this in our model lesson plan to maintain the classroom rou- tine and the teachers’ objectives for the day. However, as we reviewed the students’ answers to the day’s worksheet, it became clear that students were not utilizing symbolic infor- mation, even though a legend was prominently featured on

Brugar and Roberts 269

the map. Rather, children appeared to be guessing, and when pressed as to how they arrived at their answers, either stated that they did not know or that they “just looked at the words on the map.” This was consistent with the finding that few children effectively used the map legend on the VLA. Because we knew that students would continue to do Daily Geo worksheets, we decided to switch gears and model how teachers could facilitate that work in ways that would help students understand how to strategically read maps for infor- mation within a Daily Geo Lesson (e.g., C3, Geographic Representations; CCSS Writing Standard 9, Research to Present and Build Knowledge).

We introduced the terms legend/key, symbol, and label to help the children better understand the types of conceptual information available on the map. Then, we went through the five Daily Geo questions together, guiding students through the thought process for utilizing the map features to answer each question as they followed along on their own maps.

In subsequent lessons, we observed each of the fourth- grade teachers displaying and annotating the maps associ- ated with the Daily Geo and reinforcing map vocabulary (e.g. labels, symbols), not only discussing what the correct answers were but also modeling or asking students to describe how they arrived at them.

Tables. No student correctly named tables, rather they identi- fied the tables as “graphs,” “charts,” or simply “informa- tion.” When asked to describe the purpose(s) of tables, 9.09% (n = 1) of third, 25.00% (n = 3) of fourth, and 38.89% (n = 7) of fifth graders were able to demonstrate a complete under- standing, making statements such as “organizing data” and “listing different things and giving you information about it at the same time in a short answer.” Students were then shown two tables (one at a time) and asked to describe what each showed. Averaging across two tasks, 36.36% of third- grade, 62.50% of fourth-grade, and 83.33% of fifth-grade responses accurately described the information conveyed in the tables. Finally, students were shown two tables (one at a time) and read the titles and column and row headings for each. They were then asked to use each table to answer a question. For these tasks, 54.55% (n = 6) of third graders, 33.00% (n = 4) of fourth graders, and 66.67% (n = 12) of fifth graders accurately identified both correct answers.

With this in mind, our lesson in Ms. Allen’s third-grade classroom on push/pull factors associated with the Michigan economy included a table to enhance understanding. First, we defined “push factor” and “pull factor” (C3, Dimension 2, Economic Decision Making; CCSS Language Standard 4, Vocabulary Acquisition and Use) and identified common examples of each, which we documented on a table. We then transitioned to reading segments of the textbook describing why various immigrant groups left their home countries (push factors) and came to Michigan (pull factors). Students worked in small groups, each focused on a passage about a particular group of immigrants, to create a one-row table

with the column headings “Country,” “Push Factor,” and “Pull Factor” (D3.1.3-5, Gathering and Evaluating Sources). We then combined the rows to create a master table of push and pull factors associated with several countries and guided students through reading it (e.g., looking for similarities, locating particular information). The table was later copied and inserted into their textbooks as a point of reference for the remainder of the unit.

Subsequently, we observed the use of tables across the curriculum in both third-grade classrooms. For example, Ms. Paul’s students created a table documenting examples of things imported to and exported from Michigan (Figure 2). When the table was complete, the class reviewed the infor- mation in each column and definitions of import and export.

Timelines. Many students were able to accurately name time- lines (41.46%, n = 17). When asked to describe their purpose(s), 18.18% (n = 2) of third graders, 33.33% (n = 4) of fourth graders, and 50.00% (n = 9) of fifth graders were able to do so accurately. The children were then shown two timelines (separately) and asked to describe the information depicted. Across these two examples, students were able to accurately describe the information in 62.20% of their responses (40.91% of third-grade, 50.00% of fourth-grade, and 91.18% of fifth-grade responses).

With this in mind, we modeled a lesson using chronologi- cal order and timelines in Ms. Gaines’s fifth-grade class- room. Previously, the students had completed a unit about the development of The Constitution and The Bill of Rights. We planned a lesson to help students understand that the causal relationships between pre-American Revolutionary events and these documents (D2.His.5.3-5. Explain connec- tions among historical contexts and people’s perspectives at the time; CCSS, Key Ideas and Details, Standard 5).

We opened the lesson by reviewing The Bill of Rights— reminding students these are the first 10 amendments, or changes, to The Constitution. Next, we put students into small groups and gave them seven event cards associated with the American Revolution and related to the rights of the people. The students were prompted to read the cards and put them in the order in which they occurred, first based on memory, then checking their answers by using their textbook. Then, each group contributed one card to construct a whole-class chro- nology at the front of the room. We then explained that it was easier to understand how far apart events occurred if we could


Import Export

Definition Things brought in Things sent out

Examples Bananas Oranges Clothes

Cars Apples Corn

Figure 2. Table completed by students.

270 Journal of Teacher Education 68(3)

see the relative distance between them. Subsequently, we converted the chronology to a timeline with even increments of 5 years per foot from 1755 to 1775.

Next, we explained that creating this timeline helped us see what was happening during the era leading up to the cre- ation of The Constitution. Then, we posed the question, “What are some of the events that led to the need for the Bill of Rights?” We proceeded to read each amendment and stu- dents made assertions about the events on the timeline that may have influenced the development of each. This was the final social studies unit for the year; thus, teachers did not have an opportunity to implement instruction around this device.


Teacher Implementation

Our teacher participants made it clear that they felt they had a tenuous grasp on some of the knowledge and skills they were expected to teach, which they attributed to little prepa- ration/support, changing curricular expectations, and fre- quent changes in teaching assignments. Perhaps, because of this lack of content knowledge, teachers had little idea of what common understandings, misconceptions, skills, and knowledge students might have in relation to that content, any of which might have been impeding student learning. Thus, it was understandably difficult, for teachers to identify appropriate pedagogy to meet the needs of their students in relation to the standards they were tasked with teaching; this study sought to address this problem.

As is the case in most PD initiatives, for a variety of pos- sible reasons (e.g., interest in the topic, confidence, comfort level with new pedagogy), the degree of uptake of the PD varied by teacher. In the following sections, we discuss teachers who had high (i.e., had at least one example of Level 4 teacher interactions or increased their highest level of inter- action in an observation by at least two levels from pre- to postintervention) and low (i.e., highest level of interaction increased by less than one level or decreased) uptake.

High uptake. Five teachers demonstrated a high level of uptake: Ms. Allen and Ms. Paul (Grade 3), Ms. Ruler (Grade 4), and Mrs. Henderson and Ms. Miller (Grade 5).

Ms Allen. In our initial observation, Ms. Allen did not teach about graphical devices or elicit engagement with them to help children construct or convey meaning. However, she did reference the graphics as she quizzed the students on fac- tual information (Level 2, reference), as in this discussion of a projected map:

Teacher: Which two rivers flow into the Atlantic Ocean? Child: [inaudible, presumably says “Ipswitch”]. Teacher: That should be written down for number one.

Ipswitch (writes on overhead).

Postintervention, we observed Ms. Allen guiding her stu- dents through a lesson in which they created a table on the division of labor between males and females in lumberjack camps, discussing both the content and the structure of the table (e.g., asking what the table was about, talking explicitly about the location and purpose of the title; Level 3, teaching about a graphic). She also guided the children to use the table to make inferences about what their own roles would have been in that context (Level 4, purposefully engaging children with graphics to construct meaning).

Ms Ruler. Although we did not have an opportunity to observe Ms. Ruler teaching social studies preintervention, we consider her level of uptake to be high as she engaged in instruction in the postintervention observation that reached a Level 4. Ms. Ruler guided her students to create a table showing the relationship between resources and goods. She began by referencing a table they had created the day before about capital, natural, and human resources (i.e., each resource type was a column on the table). During the observation, students identified examples of these resources and filled in a new table linking the resources they selected to goods that could be produced from them. As students worked, Ms. Ruler capitalized on teachable moments. For example, one student brought up the idea that each resource might be used to make multiple goods, and Ms. Ruler talked them through how they could represent that in their tables (Level 4, purposefully engaging children with graphics to convey meaning).

Ms Henderson. In her preintervention observation, Ms. Henderson did a think aloud in which she modeled using pictures in a biography of J. K. Rowling that she had already read to remind herself of what she had learned (Level 3, teaching about a graphical device). She modeled looking back for information on her education, stating, “Look at this. I see ‘University of Exeter’ [on a sign in this picture]. Does that sound like education?”

During her postintervention observation, she incorporated graphical literacy instruction at a higher level, teaching a les- son that was very close to the timelines lesson she had observed in Mrs. Gaines’s room. However, she extended the lesson by leading her students through a discussion, using the timeline as a tool, of what they would say in response to the question, “What events led up to the creation of the Bill of Rights”? (Level 4, purposefully engaging children with graphics to construct or convey meaning).

Ms Miller. In our preintervention observation, Ms. Mill- er’s students were also working on reading biographies and taking notes. Their task was to highlight their (photocopied) biographies, using three different colors to indicate three different types of information. Although graphics were not incorporated in this lesson in a conventional way, the chil- dren had been asked to use a color-coded key to categorize

Brugar and Roberts 271

information. As the students worked, Ms. Miller reminded them several times to use the key (Level 2, referencing graphics) to make sure they were using the correct colors.

When we returned for the postintervention observation, Ms. Miller was leading her class through a modified version of the VTS process that she had observed in Ms. Gaines’s room. The students had photocopies of four images, and for each, she asked them to describe what they saw and then helped them to interpret it “correctly,” as in the following exchange:

Teacher: I want you to look at the picture and try to figure out what’s going on. [Displays John Trumbull’s paint- ing, “Declaration of Independence”] on the overhead.] [Reading] “In the painting: [Thomas] Jefferson and other committee members presented the Declaration of Independence to the Continental Congress.” Take a look at page 126 in your book. [Reading] “After the Continental Congress approved the Declaration of Independence, a handwritten copy was prepared for the delegates to sign.” Does it look like they are pre- senting it?

Child 1: No, because there are papers on the ground . . . This scene takes place before July 4th, 1776 because—

Child 2: They are rough drafting!

She proceeded with additional images, asking students to describe what they saw and then infer what those pictured might be thinking or saying (Level 4, purposefully engaging children with graphics to construct meaning). Although not a prototypical VTS lesson, Ms. Miller made a clear and suc- cessful attempt to help children make inferences by integrat- ing the graphics and text (captions).

Low implementers Ms Paul. In our preobservation in Ms. Paul’s class, stu-

dents were engaged in a lesson she described as interdisci- plinary (language arts and social studies). The students were writing a story as if they were a traveler to a Michigan pio- neer village; no graphical devices were used (Level 0).

In our postobservation, Ms. Paul opened the lesson by assigning parts of a chapter in the textbook to small groups of children, and tasking them with creating a poster to teach other students in the class the information in their section. Although there was no instruction associated with the cre- ation of these posters, as Ms. Paul circulated among the stu- dents she did refer to graphics (Level 2), reminding students, “If you draw pictures, label it . . . Words and pictures!” Later in this lesson, she shared a completed poster and explicitly pointed out the positive attributes, including that it included a title, words describing the title, and pictures with labels.

Ms Rollins. We were not able to observe a preintervention lesson in Ms. Rollin’s classroom; thus, we cannot speculate

on her growth over time with regard to inclusion of graphics in her social studies instruction. At the time of the postint- ervention observation, Ms. Rollins was integrating graphics into her instruction, though not in ways that reflected PD learning. In her postintervention observation, Mrs. Rollins tasked her students with filling in a “double-bubble” mind map, placing vocabulary words only pertinent to the lumber industry in the 1900s in the left-most bubbles, only pertinent to today’s society in the right-most bubbles, and pertinent to both in the middle. After providing the directions, she circu- lated and pointed out when information was placed correctly or incorrectly, and in the case of the latter, to which bubble it should be moved. Graphics (the mind map) were referenced in this lesson (Level 2), but without related instruction or facilitating their use to create or convey meaning.

Student Learning

We first analyzed students’ pre- and postintervention responses on the VLA using nonparametric statistics to determine whether students were better able to name and understand individual graphical devices postintervention when asked about them, explicitly. The mean preinterven- tion total score was 65.56 on a 100-point scale, which increased to 72.39 postintervention (statistically signifi- cant at p < .001). Also, students showed statistically sig- nificant improvement at p < .01 for captions, timelines, and naming subscores. Growth in scores for maps and tables was positive, but did not reach statistical signifi- cance (Table 4).

We then analyzed pre–post differences in passage scores to determine whether children’s abilities to apply their under- standing of the instructed graphical devices in authentic reading situations had improved (i.e., transfer). Results indi- cated that there were no statistically significant differences between time points on graphic (M = 4.11, SD = 2.87 for Time Point 1; M = 4.07, SD = 2.88 for Time Point 2) or total scores (M = 9.67, SD = 5.00 for Time Point 1; M = 9.35, SD = 4.77 for Time Point 2). Interestingly, there was a small, but significant difference favoring Time Point 1 on text scores (M = 5.56, SD = 2.60 for Time Point 1; M = 5.28, SD = 2.48 for Time Point 2).


Our research provides a description of the type of collabora- tive (university–elementary school), embedded intervention that research suggests is most likely to influence teacher practice and, in turn, student outcomes. It is also reflective of the existing research on visual literacy that emphasizes the importance of graphical devices to understanding elemen- tary-level texts. This study furthers both of these lines of inquiry by presenting a context-embedded model for PD to

272 Journal of Teacher Education 68(3)

increase teachers’ and students’ purposeful engagement with graphical devices, and in turn, students facility with skills integral to the English Language Arts (NGACBP & CCSS, 2010) and social studies standards (NCSS, 2013).

Although the relationships between standards, teaching, and learning the application of visual literacy skills to com- prehension may seem natural given the amount of graphical material in text and trade books, for the elementary school teachers with whom we worked, this was not always the case, even with the support of PD. It is possible the differ- ences in uptake were related to factors internal to the par- ticipants such as interest or efficacy. However, given the data from teacher interviews, it is also likely due, at least in part, to the fact that these teachers (early career and vet- eran) had limited preparation and PD to support PCK in the areas of content-area literacy and social studies. Yet, stan- dards for both English Language Arts and social studies call for students to engage with visual elements of text in critical ways.


In an era of decreased time for elementary social studies instruction (Fitchett & Heafner, 2010) and increased expec- tations related to elementary students’ reading of informa- tional texts, this study expands the ways in which teachers may view elementary social studies instruction and the resources/points of access for students to acquire social stud- ies content. Specifically, this work addresses how we might support teachers and students to use the informational text comprehension skills called for by the CCSS–ELA to con- struct meaning from social studies texts. In addition, this study expands cross-curricular social studies/history and lit- eracy intervention research, which most often focuses on middle and high school students (e.g., De La Paz, 2005; De La Paz & Felton, 2010; Monte-Sano, 2010), to include research on younger students.

This research also has practical implications for instruc- tional leaders and teacher educators. First, we need to be sure that the preservice and inservice teachers with whom we work are aware that graphical comprehension plays an important role in comprehension of informational text (e.g., Roberts et al., 2015), and is thus worth instructional time. Then, we need to be sure that teachers have the PCK that will allow them to read and create these types of devices with their students. Currently, the CCSS is driving a strong focus on print literacy and reading across and within the content areas, but we need to acknowledge in our instruction that nearly every trade and textbook that students are given to read in the content areas is replete with graphical devices, and that simply being able to see these graphical devices does not ensure comprehension. Rather, like print-based reading skills, we need to be sure

that classroom teachers understand how to teach students to read the graphical aspects of text. In addition, the texts used by children in all grades include graphical devices, and the standards associated with all grades call for related instruction. Thus, sustained and scaffolded instruction and practice for students is essential across their academic careers.

In response, teacher educators need to work with teachers, inservice and preservice alike, to create opportunities to learn about graphical comprehension instruction. These opportunities would vary depending on each teacher’s PCK but may include readings, modeling instructional strategies, or providing critical feedback on instructional ideas or prac- tice. We also need to engage in sustained work around the topic, as opposed to the more common “one and done” approach to PD. Our relatively short, but intensive, interven- tion with teachers and students was enough to contribute to statistically significant gains on the VLA (which entailed direct prompting to use graphics), whereas transfer to inde- pendent construction of knowledge on the passage assess- ment continued to pose difficulties. This may have been because students and teachers alike simply needed more time, more support, and a more gradual release of control. However, the fact that scores increased at all over such a short time period indicates that some facet of the intervention was likely effective because maturation is an unlikely factor given the time frame.


As with any study, there were compromises to be made, which resulted in four notable limitations. First, there was the potential for response bias. The participating teachers were aware of our interest in graphical comprehension as a possible PD topic based on preintervention interviews, and, subsequently, the consent forms we sent home with the chil- dren that described our assessments of graphical compre- hension, which the teachers saw. Therefore, it is possible that some teachers may have given more attention to the instruction of graphical devices in the preintervention obser- vations than they might have under other circumstances.

Second, our intervention lasted 8 weeks at the end of the school year (April and May). Although it was more intensive than is the norm, we would have liked a full academic year to release our level of support more gradually and allow teach- ers more time to implement new learning with each device, observe and give feedback to each other, and refine their practices to best meet the needs of their students. With these time constraints, we were only able to observe one pre- and one postintervention lesson, as scheduling additional obser- vations would have shortened the intervention period. It is possible that one or more of the lessons we observed were anomalies, not representative of teachers’ typical practice.

Brugar and Roberts 273

On a related point, we did not have the opportunity to follow these teachers and students into the next year to see whether the intervention “stuck,” influencing teaching and student skills and knowledge.

Third is the issue of replicability: The time and resource commitment required for this model PD experience was significant for teachers and researchers. While the oppor- tunities these teachers had to observe, as a group, as we or one of their colleagues taught a lesson was invaluable to their learning and ours, they also required juggling of classroom schedules and teaching assistants, writing plans to be implemented in their absence, and all the other logis- tical issues that arise when a teacher is out of the class- room. Also, to embed the PD in everyday classroom life and the curriculum, we met with teachers in grade-level teams to provide the PD sessions and plan the lessons, put- ting the researcher to teacher ratio at 1:2-3, which if scaled up to the district or building level, would be resource prohibitive.


This study highlights a successful approach to PD that addresses the academic needs of a teacher, and facilitates stu- dent growth in the use of graphical devices and social studies knowledge, as called for in standards documents for both con- tent areas. This is important in the face of a persistent achieve- ment gap in indicating that children in the United States are generally not as adept at reading informational texts of the types used in this study as they are at reading narrative text.

Taking a step back from this particular PD, this study also serves as a model for one way in which we might meet the recommendation that successful PD be contextually bound (Garet, Porter, Andrew, & Desimone, 2001). Removing some of the demand that exists in traditional PD for teachers to learn new skills in isolation from the content to the context in which they would apply them may increase levels of imple- mentation by increasing self-efficacy and perceived value of the instructional model.

274 Journal of Teacher Education 68(3)

Appendix A Japan Passage and Questions.

Brugar and Roberts 275

276 Journal of Teacher Education 68(3)

Brugar and Roberts 277

278 Journal of Teacher Education 68(3)

Name: —————Teacher(s):—————

1. Japan has had problems with three kinds of natural disasters:

––––––––––––, ––––––––––, and –––––––––. 2. At the end of World War II, the United States bombed

the Japanese cities of ––––––––and –––––––––––. 3. In the 1500s, Japan traded with ––––––––––– and

––––––––––––––––. 4. The most Japanese-born baseball players debut in

Major League Baseball during the –––––––––––––––. 5. Rice and vinegar and raw fish is called

–––––––––––––. 6. In the year –––––––––––––––––– Buddhism was

introduced in Japan. 7. ————————a. over 70% of Japan. 8. Kanji are––––––––––––––––––.

Appendix B

Vocabulary Recognition Task—Japan

Below you see a list of words. Put a circle around the words you that you are able to read and have something to do with Japan. Do not guess because wrong answers will lower your score.

Correctly chosen targets or hits (H). Incorrectly chosen foils or false alarms (FA). Proportion (P) known calculation: C (P[H] – P [FA]/ 1 − P

[FA] = P[K]).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.


The author(s) received no financial support for the research, author- ship, and/or publication of this article.


Bailey, L. B. (2010). The impact of sustained, standards-based pro- fessional learning on second and third grade teachers’ content

and pedagogical knowledge of integrated mathematics. Early Childhood Education, 38(2), 123-132.

Bain, R. (2006) Rounding up unusual suspects: Facing the author- ity hidden in the history classroom. Teachers College Record, 108(10), 2080-2114.

Ball, D. L., & Cohen, D. K. (1999). Developing practice, devel- oping practitioners: Toward a practice-based theory of profes- sional education. In G. Sykes & L. Darling-Hammond (Eds.), Teaching as the learning profession: Handbook of policy and practice (pp. 3-32). San Francisco, CA: Jossey-Bass.

Barton, K. C. (2002). “Oh, that’s a tricky piece!”: Children, medi- ated action, and the tools of historical time. The Elementary School Journal, 103(2), 161-185. Retrieved from http://www .jstor.org/stable/1002233

Bogdan, R., & Biklen, S. K. (2006). Qualitative research in educa- tion: An introduction to theories and methods. Upper Saddle River, NJ: Pearson.

Brugar, K., Roberts, K. L., & McGuire, D. (October, 2013). Maps, timelines, and tables: Reading graphics in the social studies. Paper presented at the annual conference for the Michigan Council for Social Studies, Lansing, MI.

Chall, J. S., & Jacobs, V. A. (2003). Poor children’s fourth-grade slump. American Educator. Retrieved from http://www.aft .org/newspubs/periodicals/ae/spring2003/hirschsbclassic .cfm

Darling-Hammond, L. (1998). Teacher learning that supports teacher learning. Educational Leadership, 55(5), 6-11.

De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in culturally and academically diverse middle school classrooms. Journal of Educational Psychology, 97, 139-156.

De La Paz, S., & Felton, M. K. (2010). Reading and writing from multiple source documents in history: Effects of strat- egy instruction with low to average high school writers. Contemporary Educational Psychology, 35, 174-192.

Duke, N. K. (2000). 3.6 minutes per day: The scarcity of infor- mational texts in first grade. Reading Research Quarterly, 35, 202-224.

Duke, N. K. (2005). Comprehension of what for what: Comprehension as a non-unitary construct. In S. Paris & S. Stahl (Eds.), Children’s reading comprehension and assess- ment (pp. 93-104). Mahwah, NJ: Lawrence Erlbaum.

Duke, N. K., Norman, R. R., Roberts, K. L., Martin, N. M., Knight, J. A., Morsink, P. M., & Calkins, S. L. (2013). Beyond con- cepts of print: Development of concepts of graphics in text, prek to grade 3. Research in the Teaching of English, 48, 175-203.

Duke, N. K., Roberts, K. L., & Norman, R. R. (May, 2011). Young children’s understanding of specific graphical devices in informational text. Poster presented at the annual meeting of the International Reading Association, Orlando, FL.

Feiman-Nemser, S. (2001). From preparation to practice: Designing a continuum to strengthen and sustain teaching. Teachers College Record, 103(6), 1013-1055.

Fingeret, L. (2012). Graphics in children’s informational texts: A content analysis (Doctoral dissertation). Michigan State University, East Lansing, MI.

Fitchett, P. G., & Heafner, T. L. (2010). A national perspective on the effects of high-stakes testing and standardization on

Anime Kanji Sushi Archipelago Kobe Tapas Beijing Manga Tokyo Cyrillic Manufacturing Tornados Earthquakes Mount Fuji Trade Electronics Napoleon Wheat farming Gobi Desert Rice farming Woodblock Haiku Subway World War II Kabuki

Italics = foils.http://www.jstor.org/stable/1002233http://www.jstor.org/stable/1002233http://www.aft.org/newspubs/periodicals/ae/spring2003/hirschsbclassic.cfmhttp://www.aft.org/newspubs/periodicals/ae/spring2003/hirschsbclassic.cfmhttp://www.aft.org/newspubs/periodicals/ae/spring2003/hirschsbclassic.cfm

Brugar and Roberts 279

elementary social studies marginalization. Theory & Research in Social Education, 38(1), 114-130.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effec- tive? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.

Gersmehl, P. J., & Gersmehl, C. A. (2007). Spatial thinking and young children: Neurologic evidence for early development and “educability.” Journal of Geography, 106(5), 181-191.

Hawley, W. D., & Valli, L. (2001). The essentials of effective pro- fessional development: A new consensus. In D. Boesel (Ed.), Continuing professional development (pp. 1-17). Washington, DC: U.S. Department of Education, National Library of Education.

Jeong, J., Gaffney, J. S., & Choi, J.-O. (2010). Availability and use of informational texts in second-, third-, and fourth-grade class- rooms. Research in the Teaching of English, 44(4), 435-456.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed meth- ods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.

Johnson, S. (2004). Daily geography practice, grade, 4, Monterey, CA: Evan-Moor Educational.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate periph- eral participation. New York, NY: Cambridge University Press.

Lieberman, A., & Grolnick, M. (1996). Networks and reform in American education. Teachers College Record, 98, 7-45.

Masterman, E., & Rogers, Y. (2002). A framework for designing interactive multimedia to scaffold young children’s understand- ing of historical chronology. Instructional Science, 30, 221-241.

Meyer, B. J. F., & Poon, L. W. (2001). Effects of structure on strategy training and signaling on recall of text. Journal of Educational Psychology, 93, 141-159.

Michigan Department of Education. (n.d.), MI school data: District/ school information. Retrieved from https://www.mischooldata. org

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: SAGE.

Moje, E. B. (2007). Developing socially just subject-matter instruc- tion: A review of the literature on disciplinary literacy teach- ing. Review of Research in Education, 31(1), 1-44.

Monte-Sano, C. (2010). Disciplinary literacy in history: An explo- ration of the historical nature of adolescents’ writing. Journal of the Learning Sciences, 19(4), 539-568.

National Council for the Social Studies. (2013). The college, career, and civic life (C3) framework for social studies state standards: Guidance for enhancing the rigor of K-12 civics, economics, geography, and history. Silver Spring, MD: Author.

National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common core

state standards. Retrieved from http://www.corestandards.org/ read-the-standards/

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.

Pozzer-Ardenghi, L., & Roth, W. M. (2004). Making sense of pho- tographs. Science Education, 89, 219-241.

RAND Reading Study Group. (2002). Reading for understanding: Toward an R & D program in reading comprehension. Santa Monica, CA: Author.

Roberts, K. L., Norman, R. R., & Cocco, J. (2015). Relationship between graphical device comprehension and overall text com- prehension for third-grade children. Reading Psychology, 36, 389-420.

Roberts, K. L., Norman, R. R., Duke, N. K., Morsink, P., Martin, N. M., & Knight, J. A. (2013). Diagrams, timelines, and tables— Oh, my! Fostering graphical literacy. Reading Teacher, 67, 12-23. doi:10.1002/TRTR.1174

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4-14.

Stahl, K. A. D. (2008). The effects of three instructional methods on the reading comprehension on content acquisition of novice readers. Journal of Literacy Research, 40, 359-393.

Staley, D. (2006). Images of the rise of the West: Cognitive art and historical representation. The Journal of the Historical Society, 3, 383-406.

VanSledright, B. A. (2005). On the importance of historical position- ality to thinking about and teaching history. The International Journal of Social Education, 12(2) 1-18.

Williams, J. P., Stafford, K. B., Lauer, K. D., Hall, K. M., & Pollini, S. (2009). Embedding reading comprehension training in con- tent-area instruction. Journal of Educational Psychology, 101, 1-20.

Yenawine, P. (2013). Visual thinking strategies: Using art to deepen learning across school disciplines. Cambridge, MA: Harvard University Press.

Author Biographies

Kristy A. Brugar, PhD, is an assistant professor in the Department of Instructional Leadership and Academic Curriculum at University of Oklahoma. She teaches undergraduate and graduate courses in ele- mentary and secondary social studies education. Her research focuses on social studies education, interdisciplinary instruction involving social studies, literacy, and visual arts, and teacher development.

Kathryn L. Roberts, PhD, is an associate professor of reading, lan- guage, and literature at Wayne State University in Detroit, Michigan. She is a former kindergarten teacher, and currently teaches preservice and graduate-level courses in literacy education. Her research interests include content-area literacy, visual literacy, and emergent literacy.https://www.mischooldata.orghttps://www.mischooldata.orghttp://www.corestandards.org/read-the-standards/http://www.corestandards.org/read-the-standards/


Section 1: Lesson Preparation

Teacher Candidate Name:
Grade Level:
Instructional Plan Title:
Lesson Summary and Focus:In 2-3 sentences, summarize the lesson, identifying the central focus based on the content and skills you are teaching.
Classroom and Student Factors/Grouping:Describe the important classroom factors (demographics and environment) and student factors (IEPs, 504s, ELLs, students with behavior concerns, gifted learners), and the effect of those factors on planning, teaching, and assessing students to facilitate learning for all students. This should be limited to 2-3 sentences and the information should inform the differentiation components of the lesson.
National/State Learning Standards:Review national and state standards to become familiar with the standards you will be working with in the classroom environment.Your goal in this section is to identify the standards that are the focus of the lesson being presented. Standards must address learning initiatives from one or more content areas, as well as align with the lesson’s learning targets/objectives and assessments.Include the standards with the performance indicators and the standard language in its entirety.
Specific Learning Target(s)/Objectives:Learning objectives are designed to identify what the teacher intends to measure in learning. These must be aligned with the standards. When creating objectives, a learner must consider the following:· Who is the audience· What action verb will be measured during instruction/assessment· What tools or conditions are being used to meet the learningWhat is being assessed in the lesson must align directly to the objective created. This should not be a summary of the lesson, but a measurable statement demonstrating what the student will be assessed on at the completion of the lesson. For instance, “understand” is not measureable, but “describe” and “identify” are.For example:Given an unlabeled map outlining the 50 states, students will accurately label all state names.
Academic LanguageIn this section, include a bulleted list of the general academic vocabulary and content-specific vocabulary you need to teach. In a few sentences, describe how you will teach students those terms in the lesson.
Resources, Materials, Equipment, and Technology:List all resources, materials, equipment, and technology you and the students will use during the lesson. As required by your instructor, add or attach copies of ALL printed and online materials at the end of this template. Include links needed for online resources.

Section 2: Instructional Planning

Anticipatory SetYour goal in this section is to open the lesson by activating students’ prior knowledge, linking previous learning with what they will be learning in this lesson and gaining student interest for the lesson. Consider various learning preferences (movement, music, visuals) as a tool to engage interest and motivate learners for the lesson.In a bulleted list, describe the materials and activities you will use to open the lesson. Bold any materials you will need to prepare for the lesson.For example:· I will use a visual of the planet Earth and ask students to describe what Earth looks like.· I will record their ideas on the white board and ask more questions about the amount of water they think is on planet Earth and where the water is located.Time Needed
Multiple Means of RepresentationLearners perceive and comprehend information differently. Your goal in this section is to explain how you would present content in various ways to meet the needs of different learners. For example, you may present the material using guided notes, graphic organizers, video or other visual media, annotation tools, anchor charts, hands-on manipulatives, adaptive technologies, etc.In a bulleted list, describe the materials you will use to differentiate instruction and how you will use these materials throughout the lesson to support learning. Bold any materials you will need to prepare for the lesson.For example:· I will use a Venn diagram graphic organizer to teach students how to compare and contrast the two main characters in the read-aloud story.· I will model one example on the white board before allowing students to work on the Venn diagram graphic organizer with their elbow partner.Explain how you will differentiate materials for each of the following groups:· English language learners (ELL):· Students with special needs:· Students with gifted abilities:· Early finishers (those students who finish early and may need additional resources/support):Time Needed
Multiple Means of EngagementYour goal for this section is to outline how you will engage students in interacting with the content and academic language. How will students explore, practice, and apply the content? For example, you may engage students through collaborative group work, Kagan cooperative learning structures, hands-on activities, structured discussions, reading and writing activities, experiments, problem solving, etc.In a bulleted list, describe the activities you will engage students in to allow them to explore, practice, and apply the content and academic language. Bold any activities you will use in the lesson. Also, include formative questioning strategies and higher order thinking questions you might pose.For example:· I will use a matching card activity where students will need to find a partner with a card that has an answer that matches their number sentence.· I will model one example of solving a number sentence on the white board before having students search for the matching card.· I will then have the partner who has the number sentence explain to their partner how they got the answer.Explain how you will differentiate activities for each of the following groups:· English language learners (ELL):· Students with special needs:· Students with gifted abilities:· Early finishers (those students who finish early and may need additional resources/support):Time Needed
Multiple Means of ExpressionLearners differ in the ways they navigate a learning environment and express what they know. Your goal in this section is to explain the various ways in which your students will demonstrate what they have learned. Explain how you will provide alternative means for response, selection, and composition to accommodate all learners. Will you tier any of these products? Will you offer students choices to demonstrate mastery? This section is essentially differentiated assessment.In a bulleted list, explain the options you will provide for your students to express their knowledge about the topic. For example, students may demonstrate their knowledge in more summative ways through a short answer or multiple-choice test, multimedia presentation, video, speech to text, website, written sentence, paragraph, essay, poster, portfolio, hands-on project, experiment, reflection, blog post, or skit. Bold the names of any summative assessments.Students may also demonstrate their knowledge in ways that are more formative. For example, students may take part in thumbs up-thumbs middle-thumbs down, a short essay or drawing, an entrance slip or exit ticket, mini-whiteboard answers, fist to five, electronic quiz games, running records, four corners, or hand raising. Underline the names of any formative assessments.For example:Students will complete a one-paragraph reflection on the in-class simulation they experienced. They will be expected to write the reflection using complete sentences, proper capitalization and punctuation, and utilize an example from the simulation to demonstrate their understanding. Students will also take part in formative assessments throughout the lesson, such as thumbs up-thumbs middle-thumbs down and pair-share discussions, where you will determine if you need to re-teach or re-direct learning.Explain how you will differentiate assessments for each of the following groups:· English language learners (ELL):· Students with special needs:· Students with gifted abilities:· Early finishers (those students who finish early and may need additional resources/support):Time Needed
Extension Activity and/or HomeworkIdentify and describe any extension activities or homework tasks as appropriate. Explain how the extension activity or homework assignment supports the learning targets/objectives. As required by your instructor, attach any copies of homework at the end of this template.Time Needed

© 2019. Grand Canyon University. All Rights Reserved.

Eagle Productivity Solutions Creative & Interactive Team Onboarding

EValuation Plan 2019

Prepared by: Sarah Bernhard Zachary Schuster Abdullah Albelehy

Table of ContentsIntroduction and Background 02Evaluation Purpose 03Evaluation Audience 04Questions 04Decisions 06Methods 07Sample 07Instrumentation 08Limitations 09Logisitics and Timelines 10Budget 11Introduction and BackgroundEagle Productivity Solutions (or just “Eagle”) is a 200-person training vendor based in Rochester, New York. Since its establishment in 1988, Eagle has partnered with local and international clients to design, develop, and deliver custom training solutions, mostly focused on proper technology usage. Along with traditional offerings like instructor-led training and printed reference materials, Eagle also offers more modern solutions like web-based reference guides and on-demand video tutorials. Although most of its customers are pharmaceutical companies – Pfizer, Bayer, and Bristol-Myers Squibb, to name three – Eagle’s client base has expended in recent years to include non-pharma clients such as McDonald’s, Facebook, and the City of Rochester.The forthcoming evaluation plan concerns one of Eagle’s critical production teams, Creative & Interactive (C&I). The C&I team, itself, is one of several teams that comprise the Solutions Development department, which is responsible for producing the deliverables sold to Eagle’s clients as part of a scope of work. C&I, specifically, is responsible for tasks such as the graphical enhancement of deliverables like presentations and reference cards, the assembly and production of eLearning courses, and the programming of custom websites. The personnel who complete these tasks are officially called Creative Developers or Creative FX Developers, but (depending on their specific area of expertise) they are comparable to graphic designers, video producers and editors, and web developers.Despite Eagle’s reputation as a leader in custom training solutions, most of its employees have likely heard (or even said) some variation of the following phrase: “Eagle can train anyone, except its own employees.” Solutions Development, specifically, has been criticized for its lack of formal onboarding* for its new hires. Oftentimes, seasoned developers can be heard telling newcomers that most people learn through “trial by fire,” not the training provided by management. The general perception is that the lack of formal onboarding has resulted in low-quality products, inconsistent quality standards, and job dissatisfaction, among other side effects. This lack of onboarding inspired the forthcoming evaluation plan when it was brought to one of the evaluator’s attention by Stephanie Giles, the Director of Solutions Development. Giles expressed the need to begin evaluating the current onboarding program in response to the aforementioned issues, especially as Eagle continues to take on more clients, expand its offerings, and cultivate its own talent.Although an all-inclusive evaluation of the Solutions Development department at large may be planned for the future, the one outlined in this document concerns the C&I onboarding process specifically, per Giles’ recommendation to use it as a starting point. Currently, their onboarding process is rarely consistent across new hires, but it usually entails a face-to-face overview of general Solutions Development department processes, then additional face-to-face training and accompanying reference resources dependent on their role. It should be noted that, due this varied nature of the onboarding process, a comprehensive description is difficult to achieve in the planning phase. Although not an explicit goal of this evaluation, it can be anticipated that in investigating the current process further, a better, more thorough understanding of it can be established.*Important Note: For the purposes of this evaluation, the term “onboarding” refers to the orientation activities administered by Eagle to help acclimate new employees to the responsibilities and expectations of their role.PurposeThe motivation behind this evaluation is to calculate the effectiveness of Eagle Productivity Solutions’ Solutions Development’s current onboarding process. Eagle Productivity Solutions was recently named one of Rochester, NY’s Top 100 companies and was included in INC Magazine’s list of the fastest growing companies in the United States. For the purpose of this evaluation, the practitioners have be asked to focus on the Creative and Interactive team within the Solutions Department. Conducting a robust evaluation to access the overall effectiveness of the onboarding program could not come at a more critical time. Due to intensified market demands and increased workload, Eagle Productivity Solutions expects the Creative and Interactive team to grow in the upcoming months. It has also been difficult to retain workers on the Creative and Interactive team as the turnover rate is high. Bringing in new hires has proved to be extremely costly in time and financial resources. No previous evaluations have been conducted on the current onboarding process.This report will serve as a summative and formative evaluation of Eagle Productivity Solution’s Creative and Interactive’s onboarding process. The organization’s goals are to determine if the current onboarding program has been successful and make recommendations to improve their process in the future. In addition to evaluating the effectiveness Eagle Productivity Solution’s Creative and Interactive’s onboarding process, this evaluation also identifies specific sub-purposes.1. Collect information concerning the overall impact of the onboarding process on Creative and Interactive team members2. Collect new hires’ confidence levels upon completing their onboarding3. Collect attitudes and opinions on the current onboarding process.4. Collect the average percentage of rework needed for deliverables completed by new hires5. Collect the average time elapsed between new hires’ start dates and when they are considered competent enough to operate on their own (i.e., without required pre-review or frequent intervention from a manager/supervisor6. Collect information regarding where progress can be made in the Eagle Productivity Solutions onboarding process.7. Collect information to analyze if the onboarding program achieved its original goals and intentions.Evaluation AudienceAudience for the Evaluation ReportImmediate audience of this evaluation report include:The primary audience for this evaluation• Stephanie Giles – Director of Solutions Development (main client/sponsor)• Andrew McGinnis – Creative & Interactive Supervisor.Primary stakeholders include the people most directly involved in or affected by the evaluation.The secondary audience for this evaluation includes• Senior/Upper Management – General Manager (Alex Orlando), VP of Production Services (Sharon Sienkiewicz) Human Resources – HR Manger (Michele Tucker)• Creative & Interactive Team Members – Creative Developers, Web Programmers, Creative FX Developers, Studio Leads, etc.Secondary audiences are any people judged to have a stake in the evaluation and thus a right to know about its methods and resultsQuestionsThe evaluation will seek to answer the following primary and secondary questions (note: the process through which these questions were formulated is described following the questions, themselves):· Primary Question #1: Is the current onboarding process for Eagle’s Creative & Interactive employees effective?· Do developers feel confident in their ability to perform their duties upon completion of formal onboarding?· Is the onboarding process more effective for some roles than it is for others?· How well to new hires retain the information and skills they learn as part of their onboarding?· Primary Question #2: What key skills do Creative & Interactive employees learn over the course of their onboarding?· What skills are considered critical for each role?· What skills are developers expected to already have when hired?· How are new skills and concepts conveyed during onboarding?· How is mastery of these skills and concepts measured or validated?· How soon do developers begin to apply these skills after their hire date?Since its inception, the parameters of this evaluation have evolved for a multitude of reasons. One of the biggest decisions was to reduce the overall scope of the evaluation, narrowing the focus from the entire Solutions Development department to just the Creative & Interactive team. The evaluators first identified the need to narrow the scope early in the project, as evaluating a highly-complex and varied onboarding program for a department of over fifty people would have proved cumbersome for a three-person team with limited time and resources. Once this was established, the primary stakeholder was asked which sub-department or team within Solutions Development would benefit most from the proposed evaluation. She stated that, due to its relative newness and high potential for growth, the Creative & Interactive team would be the best candidate. Given its size (roughly 20 people), the evaluators agreed this would be more manageable and decided to proceed with the narrowed focus on just Creative & Interactive.Another key decision was to clarify what, exactly, is meant by “onboarding.” Although it is a common term for those familiar with corporate culture and procedures, those who have never worked in an office environment may not immediately recognize the term. Likewise, the term “onboarding” may have a variety of meanings, even for those who are familiar with it based on their own experience. Given its significance in this evaluation, the evaluators determined it was necessary to clearly establish the term’s meaning and assumptions before proceeding further, resulting in the definition cited in the “Introduction and Background” section of this document.A number of critical decisions were also made while formulating the evaluation’s primary and secondary questions. For example, the original proposal included several primary and secondary questions that have since been eliminated or reprioritized. More specifically:· The question “What percentage of billable work completed by onboarded new hires needs to be reworked by more experienced team members?”, originally a secondary question of Primary Question #1, was removed. To answer this question, the evaluators considered analyzing timecard and project data from Salesforce, the system Eagle uses to track its project metrics. While answering this question would have provided quantifiable evidence of the impact and efficacy of the current onboarding program, the evaluators anticipated such data would be too complex and voluminous to analyze fully in the given timeframe. The proprietary nature of such data could have also made it difficult to obtain, despite one of the evaluators being an Eagle employee. Additionally, the evaluators decided this data may not be entirely reliable, as it may have been influenced by a variety of factors unrelated to the efficacy of the responsible team members’ onboarding (e.g., client demands, timeline shifts, system readiness for technical projects, etc.).· The question “Is the onboarding process more effective for some roles than it is for others?” was originally a primary question, but was changed to a secondary question of the current Primary Question #1. Although there are fewer unique roles within the Creative & Interactive team than there are in the department at large, a degree of variety doesstill exist, meaning this question is still relevant. As confirmed by the primary stakeholder, even within Creative & Interactive, different roles have different onboarding experiences. Therefore, it can be assumed that a new hire’s role may have a direct impact on how effectively they adapt to their position, which directly relates to the topics addressed by Primary Question #1, making it an ideal sub-question.· The original Primary Question #2, “What key skills do developers learn while onboarding?” was revised slightly to accommodate the narrowing of the overall scope, as well as to be more specific. This original wording also revealed the previously-described need to more clearly define the term “onboarding” at forefront of all documentation, as doing so ensures all instances of the term can be easily understood by all audiences.· The original proposal also included the primary question “Who is responsible for new hires’ onboarding?”, but it was eliminated along with its secondary questions. Although the evaluators agree that the personnel conducting the onboarding certainly play an important role in whether or not the process is effective, the subject was deemed too broad and complex to include with the given timeframe.In addition to the decisions regarding the evaluation’s scope and driving questions, it is worth noting that all the evaluators’ decisions were almost universally influenced by two of the greatest limitations: distance and access to research subjects. These factors are described in greater detail in the “Limitations” section of this document, but generally speaking, both had a substantial impact on the decision-making process. Because the evaluation site is in Rochester instead of Syracuse, the evaluation is structured to accommodate two-thirds of the team’s inability to be physically at the evaluation site. As a result, certain methods of data collection have been chosen specifically because they can be administered virtually – for example, surveys, questionnaires, and remote interviews with relevant subjects. Similarly, the client expressed the need to minimize intrusion on subjects and interviewees’ working time, ruling out more time-consuming activities such as focus groups or extended interviews.DecisionsIn consulting Giles, it has become apparent the evaluation results will be welcomed by Eagle management for their potential to help enhance C&I’s onboarding process. Although it is difficult to predict the exact decisions this evaluation will influence, the following subjects are likely to be addressed following dissemination of the report:1. The amount of time and resources dedicated to onboarding a single new hire.If the evaluation results indicate that not enough time and resources are dedicated to onboarding C&I team members, Eagle management may decide to increase them to ensure a more efficient and worthwhile experience.2. Creation and deployment of a uniform onboarding process for C&I. If the evaluation results indicate that the process’s inconsistency is a major detractor from its efficacy, Eagle management may decide action is required to create a more standardized curriculum for all new team members.3. Creation and deployment of “add-on” onboarding resources, dependent on role. If the evaluation results indicate existent, role-specific resources are not currently sufficient, Eagle management may decide action is required to create a more comprehensive, role-specific curriculum to accompany the general C&I onboarding described in bullet 3.4. Specific modalities and methods to be used for onboarding. Depending on responses received from interviews and surveys, the evaluation results may help Eagle management decide what modalities and methods could be most effective for onboarding – for example, instructor-led training, on-demand videos, etc.Look at the section for “methods” on the attached example. You really need to explain each method to the reader. You really need to explain each method to the reader.Also, hash out the sampling more. Who will be sampled for each section? How many/who will be interviewed? How many/who will be sent the satisfaction survey?SampleSample for the Evaluation ReportStephanie Giles, Andrew McGinnis, Roger Trinh, Christine Wolfanger, Kama Post, various other members of the Creative & Interactive Team (for surveys/questionnaires, around people from varying roles)Methods• Surveys/questionnaires (distributed to C&I team members)• Interviews (conducted with both stakeholders and C&I team members)• Literature review (analysis of existing materials and documentation)• Anecdotal record form (ask people to give a brief description or notable event in their onboarding experience).InstrumentationThis evaluation will measure the Eagle Productivity Solutions onboarding process’s effectiveness by using the Kirkpatrick Four-Level Training Evaluation Model. The Kirkpatrick Four-Level Training Evaluation Model is a tool designed to evaluate training programs through a sequence of stages. At each stage of the evaluation, our practitioners will utilize different instrumentation to identify the success of the onboarding process, areas of improvement and other measurables. A description of the Kirkpatrick Four-Level Training Evaluation Model and each instrument used in the evaluation is located below.Kirkpatrick Four-Level Training Evaluation Model· Level 1 – Reaction: End-users will be sent an Eagle Productivity Solutions onboarding satisfaction survey to gauge its effectiveness, what they learned, how they value the training, areas to improve, etc.· Level 2 – Learning: Creative and Interactive Developers will be interviewed to access their confidence in understanding their specific job duties, corporate quality standards, etc. after the onboarding process.· Level 3- Behavior: Due to the limited amount of time and scope of our evaluation, the evaluation team will not conduct a formal evaluation on level 3 of the Kirkpatrick Four-Level Training Evaluation Model.· Level 4 – Results: At this stage the evaluation will clearly state its final results. Outcomes will be based on a combination of the interviews, user satisfaction survey and behavioral changes after training.Instrument 1 – Eagle Productivity Solutions Onboarding Satisfaction SurveyInstrument DescriptionTo determine new and existing employee satisfaction with Eagle Productivity Solutions onboarding process, the practitioners of this evaluation will utilize a satisfaction survey. The Eagle Productivity Solutions Onboarding Satisfaction Survey will be comprised of 15 questions using a Likert-type scale to measure trainee’s attitudes, feelings, opinions, etc. with the current onboarding methods and its efficiency. Trainee’s will score their satisfaction of the current onboarding process using a 5 point scale and will also have the option to include personal remarks about each question with optional comments box.1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly AgreeThe Eagle Productivity Solutions Onboarding Satisfaction Survey will include questions regarding the user’s ability to perform their required duties following the completion of the onboarding process, skills learned during the trainings, important areas that were not covered during onboarding, etc.Instrument TypeFor this survey, the evaluators will be utilizing the Qualtrics Survey Tool and Research Suite. The tool is able to administer an online anonymous survey. Creative and Interactive team members will self-report their satisfaction using the 5-point Likert-type scale and make additional comments in fillable text boxes.Instrument AudienceThe principal audience that will be surveyed is the Creative and Interactive Team’s developers, managers and supervisors.Objectives of the InstrumentEnd-users will be sent an Eagle Productivity Solutions onboarding satisfaction survey to gauge its effectiveness, what they learned, how they value the training, areas to improve, etc.Instrument 2 – Eagle Productivity Solutions Onboarding Learning and Behavior InterviewsInstrument DescriptionTrainee’s and managers of the Eagle Productivity Solutions onboarding process will be interviewed to access their confidence in understanding their specific job duties, corporate quality standards, etc. after the onboarding process. Our team of independent evaluators will gather information from the Creative and Interactive Team’s developers, managers and supervisors through a semi-structured interview approach. A variety of interview questions will be used to access each team member’s satisfaction, discontent, confidence and other feelings related to the Eagle Productivity Solutions onboarding process. Example questions include: “Do developers feel confident in their ability to perform their duties upon completion of formal onboarding?” and “What key skills do developers learn while onboarding?” The interviews with be a hybrid of in-person/virtual meetings with one evaluator on site, and two evaluators attending through video conferencing. Do to the interview participants busy schedules, the evaluators will only spend 15 minutes with each team member. Using video conferencing technology, the evaluators will capture and record each interview for future use and assessment.Instrument TypeA set of formal and informal procedures will be used to access satisfaction and the success of the Eagle Productivity Solutions onboarding process. Evaluators will use a semi-structured interview approach utilizing common job principles and set questions regarding their scope of work. This will allow the evaluator to adapt questions to individuals, including open and closed questions, scripts, etc. throughout the session to make the best use of time.Instrument AudienceThe principal audience that will be interviewed is the Creative and Interactive Team’s developers, managers and supervisors.LimitationsAs in any evaluation, the evaluators are faced with a number of limitations. The most significant of these limitations are listed and described below and referenced in other sections of this document, where appropriate:· Time: The evaluation itself is expected to be executed between March 3rd, 2019, and April 28th, 2019, a total of 56 days or approximately eight weeks. This relatively short amount of time dictated many of the decisions with regard to the evaluation’s scope and focal points, as described in the “Decisions” section of this document.· Distance: The primary evaluation site, the Eagle office, is in Rochester, NY. While one of the evaluators resides in Rochester, herself being an Eagle employee, the other two evaluators reside in Syracuse, New York, roughly ninety miles to the east. This distance makes physical site visits by the latter two evaluators difficult, with the drive being roughly one-and-a-half hours both ways with the potential to encounter inclement weather, particularly in the winter months (i.e., when this evaluation will occur). Therefore, the evaluators are limited primarily to remote evaluation methods, except where the Rochester-based evaluator is able to operate independently or serve as an effective liaison.· Access to research subjects: Because all stakeholders, clients, and subjects have their own day-to-day responsibilities to which they must attend (many of which are related to billable project work and deadlines), the evaluators are limited in the amount of time and effort they can request from these parties. Specifically, the main stakeholder requested no more than a half-hour at a time be taken for interview purposes and that only a select group of personnel be interviewed (five people in total, herself included). Considering this, data gathered from interviews will be drawn from only a portion of the larger group being studied in the evaluation, which may lead to conclusions that do not necessarily apply to everyone on the team. Likewise, three of the five recommended interviewees are in management positions, meaning data drawn from discussions with them may not accurately reflect the perspectives of those in the production workforce.· Resources: Due to the academic nature of this evaluation (i.e., it is being completed as coursework), the evaluators do not have any extra resources aside from what they can provide themselves (software provided as part of tuition, self-made instrumentation, etc.). Additionally, the team consists of three first-time program evaluators, relying on their own working knowledge of the process and without outside help from additional resources.· Information sources: As mentioned earlier, the circumstances of this evaluation require most of the data to rely on first-person testimony and word-of-mouth via interviews and survey, meaning there is little potential for results based in concrete numbers. Although the opinions and experiences of the subjects is indeed important, the lack of numeric, fact-based data may put the validity of the results into question. Another limitation is the widely varied nature of the onboarding program. Based on preliminary discussions and materials received from the clients, it is clear that onboarding to the Creative & Interactive team can be a widely varied process and often dependent on the individual; therefore, it may be difficult to identify conclusive evidence or trends.Logistics and TimelineThree people (evaluators) will be responsible for the implementation of this evaluation: Abdullah Albelehy, Sarah Bernhard, and Zachary Schuster. The distribution of labor will be as follows (note: these assignments are planned and therefore subject to change):· All three evaluators will partner on the following tasks:· Formulation of interview and survey questions· Conducting one-on-one interviews· Processing, analyzing, and interpreting of data gathered· Drafting of the final report (specific section/topic assignments TBD)· Monitoring of and adherence to budgets and timeline· Zachary Schuster will import and disseminate the finalized survey(s) in Qualtrics and grant the rest of the team access to the results for analysis. He will also be responsible for the dissemination of the evaluation report to client stakeholders upon its completion.· Sarah Bernhard, herself a subject-matter expert given her employment and tenure at Eagle, will review all materials prior to their implementation to ensure the approaches used will resonate with Eagle employees. Likewise, Bernhard will review the final report, in its entirety, for accuracy and consistency of terminology. She will also confirm meeting times with Eagle employees involved in the evaluation (client stakeholders, subjects, etc.).· Abdullah Albelehy will be responsible for scheduling client and/or subject-facing interviews, partnering with Bernhard to confirm schedules and contact information for Eagle employees. He will also be responsible for coordinating and scheduling weekly team meetings.The following timeline will be used as a guide for evaluation proceeding (note: like the distribution of labor, this timeline is subject to change):TaskStart DateCompletion DateDevelopment of Survey QuestionsMarch 4th, 2019March 6th, 2019Development of Interview QuestionsMarch 7th, 2019March 11th, 2019Creation of Survey in QualtricsMarch 12th, 2019March 14th, 2019Deployment/Completion of SurveyMarch 15th, 2019March 22nd, 2019Client and Subject Interviews (virtual)March 25th, 2019March 29th, 2019Analysis of Survey DataApril 1st, 2019April 4th, 2019Analysis of Interview DataApril 5th, 2019April 9th, 2019Drafting of Evaluation ReportApril 10th, 2019April 18th, 2019Finalization of Evaluation ReportApril 19th, 2019April 28th, 2019Delivery of Final Evaluation Report to ClientApril 29th, 2019April 30th, 2019Evaluation BudgetThe approximate cost to evaluate the Eagle Productivity Solutions onboarding process is $1,410. The SZA Evaluation Team charges a one-time $250.00 consultation fee, $22.00 per hour (detailed minutes of work will be provided) and a one-time technology fee of $500.00 for all services and deliverables.SZA estimates that the evaluation will take 30 hours of time to complete. These billable hours include, but are not limited to, client discussions, surveys (questions, format creation, analysis etc.) interviews (semi-structured in 15 minute intervals, data collection, etc.) and other evaluation techniques. The technology fee covers any software and technical solutions used by SZA during the Eagle Productivity Solutions onboarding evaluation.