Year: 2008

Authors: Sowmya Ramachandran, Jeremy Ludwig, Eduardo Salas, Michael Rosen

Abstract: Job performance measurement is of critical importance to any organization.s health. It is important not only to recognize and reward good performance, but also to groom future leaders. Developing effective assessment techniques that are valid, effective and fair is an ongoing challenge. Assessing factual knowledge using multiple-choice test batteries relatively inexpensive and tends to be commonly used. Hands-on assessment is the most effective in assessing task proficiency but is very resource intensive and expensive. Computer-based simulations provide an alternative where users can be assessed in the context of skill application under controlled conditions. However, simulations are expensive to produce and maintain. Validated guidelines and methodologies are needed to help organizations develop effective assessment simulations. In this paper we present a standard, prescriptive methodology for developing simulations for job performance assessment. We then describe a performance assessment simulation for Light-Wheeled Vehicle Maintenance constructed according to this methodology. This simulation includes automated assessment methods that borrow heavily from existing work in intelligent tutoring systems. Finally, we discuss future research directions based on the results of this initial methodology and assessment.

Year: 2008

Author: Charles Hutchings

Abstract: Computer models have been used extensively in U.S. Government Federal agencies for decades. A study by the U.S. General Accounting Office in 1982 lists 357 models being used to support policy decisions from among 43 different Federal departments, agencies, and entities. As computing power has grown and computing costs have declined since the 1980's, digital models and simulations have become increasingly used to support analysis, acquisition, and training; however, only a few agencies, most notably the Department of Defense, have agency-wide policy, standards, training, or guidance on how these should be developed, managed, or evaluated. DHS, created by the Homeland Security Act of 2002, has several dedicated M&S centers to support the Department's mission. These include the National Infrastructure Simulation and Analysis Center (NISAC) which provides analysis for the National Infrastructure Protection Plan (NIPP). The National Response Plan designates the DHS led Interagency Modeling and Atmospheric Assessment Center (IMAAC) as the single Federal source of airborne hazards predictions during an Incident of National Significance (INS). The Federal Emergency Management Agency (FEMA), a DHS component, is currently establishing a National Exercise Simulation Center that uses M&S for training, exercises, and command and control functions at the operational level, including a mix of live, virtual, and constructive simulations to prepare elected officials, emergency managers, emergency response providers, and emergency support providers at all levels of government to operate cohesively. This paper discusses the development of M&S training and guidance for DHS personnel which includes an overview of policy, standards, guidance, and best practice from Federal agencies.

Year: 2008

Authors: Paul Vogel, Seokbin Lim

Abstract: It has become common practice to employ comprehensive simulations in performing military utility analysis (MUA) to evaluate candidate military systems and architectures. But the accuracy and flexibility of these simulations rely on accurate individual models focused on detailed resolution of local events. This study evaluates the efficient and advanced employment of the Gurney equations within the context of a comprehensive MUA model. The resulting model evaluates a wide range of criteria including multiple mixed detonations, target armor, warhead placement and endgame maneuvering, buildings and terrain, and environmental criteria to estimate shrapnel lethality. The result is an efficient model that can be used either stand-alone or embedded within the larger framework. It has sufficient detail to analyze shrapnel effects in munitions ranging from the individual IEDs (Improvised Explosives Devices) employed in asymmetric warfare up through the employment of alternate artillery rounds in conventional warfare. This model evaluates individual warhead detonations by predicting shrapnel velocity and geometric distribution based upon the type and amount of the core charge, physical properties of the outer casing creating fragments, the incoming vector velocity of the warhead, etc. Burst density and a kinetic energy distribution are derived from these factors. These are compared to the presented vulnerable area of all targets within the region in lethality evaluation. When embedded in the JFORCES simulation environment the results of this model are directly used to measure to impact on localized operations and effectiveness. In addition to introducing the model this paper includes a sample analysis to demonstrate its within a larger framework.

Year: 2008

Authors: Randy Elms, Austin Riddle, Keith Biggers

Abstract:

Realistic collaborative team training has been facilitated by recently improved simulation capabilities. Simulation-based exercises are more realistic and adaptable than ever before to a wide range of collaborative team operational settings. Before training developers can fully leverage the power of this new generation of simulations, scenarios that provide command posts and operations centers rigorous and cross-functional training in a relatively limited development cycle must be quickly generated. Veteran scenario designers are well aware that there are few "easy buttons" for alleviating what is at best a laborious, time consuming process. In many organizations, the scenario supply cannot maintain pace with the training demand. Although the scenario production challenge applies to the military to a great degree, it is even more pronounced in the civilian emergency management community where there is a lack of organizational uniformity and standardized tactics, techniques and procedures (TTP).

Our research has addressed the collaborative team training challenge. Several computer-based systems have been developed which have been tested and used for formal emergency management training in both civilian and military (U.S. Air Force) venues. The computer-based training tool that was most recently produced is the Emergency Management Exercise System (EMES). The scenario engine of the EMES is a flexible and adaptable system known as the Scenario Design and Execution System (SDES). It has capabilities for reconfiguring scenarios with similar databases so as to produce meaningful, multi-level team exercises regardless of a wide range of operational complexities. It can be used for a variety of training venues to include full-scale exercises, seminars, table top exercises and classroom instruction.

This paper describes the SDES, its scenario reconfigurability features and its application to the team training development phases of exercise design, exercise execution and post-exercise performance analysis.

Year: 2008

Authors: Steve Slosser, Curtis Conkey, Maj Magidson

Abstract: The use of commercial gaming technology by the US military for PC-based training and mission rehearsal applications is increasing. However, deployment of this technology typically requires high-end hardware and large, lengthy software downloads; or worse, deployment via physical media and technical support. Additionally, DoD Information Assurance (IA) requirements forbid the execution of any software that has not been through an arduous certification process. Finally, the services are attempting to standardize computer and networking resources to specifications geared towards typical office applications (e.g., document editing, email, web browsing, etc.). These platforms typically have a multiyear refresh cycle so that any particular computing station could be up to 3 years old. Networking performance is "sized" for office expectations and not interactive content delivery. The push to adopt gaming and simulation platforms for interactive training is running counter to the push to standardize computing and networking platforms across the DoD. Services often cannot afford the costs of deploying additional equipment and resources for gaming and simulation-based training. Web client solutions, interactive applications that work primarily within a web browser environment, run on existing DoD computing and networking assets. The US Air Force and Navy have teamed to achieve a highly interactive web-based mission rehearsal trainer capable of being delivered through the browser on a typical enterprise desktop computer. This paper discusses the requirements, trade-offs and technology researched this year to meet this need.

Year: 2008

Authors: Gregory Goodwin, Jean Dyer, Bruce Leibrecht, Richard Wampler

Abstract:

The Army Battle Command System (ABCS) is a force multiplier because it enhances the dissemination, analysis, and storage of critical battlefield information. Effective employment of this system in operational contexts requires well-trained Soldiers and leaders. To evaluate the effectiveness of digital system training, we examined the techniques and practices of ABCS instructors of four major ABCS systems.

This paper describes the training techniques of digital instructors from the standpoint of cognitive, behavioral, and constructivist theories and offers guidance for improvement based on the literature. A total of 24 days of training was observed across eight separate classes covering four ABCS systems. Observers recorded a number of instructional activities including the incidence of training techniques (e.g., use of memory aids, pointing out screen prompts and cues, emphasizing active learning) and classroom activities (e.g., lecture, guided demonstration, practical exercise). Instructors predominantly taught with cognitive and behavioral techniques such as demonstrating the steps of a task while the students repeated the steps on their own computer. Research has shown that constructivist techniques such as guided exploration can improve the acquisition and transfer of digital skills over the techniques currently in use; therefore, instructors would do well to incorporate these types of techniques into their teaching repertoire. In addition, best practices from across classrooms were identified and discussed.

Year: 2008

Authors: Paul Pine, Steven Fulthorpe

Abstract:

With the Royal Navy's Type 45 Destroyer now entering into Service, a new generation of training is required for this next generation of ship. With the recent growth of internet training technologies, the Royal Navy has decided that the Type 45 would be their vanguard in the large scale adoption of modern e-learning solutions, bringing the rest of the Fleet in its wake. With no indigenous e-learning capability of its own, the Royal Navy formed the Type 45 Training Unit to develop this new breed of training, moving it towards an e-learning culture and 21^st Century ways of working.

This paper highlights the production of this new Blended Facilitative Training which is being developed on an almost industrial scale. By using e-learning technologies including LMS, LCMS and bite size synthetic training media, along with an up-skilled team of developers, this paper will describe how the Royal Navy has changed the way it designs and delivers its training to provide flexible and responsive personnel that deliver and enhance Operational Capability. Throughout this process the Royal Navy has used recognized academic educational theories to further enhanced training outputs, through exercising higher level cognitive skills, delivering new knowledge in a more interactive way and producing assessments that assure competence and reduce skill fade. These theories have also been used to improve design processes, by guiding novice developers to produce high quality content, aided by templates, that would be expected of professional training designers. Through sharing many of the lessons learnt, this paper provides examples of how to bring learning theory into practice and improve the pedagogy of blended elearning through facilitative delivery.

Year: 2008

Author: Annie Patenaude

Abstract:

The Training Transformation (T2) program instituted systematic assessment as an integral part of the Joint Training System (JTS), in concert with a goal of better enabling integrated operations. As a result, the 2007 Block Assessment provided leadership insights into the impact of T2 on joint training readiness when it was completed in December 2007.

Supporting the Office of the Secretary of Defense (OSD), the assessment team developed issue statements to link the assessment to program goals and strategic guidance. The team supported the issues with metrics and data elements that provided rigor for analytic assessments of processes and procedures. They strived for outcome measures to show the results of joint training, as well as measures of efficiency. Among the hurdles overcome through the cooperation of the military Services and combatant commands were the lack of automated data collection and reporting tools. The team developed interim solutions that met the short-term requirement and resulted in lessons for longer-term solutions.

The 2007 Block Assessment provided leadership with many indications of the impact of joint training and the T2 program. It also included recommendations to improve future assessments, such as: (1) Develop an automated framework (tools and processes) to support both internal and leadership assessments; (2) Organizations should develop performance measures to support assessment of the Combatant Commander Exercise and Engagement program and Training Transformation (CE2T2); and (3) Combat Support Agencies should be included in future assessments to better indicate the full scope of "training the way we intend to operate," a goal of the T2 program.

In both block assessments, analysts spent the majority of their effort collecting, screening, and collating data. To become more efficient, analysts need to automate the process in two ways -- by setting up links to electronically collect data from existing sources and by establishing a framework to use available data. This paper will report on OSD's experience, results, and lessons learned automating an enterprise-level joint training assessment framework.

Year: 2008

Authors: Patrick Gallagher, Hasan Altalib

Abstract: Johnson Space Center (JSC) has been directed to establish new innovative programs in engineering while continuing to fulfill the requirements of its current programs. As a result, the Space Shuttle Program is transitioning to a new innovative program called Constellation which is expected to become just as, if not more, critical. To aid in this transition while retaining and preserving institutional knowledge and expertise for the next generation of engineers, program managers and leaders, the center needed mechanisms in place to manage the sharing and acquisition of knowledge throughout the entire knowledge lifecycle. First, the CKO sponsored a Knowledge Management Assessment Project (KMAP) to determine the knowledge management maturity of the center. This was the first step in developing and implementing a KM roadmap. To accomplish the KMAP, SAIC used a mixed methods approach to assess the approximately 10,000 civil servants and contractors across the center. First, focus groups and interviews were used to understand JSC's as-is environment. Utilizing the qualitative Global Knowledge Management Maturity Model (G-KMMM) developed by Pee, Teah, and Kankanhalli (2006), a quantitative instrument (Q-Assess) was developed by SAIC and tailored using results from the qualitative analyses. Implemented online, the Q-Assess represented 12 sub-assessments for assessing levels of maturity across the KPAs (key performance areas) of people, processes, and technology. Using the Q-Assess results as the condition and the G-KMMM itself as the criterion, a multi-unit gap analysis was produced. As JSC's KM maturity was identified within each unit, recommendations were than developed to facilitate the attainment of the next level of maturity. These results fed the development of the roadmap. This roadmap contained the implementation of knowledge and technology audits leading to the development of an organizational knowledge architecture and enterprise architecture in support of knowledge management, ensuring shareability, compatibility, and preservation.

Year: 2008

Authors: Eric Burns, Ty Christopher, Rett Crocker

Abstract: A growing trend within the simulation and training industry is to build 3D simulations on top of "game" engines used as platforms in the computer gaming industry. Recently, several "next generation" game engines have been released that take advantage of the latest techniques in computer graphics and the latest graphics hardware. These engines hold a lot of promise for the future of the simulation and training industry, but currently, many simulations are still built on the previous generation of technology. As a guide for upgrading to the latest platforms, we evaluated three next generation game engines: Garage Games' Torque, Epic Games' Unreal Engine 3, and Emergent Game Technologies' Gamebryo. For the evaluation, a team comprised of engineers and artists created an application on top of each engine that loaded a complex model (~500,000 polygon, working engine room) and allowed a user to navigate through it without penetrating objects. The engines were evaluated based on the software development process, the art content development process, performance, and licensing.