Equipment Condition’s Effect on Student Perceived Workload and Efficiency of Problem Based Projects in an Aeronautical Engineering Technology Program

Aeronautical Engineering Technology (AET) of Purdue University‟s School of Aviation and Transportation Technology offers its students a comprehensive and holistic approach to engineering in an applied fashion. Students‟ learning outcome includes an application of technical knowledge and hands-on skills in areas of aerospace design, operations, and manufacturing. The curriculum of the program includes a course in which students must learn advanced maintenance concepts and practices including the overhaul of reciprocating engines. This study examined the relationship between engine operational vs. non-operational statuses, time spent to finish a task, and student perceived workloads regarding the maintenance procedures. The tests followed the Federal Aviation Administration (FAA) requirements for the practical test questions from the Airframe and Powerplant Certificate (A&P) Exam. Students were also required to fill out a task perceived load index developed and used by NASA.


Introduction
Purdue University is a large public university located in Indiana. The university is known for its research and exploration in a wide variety of disciplines and fields of study including engineering, agriculture, science, mathematics, and technology. At this time, Purdue University consists of 10 academic colleges including the Purdue Polytechnic Institute. The School of Aviation and Transportation Technology (SATT) exists within the Polytechnic Institute. SATT offers multiple bachelor"s degree programs including Professional Flight, Aeronautical Engineering Technology (AET), and Aviation Management. In addition to the undergraduate fields of study in aviation, SATT also offers several advanced degrees including a master"s degree program in Aviation and Aerospace Management and a PhD program in technology with a concentration in Aviation (Purdue University, 2018). SATT has a rich history at Purdue that can be traced back as early as the 1940s. However, it was not until the 1980s that the Aeronautical Engineering Technology degree took on its shape that most closely resembles the program that exists today. In the "80s, the degree formerly known as Aviation Maintenance Technology was laid to rest and AET was formed (Purdue University, 2018).
This newly restructured area of study offered its students a much more comprehensive and holistic approach to engineering in an applied fashion. Student learning outcome priorities now included application of technical knowledge and hands-on skills in areas of aerospace design, operations, and manufacturing. Shortly after this shift in priority, AET received an engineering technology accreditation from the Accreditation Board for Engineering and Technology (ABET) (Polytechnic Institute, 2018). Students studying AET at Purdue are taught using multiple disciplines of science, engineering, and mathematics. AET graduates have learned how to design, manufacture, maintain, operate, and support many different aerospace concepts using critical thinking skills developed throughout their time at SATT.
While the AET graduates are already quite competitive within the aviation and aerospace industries given their skill sets and abilities, SATT offers AET students an opportunity to earn an A&P certificate through the Federal Aviation Administration (FAA) if they so desire. This certificate is highly sought after within the aviation industry because it not only confirms a student"s technical abilities and knowledge regarding aviation operations, but it also allows the certificate holder to conduct, inspect, and supervise aviation maintenance activities. For example, at the time of writing of this paper, various companies advertised job positions such as SimplyHired (2019) (estimated salary: $77,000 -$100,000 a year), (SimplyHired, 2019) (estimated salary: $86,000 -$110,000a year), and Aviation Repair Engineer (estimated salary: $75,000 -$100,000 a year) (Simply Hired, 2019 a,b,c.). This is given that two first positions listed with "an A&P license would be an advantage" notation. At SATT, the A&P certificate program is taught under the FAA 14 CFR Part 147 aviation maintenance training guidelines. Training from the certificate program is complimentary to the learning priorities that AET program encompasses within its ABET accreditation because it emphasizes students" abilities to apply knowledge and skills.

Project-Based Learning
In order to accomplish the learning objectives of AET discussed above, faculty use project-based learning to allow students to develop skills through application. Faculty members and administrators at Purdue SATT both agree that project-based learning is the most effective way to accomplish the goals of the Engineering Technology curriculum and 14 CFR Part 147 training at Purdue University. Historically, learning has been instructor-centric but within the last decade there has been a shift to teach students using projects and real-life application (Zheng et al., 2011). Purdue is no exception to this trend.
Dr. Nathan Scott, Roger Hadgraft, and Dr. Vojislav Ilic explored the implementation of project-based learning over traditional methods specifically within engineering (2003). These researchers first looked into engineering curriculum and found that programs that are content driven do not provide enough integration and relatability to industry practices. It was found that perhaps the reason that project-based learning is so transferable between classrooms and the industry is because professional practice completed by engineers is typically project-based. Furthermore, project work tends to lend itself towards the application of knowledge accrued from the classrooms.
With guidance from the researchers and input from the instructors, Monash University in Australia, implemented project-based learning into the engineering curriculum. The first group of students to utilize the new curriculum graduated at the end of 2001. At that time, Scott and his team surveyed those graduates. On average, students" responses were very positive regarding the application of real-world activities. While the group concluded that project-based learning was more successful than traditional learning this was not without negative feedback from the students as well. Many students indicated that the drawbacks to the new curriculum included the amount of workload and time it took to complete the projects as well as the stress from team members not adequately helping (Scott et al., 2003).
Some universities are moving to unique curriculum that embrace project-based learning as the only method of teaching students (Feder, 2017). Quest University of Canada, Olin College of Engineering in Massachusetts and Harvey Mudd College are three learning institutes where project-based learning is the center of the teaching curriculum. After the curriculum was revamped at Harvey Mudd College, the school discovered that classes where males had typically dominated and outperformed females, there was no longer a gender difference with regards to performance and students of both genders had improved overall.
Tiago Andrade documented his time as a student studying Mechanical Engineering at the University of Porto and then reflected about his experience once his education was completed (2013). Andrade felt that project-based activities helped understand engineering a more practical sense and also helped develop his maturity, responsibility, and other key skills that could be easily transferrable into the work place such as dealing with data and working with people.
Previous research completed by Dr. Johnson of Itasca College rated students" competency levels on 16 key areas ranging from acting safely to problem solving. Over the period of two semesters, his study analyzed students" growth in a project-based learning curriculum and compared it to a control group of students in a legacy curriculum. The data he compiled, illustrated that there was a statistically significant growth in all 16 areas for students within the project-based learning curriculum when compared to those in the control group (2017). This difference was prevalent in both individual and team environments and is highly relevant due to the students within the study being engineering students.
Due to the inherent risk associated with the aviation industry, ethics are of the utmost importance and education is an opportune point to instill that importance with students. Rahman and Hendikawati (2017) research showed a correlation between not only the improved ability of students to solve problems when in a project-based curriculum but also a higher level of improvement in ethics usage when compared to the control class (2017). Their research showed that 75% of students in project-based curriculums showed improvement in their while the control class had significantly lower performance. Thomas (2000), analysis of an experiment involving two British schools that compared project-based learning methods to historically based instruction methods, showed a drastic difference between two extremely similar groups of students (2000). Both the control group and the experimental group were of a similar socio-economic status and scored comparably on national standardized testing before the experiment. Analysis completed every year, for the following three years after project-based curriculums were implemented, within the experimental group"s math classes, showed a staggering difference between the two groups. Students within the test group not only scored higher on standardized testing, but also had a more positive attitude to mathematics. Some students in the control group voiced that they felt math was a rule and memorization-based subject that would have no use in their lives outside of the classroom.
Project-based learning also has specific benefits to students within the realm of engineering. Research performed at Maastricht University in the Netherlands analyzed partial and complete implementation of projectbased learning methods into their engineering curriculum. Researchers believed that project-based learning offered an opportunity to "bridge the gap between theory and practice" and that "It"s emphasis on group work and discovery are clearly appreciated by students." Perrenet et al. (2000) This environment of learning from team and individual projects resulted in a decreased student drop-out rate at Maastricht University following the curriculum"s implementation. It is worth noting that the largest benefit of project-based learning discussed within this experiment occurred during the first three years of a student"s undergraduate degree. Researchers felt that projects were too short when compared to the work students would encounter in their careers for a student"s final year of education and the curriculum should have more parallels with industry.
The AET curriculum outside of A&P required courses uses problem-and project-based methodology to optimize its learning potential. Especially, this is most apparent in senior design capstone courses, which are demanded by ABET certification. Friesel and Dubikovsky (2019) Unfortunately, the same approach can"t be fully applied to the A&P accreditation, where the FAA specifies all topics and levels of understanding for the training. However, there is a push from aviation and aerospace industry to change decades long approach for the A&P certification, voiced by Matt Zuccaro, president of Helicopter Association International: "Requirements in the current Part 147, such as those mandating the time a student spends in a seat, dictating passing norms, requiring government approval to teach beyond set levels of instruction, imposing inflexible student/teacher ratios, demanding approval of instructor rosters, and requiring rigid adherence to static curriculum topics, are not hallmarks of a modern, competency-based system." Broderick (2018) It might take years to establish new rules, but the process had started.

FAA Practical Test Standards
The project-based focus for the AET curriculum compliments the FAA"s desired proficiency levels for students that are pursuing an Airframe and Powerplant certificate in addition to the degree. The FAA has very specific guidelines for measuring whether a candidate is qualified to receive an A&P certificate. These guidelines are outlined in the FAA document, FAA-S-8081-28A the Aviation Mechanic Powerplant Practical Test Standards.
This study used the FAA"s criteria and standards to develop questions for students. Each student had to perform two maintenance tests, one test on an operational engine and one test on a nonoperational engine. The tests were picked from the FAA"s document for practical tests. The students were scored based on the FAA criteria for satisfactory and unsatisfactory performance. In addition to scoring students based on FAA criteria, students were also timed during their tests. The time started when students were given the project and ceased once a student had announced that they felt the work was complete.
The FAA has determined that there are three different levels of performances for projects. Those levels consist of Level I, II and III. Level I performances include knowing the basic facts and principles regarding the material but no skill demonstration is required Federal Aviation Administration (2012). This would include being able to find information within manuals but not interpreting that information. Level II performance requires applicants to perform to Level I standards and also interpret information that was found and demonstrate basic operations. A high skill level is not required while performing these basic operations. For students to pass Level III questions they must be able to perform to both Level I and II performances levels. In addition, they must perform all tasks to a return-toservice standard. This being the most challenging and demanding performance level also requires a high skill level. Applicants must also understand and relate theories to the total operation of an aircraft or engine.
The FAA writes questions that encompass a range of Level I, II, and III skill levels. Different test questions will require different performance levels. Regardless of the performance level, applicants that are testing must perform to a satisfactory performance as defined by the FAA. Any unsatisfactory performance will result in an automatic failure within that subject area which would result in a student being required to retest.
Satisfactory performance requires applicants to demonstrate the prescribed level of performance for each test question. Students do not have to have knowledge such as mathematical formulas but require that applicants must be able to find the necessary information with satisfactory references. Applicants who receive unsatisfactory performance assessments must retest. Common reasons for applicants to fail any portion of the test include violating safety procedures, failing to follow approved maintenance procedures, and not being able to perform projects to a return to service standard.

Perceived Workload NASA-TLX Survey Instrument
Aside from using completion time and accuracy, another way to determine a student"s competency when performing the practical exam on operational and nonoperational equipment is by measuring the student"s perceived workload. Student"s perceived workload can be measured by employing a data collecting procedure known as the NASA Task Load Index (NASA-TLX). The NASA-TLX is a broadly accepted procedure to measure the workload of crew members and operational personals (Zheng et al., 2011). NASA-TLX measures an overall workload score by collecting six different subscales. These subscales include mental demands, physical demands, temporal demands, own performance, effort, and frustration (Hart, 1986). In a class room setting, in order to maintain the safety, productivity, and efficiency of the student, the common practice is to avoid overloading the student with task demands that are too high in difficulty and involves costly equipment. However, research has also shown that under loading the students with unchallenging tasks and subpar equipment may also create concerns with added student stress and ultimately student boredom which may lead to competency challenges (Rubio et al., 2004). Therefore, presenting the students with tasks that maintain the proper workload is vital to their performance. The student"s perceived workload while performing the practical exam comes from an interaction between multiple different factors. These factors include the requirements of the task itself, the student"s skill level, the environment and circumstances in which the exam is being performed, and the perceptions of the student being tested (Hart and Staveland, 1988). The student"s motivation for completing the assigned task directly reflects their perceptions of the task at hand, their strategies for completing the task, and the effort they are devoting (Hart and Staveland, 1988). By collecting data on the six subscales, the researcher can determine the workload each student experienced while being tested on nonoperational and operational equipment and determine whether the perceived workload is affected by their perception of the task assigned and the equipment involved in the task.

IRB Process
This research study was conducted with approval from the Institutional Review Board (IRB) at Purdue University. The IRB process was established because of ethical concerns relating to research with human subjects (Colt and Mulnard, 2006). This study complied with all criterion that the board required.

Participants
Students that were tested for this study were enrolled in the Advanced Reciprocating Engine Overhaul course. A total of 32 students were tested. The students were broken equally into two laboratory section groups comprised of thirteen students per section. Within the laboratory sections, the students were broken into two groups within the lab section, one group was tested first on operational equipment and the other group was tested first on nonoperational. After the students were assessed using the FAA standards, they were asked to complete a NASA-TLX index. Once completed, the students were tested on the other engine type and then they completed a NASA-TLX index after the test was finished.
The sample group included students from multiple class levels including sophomore, junior, and senior. There were no freshmen in the sample group because no students from that class level opted to take the class that semester. Most of the students enrolled in the program initially when they attended Purdue instead of transferring from a different area of study within the University. Twenty-eight of the thirty-two students disclosed that they expected to pursue careers in aviation. The majority of the sample group expects to test for their Federal Aviation Administration"s Airframe and Powerplant certificate. Only three students indicated that they would not be pursuing a certificate once eligible to do so.

T-Test
A series of t-tests with two-tail distributions and equal variance assumed were performed as an analysis method on the data to test the hypothesis of this study. Because the population size was limited to 32 students, performing ttests were deemed appropriate; other statistical methods would command a bigger sample sizes (Kenton, 2018). The tests compared various variables, including the time to finish the assigned projects (faster is better) and items from the TLX Scale instrument after performing assessments on both nonoperational and operational engines.
A separate discussion about the TLX Scale is required. As it was stated before, the instrument consists of six different Likert-type subscales marked from "very low" to "very high" in 20 unit increments. Then Likert (1932) proposed his scale, he assumed equal distances between the numbers in subjects" responses and the anchors. This suggests interval data and t-tests are an appropriate statistical method. Using numbers from 1 to 20 in place of verbal data points for collecting levels of certain subscales strengthen this notion (Baggaley and Hull, 1983;Dehaene et al., 1993). However, there is a strong belief that individual Likert-type format should not be used along without combining response for four or more items in a combined score. In this case, records become interval data with unquestionable statistical approach (Carifio and Perla, 2007;Desselle, 2005).
Based on the rationale above, series of t-tests were performed on the time to complete projects on nonoperational and operational engines, the TLX subscales (mental demands, physical demands, temporal demands, own performance, effort, and frustration) and combined TLX perceived load data for both scenarios. The results are presented in the following section.

Results
For time to finish required project, no statistical difference was found between the subjects" groups: students performing on operational engines (M = 22.28, SD = 14.21) and those who worked on nonoperational engines (M = 25.03, SD = 15.26), t(62) = -0.75, p = n.s.
All results of descriptive statistics and t-tests are presented in Table 1 and Table 2.

Discussion and Conclusion
The findings show that there is no difference exists between training and assessing the students on operational vs. nonoperational engines. There is no evidence that equipment operational status improves students" performance and increases mental pressure to keep the engines in airworthy state. The data showed that students spent less time on operational equipment and perceived work load was lighter then working on them. Those results are completely the opposite of the initial expectations. To better understand the present occurrence, additional studies needed.
Future investigations with a larger population size might be beneficial. Also, additional interviews with students might offer detailed explanation to this phenomenon. If further studies reliably demonstrate that there is no statistical difference between equipment operational status and it does not affect students" perceived workload, this may change the way educational institutions spend their limited resources. For example, funds might be better spent on alternative forms of training aids, such as engine mock-ups or computer simulation.
In the last seven decades, since 1940"s and 1950"s, high-rick industries such as NASA and the US Air Force widely accepted computer simulation to reduce human errors, which led to significantly improved safety (Allerton, 2010;Gerathewohl, 1969). Computer-based pilot training constantly changes with enhancement in technology. It is possible now to conduct preparation of pilots from novice to certification using only computer simulation as a primary method (Macchiarella et al., 2006). One of many of the studies confirmed success of computer-based pilot training and also supported effectiveness of the positive psychomotor skill transfer during such method (Reweti, 2014). Another field, where computer simulation is used widely and successfully, is medical profession, especially in preparation of surgeons and clinicians (Issenberg et al., 2003;Issenberg and Scalese, 2007).
One can agree that preparation of engineers and engineering technology students differs because of lack of repetitive tasks. However, Kulatunga (2003) examined learning outcomes of computer-based simulator in place of hands-on activities for electronics engineering students. The factors such as "motor characteristics" and "memory reinforcement" surfaced during the follow up tests. In addition, it was found that haptic skill transfer in both approaches was very similar. In other study, researchers assessed biomechanical measures for virtual reality simulator training vs. on-water training for rowing teams. Again, the study showed that virtual reality environment successfully transfer both theoretical knowledge and physical abilities, very similar to real workout (Rauter et al., 2013). If proven that those alternative forms of students" preparation work effectively, this might change how training organizations spend their limited resources, implement curricula, and educate future maintenance professionals.