Analyzing Validity and Reliability of Motivational Orientation of Differentiated Instruction in English Language Teaching Student Questionnaire (MoDiELT)

Recently, the implementation of differentiated instruction had been proposed by Ministry of Education of Malaysia to be implemented across all schools in the country. Consequently, as announced in the Malaysia Education Blueprint (2013) the Ministry had launched a program called Differentiated Teaching and Learning of English Language. After few years of its implementation, a measurement protocol was needed to assess the effectiveness of differentiated teaching approach in the teaching and learning of English language. In this instance, a multidimensional instrument was developed to measure student motivation toward differentiated teaching and learning of English language, and indicate teachers’ overall teaching performance. The questionnaire contains three sections A (Demography), B (78-item scale assessing student motivation based on their experience of differentiated English language teaching and learning), and C (Student Comment/Suggestion). In this paper, the researcher presents the procedures involved in evaluating the psychometric properties of the instrument and discusses its validity and reliability. The items were constructed based on an accumulation of teachers’ differentiated teaching strategies. Face and content validity were evaluated while internal consistency and factor analyses were computed. The final reliability coefficients for the whole scale and subscales range from high to very high, while changes suggested by the analyses were accepted. The overall analysis suggested that the questionnaire is deemed valid and reliable to measure student motivation toward differentiated teaching and learning of English language, and as an indicator of teachers’ performance in applying differentiated approach.


Introduction
Differentiated instruction has been widely practiced at schools in teaching and learning various school subjects at various grade levels. This pedagogical approach provides modification in the pedagogical components i.e. content, process, and product, based on learner readiness, interest, and learning profile, thus depict variation of classroom activities that cater for every individual learner needs.
Recently, the Ministry of Education of Malaysia had proposed the implementation of differentiated instruction across all schools in the country. Consequently, as announced in the Ministry of Education (2013), the Ministry had launched a program called Differentiated Teaching and Learning of English Language. After few years of its implementation, a measurement protocol was needed to assess the effectiveness of differentiated instruction in the teaching and learning of English language.
In this regard, a valid and reliable instrument is needed to provide empirically sound assessment on the effects of differentiated instruction on students. Many studies on differentiated instruction have revealed potential effects of this teaching approach on student learning outcome. Some studies have highlighted increased motivation among learners (Anderson, 2007;Bailey and Williams-Black, 2008;York-Barr et al., 2007). Since motivation has been regarded as one of the important constructs in English language teaching and learning, this established construct was utilized as the effect. In this article, the researcher presents the procedures undertaken in developing a valid and reliable instrument that measures student motivation having experienced differentiated English language teaching and learning. determination to cater for every learner's needs through differentiation was put down due to excessive workload responsibilities, demands for substantial content coverage, and negative classroom behavior (p.34).
Most importantly, many researches on differentiated instruction have highlighted the connection between particular instructional strategies and student motivation. A qualitative study assessing students, teachers, and parents' input on the practice of differentiated instruction found positive impact on students' growth (Gibson, 2005). Some studies described in their findings that differentiated instruction develops independence, competences, and the self-images of students (Lavadenz and Armas, 2008;Valiende and Koutselini, 2009). This can be explained as the results of providing appropriate challenge through differentiated instruction, because it keeps students highly engaged in learning (Palmer and Maag, 2010;Reis and Renzulli, 2010). This is indeed true because differentiated instruction leads students into interacting in social-like activities that provide them with necessary challenges that initiate the students to take charge of their learning, and thus, become motivated (Anderson, 2007;Bailey and Williams-Black, 2008;York-Barr et al., 2007).
Despite the existing studies revealing beneficial effects on learners, little is known of an appropriate measurement tool assessing the relative effects of differentiated instruction on student learning outcome in general, and especially of student motivation having experienced such pedagogical approach. These studies gathered the findings by means of perspectives from teachers and students through qualitative data collection methods such as interviews. In addition to that, while most instruments measure student motivation toward instructional practices in regular second language classroom, an instrument specifically measuring student motivation toward the use of differentiated instruction in English language teaching is needed. Thus, an empirically sound instrument that is capable to measure the motivational effects of differentiated instruction on students would bridge this gap. An empirically valid and reliable instrument measuring the effects of differentiated instruction on students, particularly of their motivation would provide valuable information for educators alike. The next section presents the process involved in developing a multidimensional questionnaire measuring student motivation toward the use of differentiated instruction strategies in English Language Teaching. This questionnaire was developed in order to answer the research question of the main study i.e. how does differentiated instruction strategies relate to student motivation?

Methods and Procedures
The purpose of this article is to present the process of instrument validation, in particular, a questionnaire called MoDiELT. These include assessing the translational validity, construct validity, and internal consistency of the instrument. In doing so, several methods were employed i.e. validating the content and face validity, factor analysis, and analyzing the internal consistency of the draft of MoDiELT. The questions guided this purpose was How does MoDiELT reliable and valid in measuring student motivation toward the use of differentiated instruction strategies in English Language Teaching?

The Instrument: Modielt
The format of the questionnaire was adapted from Guilloteaux and Dornyei (2008) Student Motivational State questionnaire scored on a Likert scale. The items pooled on the draft of MoDiELT were constructed based on the findings of a qualitative study exploring differentiated instruction strategies. Nine strategies were derived from the study. These nine strategies were then grouped tentatively according to Dörnyei (2001) L2 Motivational Teaching Components i.e. i) creating basic motivational condition, ii) generating initial motivation, iii) maintaining and protecting motivation, and iv) encouraging positive self-evaluation. The draft questionnaire, thus, is a multidimensional scale that consists of three sections A (Demography), B (78item scale assessing student motivation based on their experience of differentiated English language teaching and learning), and C (Student Comment/Suggestion). The main concern of this article is section B. The 78-item scale contains four main subscales namely 1) creating basic motivational condition, 2) generating initial motivation, 3) maintaining and protecting motivation, and 4) encouraging positive self-evaluation which are Dörnyei (2001), L2 Motivational Teaching Components. Nine sections of differentiated instruction strategies were aligned against the four subscales tentatively i.e. subject to change based on the suggestions from the factor and internal consistency analyses. The list of items can found in Appendix 1.

Participants
In order to validate the instrument, two pilot studies: i) initial pilot study, and ii) main pilot study, were conducted. The initial pilot study was meant for evaluating the content and face validity of the instrument. Six experts were consulted about the content of the instrument, while 47 students of Pusat PERMATApintar Negara, UKM, participated in the initial pilot study to determine the face validity of the instrument.
In the main pilot study, a bigger sample was required to qualify for factor analysis that refine the construct validity of the instrument. For this purpose, 180 students of SMK Tun Telanai, Kuala Terengganu, participated in the main pilot study.

Validity 4.1.1. Face Validity
Face validity, even though considered as the weakest form of validity, assesses the physical appearance of a questionnaire pertaining to its feasibility, readability, formatting, and the language (DeVon, 2007;Trochim, 2001). The usability of MoDiELT questionnaire was determined via the initial pilot study conducted on 47 students at Pusat PERMATApintar Negara, UKM, Bangi, Selangor. Upon the completion of the piloting, the researcher found that the students: a) had no difficulty with the wording, or language, used, b) Were able to complete the questionnaire without assistance.

Content Validity
This measure was conducted in order to assess the appropriateness of the content of the questionnaire vis-à-vis the purpose of the main study i.e. specifically to investigate the relationship between student motivation as the result of differentiated teaching and learning of English language. In determining content validity of a scale, the items should reflect the attribute under study and sent for expert review (DeVon, 2007). In doing so, the researcher conducted a through literature review of motivation and differentiated instruction in order to produce a conceptual framework specific for the intent of this study. A total of 6 reviewers agreed to assist the researcher in coming up with a more inclusive and valid instruments. The reviewers were approached first personally to seek their agreement and interest. The researcher sought for the expertise of these reviewers based on their merits as described in their experience involving with the following: a) involvement in researches related to English language and the teaching of English language b) involvement in the teaching of English language c) involvement as an inspectorate of the teaching of English language d) involvement in the development of survey instruments related to English language and the teaching of English language The reviewers verified the content validity of the questionnaire that the items were appropriate to the conceptual framework i.e. motivation toward differentiated instruction strategies.

Reliability 4.2.1. Internal Consistency
The data gathered form the pilot study was keyed into SPSS version 20. Table 2 below shows the output of the Item-total statistics for the differentiated instruction categories confined to their respective L2 Motivational Teaching Components. Based on this output, the decision to drop items was based on the criteria by Gable and Wolf (1993) as in the following priorities: i) The overall alpha value for each category must be at least 0.70, but the value of 0.80 and above are better, ii) The corrected item-total correlation must be at least 0.20, iii) The items neither have too low nor too high min value, as well as low standard deviation. Thus, it was revealed that the Cronbach's Alpha values for each element (Total:9) and each component (Total:4) before and after the items were dropped are shown in Table 2 below. Ten items i.e. B15, B17, B18, G62, G63, G65, E41, F57, I78, and H68 were suggested to be deleted. In order to refine the validity of MoDiELT, the item correlation with the total mean of all the elements of DI strategies in each L2 Motivational Teaching component were analyzed. In doing so, correlation coefficient procedure used to measure the strength of the correlation between dependent variables called bivariate correlations was applied. Descriptions of the coefficient values generated here were based on Davies (1971) interpretations of correlation coefficient values (r) as in the followings: 1.00 is perfect, 0.70 -0.99 is very high, 0.50 -0.69 is high, 0.30 -0.49 is moderate, 0.10 -0.29 is low 0.01 -0.09 means that it can be dismissed.

Item Correlation with Total Means of Component 1: Creating the Basic Motivational Conditions (Elements A and B)
The Pearson's correlation analysis between items and total means of component 1 creating the basic motivational conditions is shown in Table 3 below. The analysis found that there were both significant and insignificant relationships between items and total means of component 1.
The results of Pearson's correlation between items of elements A Interest and B Readiness indicate significant relationships at moderate level, ranging from 0.477 -0.670, p<0.01, and 0.357 -0.695, p<0.01, respectively.
However, while item B15 was deleted because it did not correlate with the total mean of component 1, items B17 and B18 were deleted due to low level of significance. Thus, all of the items for element A Interest and B Readiness were retained except B15, B17, and B18. These three items of element B Readiness were omitted from component 1.

Item Correlation with Total Means of Component 2: Generating Initial Motivation (Elements C and D)
The Pearson's correlation analysis between items and total means of component 2 Generating initial motivation is shown in Table 4 below. The analysis revealed significant relationships between items and total means of component 2 Generating initial motivation. The results of the Pearson's correlation between items of the elements C Learning profile and D Teacher allows choice, teacher monitors, teacher rewards indicate significant relationships at mediocre to very high levels, ranging from 0.458 -0.745, p<0.01, and 0.486 -0.679, p<0.01, respectively. Thus, all items for elements C Learning profile and D Teacher allows choice, teacher monitors, teacher rewards are retained.

Item Correlation with Total Means of Component 3: Maintaining and Protecting Motivation (Elements E, F and G)
The Pearson's correlation analysis between items and total means of component 3 Maintaining and protecting motivation is shown in Table 5 below.
The analysis revealed both significant and insignificant relationships between items and total means of component 3 Maintaining and protecting motivation. The results of the Pearson's correlation between the items of elements E Content, F Process, and G Flexible grouping indicate a range of moderate to very high levels of significance, ranging from 0.388 -0.760, p<0.01, 0.526 -0.656, p<0.01, dan 0.304 -0.571, p<0.01, respectively. Thus, all items for the elements E Content, F Process, and G Flexible grouping were retained except F57 that was deleted due to having no significant relationship between the item and the total means of component 3 Maintaining and protecting motivation. Item F57 was omitted from component 3.

Item Correlation with Total Means of Component 4: Encouraging Positive Self-Evaluation
The Pearson's correlation analysis between items and total means of component 4 Encouraging positive selfevaluation is shown in Table 6 below.
The analysis revealed significant relationships between all items and total means of component 4 Encouraging positive self-evaluation. The results of the Pearson's correlation coefficient between the items for the elements H Product and I Ongoing assessment/adjustment indicate a range of moderate to very high levels of significance, ranging from 0.368 -0.706, p<0.01) and 0.454 -0.648, p<0.01, respectively. Thus, all items for the elements H Product and I Ongoing assessment/adjustment were retained.

Factor Analysis
Construct validity refers to the degree whether the items of an instrument measure what it is supposed to measure. A reliable instrument does not necessarily guarantee that it is valid, or that it measures particular constructs. In order to enhance the degree of validity of MoDiELT, Factor analysis was conducted on all of the items that, as mentioned before, were tentatively grouped into the nine elements of DI strategies and then classified according to the four L2 Motivational Teaching Components.
A factor refers to a list of items that belong together, while loading refers to association between an item and a factor (Bryman and Cramer, 2005;Parsian, 2009). Through factor analysis method, i) the items of an instrument are clustered into common factors, ii) factors are interpreted based on items with high loading, and then iii) the items are summarized into a small number of factors (Bryman and Cramer, 1999).
Factor analysis requires appropriate sample size in order to generate appropriate factors and loadings and thus reliable (Bryman and Cramer, 2005). In order to achieve a reliable factor analysis, i) Kaiser-Meyer-Olkin (KMO) sampling adequacy, and ii) factor loadings and correlation between a variable and a factor (Hayes, 2002), were employed.
After considering the appropriateness of sample size, as well as factor loadings and correlation between a variable and a factor, Principal Component Analysis (PCA) or Principal Axis Factoring (PAF) can be applied as the extraction method to analyze specific and common variance. While specific variance refers to the variation of a variable, common variance are the variance shared by the scores of respondents with other variables (Bryman and Cramer, 2005). Unlike PAF, PCA analyzes the total variance i.e. both specific and common variance, and deemed reliable without error (Bryman and Cramer, 2005). Therefore, in this study, PCA was employed on the 78 items of MoDiELT. Only items with factorial weights at one factor are considered as genuine items, while items that fit in with two or more factors are considered complexed items (Tabachnick and Fidell, 2001).
The results of the factor analysis for all the components i.e. (1) Creating the basic motivational condition, (2) Generating initial motivation, (3) Maintaining and protecting motivation and (4) Encouraging positive selfevaluation are shown below.  Table 7 above displays the result of factor analysis for Component 1 Creating the basic motivational conditions with factorial weights and variance contribution towards each factor.

Factor Analysis of Component 1: Creating the Basic Motivational Conditions (A, B) Using Principal Component Analysis and Varimax Rotations
The percentage of variance contributed by the first and second factor are 23.318%, and 16.516% respectively. Both factors contributed to 39.833% of total variance in the original matrix. [After scrutinizing the scree plot, and considering the hypothetical factors, the researcher labeled [A] Interest: Teacher establishes learner interest in lesson, as the first factor and [B] Readiness: Teacher bases lesson on learner interest as the second factor. There is no changes in the items of each subdomain, as in the original classification. Four items were deleted i.e. A1, B15, B17 and B18 because they did not fit the subdomains. Generic: choice, monitor, reward Table 8 above displays the result of factor analysis for Component 2: Generating initial motivation with its factorial weights and variance contribution towards each factor. The percentage of variance contributed by the first and second factor are 24.911% and 16.695% respectively. Both factors contributed to 41.606% of total variance in the original matrix. [After scrutinizing the scree plot, and considering the hypothetical factors,] the researcher labeled [C] Learning profile: Teacher bases lesson based on learning profile as the first factor and [D] Generic: choice, monitor, reward as the second factor, even though items C25, D30, D31, D32, and D33 changed from the original classification. No items were dropped.  Table 9 above displays the result of factor analysis for Component 3: Maintaining and protecting motivation with its factorial weights and variance contribution towards each factor. The percentage of variance contributed by the first, second and third factors are 16.941%, 13.882% and 11.281% respectively. All the three factors contributed to 42.104% of total variance in the original matrix. [After scrutinizing the scree plot, and considering the hypothetical factors,] the researcher labeled [E] Content: Teacher varies the content based on the theme as the first factor, [F] Process: Teacher varies the process as the second factor, and [G] Flexible grouping: Teacher varies the group style, even though items G59, E42, F52 and F53 changed from the original classification. 5 items dropped were G62, G63, G65, E41 and F57 because the did not fit the subdomains. On-going assessment / adjustment: Teacher provides on-going assessment / adjustment Table 10 above reveals the result of factor analysis for Component 4: Encouraging positive self-evaluation with its factorial weights and variance contribution towards each factor. The percentage of variance contributed by the first and second factors are 19.930% dan 19.739%. respectively. Both factors contributed to 39.670% of total variance in the original matrix. [After scrutinizing the scree plot, and considering the hypothetical factors,] the researcher labeled [H] Product: teacher varies the product as the first factor, and [I] On-going assessment/adjustment as the second factor even though items I75, I78 and H69 changed from the original classification. One item was dropped i.e. H68 because it did not fit the subdomains.

The Instrument for the Main Study
The finalized instrument for the main study was prepared based on the results of the reliability and validity analyses through several measures discussed above i.e. evaluating the face and content validity, as well as internal consistency and factor analyses. The new item total for the instrument is 68 items. The number of items for each element of DI strategies are displayed in Table 11 below.

Reliability of the Final 68-Item Instrument for the Main Study
The last step in refining the instrument is analyzing the internal consistency and correlation coefficience of the final 68 items. The result of this final reliability analysis on the 68 items are shown in Table 12 and 13 below.  Table 13 shows the overall means of each element of DI strategies ranging from 0.571 -0.790 i.e. from high to very high, at significance value of p<0.01. In general, the result reveals that the items have high correlation as indicated by the overall means of each element of DI strategies.

Conclusion
In this article, the researcher presented and discussed the reliability and validity measures that had been conducted with the purpose to refine an instrument called Motivational Orientation towards Differentiated Instruction in English Language Teaching (MoDiELT) meant to measure student motivation toward the use of differentiated instruction strategies in English language teaching, and reveal teachers' overall teaching performance i.r. their ability in differentiating lessons. The initial 78 items constructed for the draft of MoDiELT, were constructed based on nine elements of differentiated instruction strategies and four L2 Motivational Teaching components. The reliability and validity measures employed i.e. evaluating the face and content validity, internal consistency, and factor analysis resulted with ten items being deleted from the instrument. The overall analyses therefore suggest that the final 68-item MoDiELT, having high degree of overall reliability and validity, is reliable and valid to measure student motivation toward the use of differentiated instruction in English language teaching.
The focus of this discussion, thus, is the potential use of internal consistency and factor analyses in determining the reliability and validity of a research instrument. These analyses are consistent and stable indicators in justifying the reliability and validity of the items on an instrument. Nonetheless, more studies replicating the procedures discussed here would further enhance the reliability and validity of the instrument. In this regard, the researcher believes that the data gathered through MoDiELT would provide beneficial information not only for teachers to plan their lessons, but also for curriculum designers, in the teaching and learning of English language especially in Malaysian schools context. (Teachers, educators, or researchers may email the main author for the MoDiELT questionnaire for research and training purposes).