Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851

1877-0428 © 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of Prof. Hüseyin Uzunboylu.doi:10.1016/j.sbspro.2011.12.154

Procedia Social and Behavioral Sciences Procedia - Social and Behavioral Sciences 00 (2011) 000–000

www.elsevier.com/locate/procedia

WCLTA 2011

College performance prediction test

María Vallejosa *, Jesús M. Alvaradob, Anibal Puenteb aUniversidad Peruana Unión, Engineering and Architecture Department, Carretera Central Km. 19, Ñana, Lima, 15,3564, Peru

bUniversidad Complutense de Madrid, Biofunctional Studies Institute, Paseo Juan XXIII, 1, 28040 Madrid. Spain

Abstract

The purpose of this study was to construct and evaluate the psychometric properties of an instrument that is designed to measure attitudinal, affective, and emotional factors relevant in predicting academic performance in engineering students. The developed instrument makes it possible to obtain a student’s profile in terms of his/her attitudes toward science, motivation and anxiety levels that have been revealed to be important factors in previous studies. The research has been developed in various universities in Peru and in different phases: an initial pilot phase with 81 university students; a second phase where the test was applied to 866 engineering students from different socioeconomic and cultural development stratum. The results, after making corrections to the original instrument and verifying the instrument's stability over time (test-retest analysis), makes it possible to affirm that the proposed objective has been achieved, and that an instrument with acceptable reliability and validity of construction is available. Additionally, it will be informed about the predictive ability of the instrument in different considered degrees, and in relation to socioeconomic and cultural aspects of the students. © 2011 Published by Elsevier Ltd.

Keywords: Prediction Test; Validity; Academic Success; University Studies

1. Introduction

Among the factors that influence academic performance we are determined to find motivational, attitudinal and personality factors, and if teachers are aware of these they may be an important tool for teaching their courses [1]. Emotions such as interest, surprise, curiosity, or anxiety may be experienced in the phase of the learning process [2]. All these emotions refocus attention on task processing, memory, or contextual information. These factors may be relevant to the representation of the learning process and therefore facilitate the representation of the task that leads to learning, and is reflected in academic performance. According to the Metacognition With Motivation and Affect in Self-Regulated Learning model (MASRL), which emphasizes the interactions between metacognitive, motivational and affective self-regulated learning factors, it is mentioned that this may be a useful framework for understanding the complex processes involved in learning and the dynamic nature of the person’s self-regulation in the development of a task, which are mostly learning self-regulated stable patterns distinguished over time [2]. Therefore, the purpose of this study is to design, build and evaluate a psychometric instrument that is designed to

* Maria Vallejos. Tel.: 051-969332643 E-mail address: mariav@upeu.edu.pe

© 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of Prof. Hüseyin Uzunboylu.Open access under CC BY-NC-ND license.

Open access under CC BY-NC-ND license.

847María Vallejos et al. / Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851 María Vallejos et al./ Procedia – Social and Behavioral Sciences 00 (2011) 000–000

2

measure attitudinal, affective, and emotional factors that are considered relevant in predicting academic performance in engineering students in Peru.

1.1. Attitudes toward Science

The study considers attitudes as an important factor in academic success, since the attitude is the performance of a person and it demonstrates his adopted behavior. Attitudes refer to rules or behavior standards, internal policies behaved externally in a consistently way. By characterizing the attitudes as an educational content Wayne [3] identifies three attitudinal components: cognitive, behavioral and emotional. These components should keep a certain balance so that the attitude can be long-lasting and transferable; professors and university students, especially from the Engineering Department, should be aware of their attitudes toward science so that they can change others’ attitudes to achieve the goals set at the beginning of their career. This indicator was adopted as an important factor in the PISA evaluation 2006, which introduces the attitude toward science as a dimension considering how to broaden the knowledge of properties, laws and theories, supporting scientific research, and valuing the work of science and what to expect from it and accountability to the environment. Salonen et al. [4] noted that many teachers have a different conception of what their students feel when they perform their tasks or write up their reports. They do not know their students’ attitudes and these teachers may have implications in what they expect from their students, which would affect in what they do or say to regulate their students' learning.

1.2. Motivation toward Learning

El Human development has required many skills, which cannot be developed without an element of reward, whether self-satisfaction or acceptance of others in the short or long term. Motivation is a feature of every person, which remains relatively constant over time and is quite difficult to modify. Tapia [5] states that a student is not motivated or unprovoked in the abstract, but in terms of how much does academic work means to him. Covington [6], affirms that motivation over learning achievement is based upon the accumulation of research and concludes that the quality of student learning and the willingness to monitor learning depends on an interaction between the types of social and academic goals. Motivation depends on the properties of these objectives and reward structures; it works upon beliefs and goals which are attached at a certain moment. That is, cognitive representations that would achieve or avoid a decisive influence on this motivation is intrinsic or extrinsic, or absence of motivation (lack of motivation) to learn. The motivation and effort will result in good performance [7], leading to decreased sensation of difficulty [8]. Motivation, such as task processing, can take an intrinsic form (eg, enjoyment of a task processing) or unpleasant feelings (eg, boredom) or the state of anxiety experienced as an increased arousal, worrying, and intrusive thoughts [9, 10]. There is also the motivation that arises from emotional states, such as surprise either from some unexpected event or a disagreement that has occurred. It could also be if some new information has been inserted [11], or from curiosity, because there is no denial of information that should be there for one to be able to continue task processing [12].

Therefore, it is not only intrinsic motivation, it also requires other forms of interest or pleasure only [13]; these are metacognitive experiences that contribute to motivation in academic performance [14]. For example, the feeling of knowledge and that it can be used at any time [15] creates the expectation that the required information is available in the memory, although at the moment it may be inaccessible despite the efforts which are being made. Also, when a student feels difficulty during the work process, awareness of this feeling provides information for effort regulation [16]. Skaalvik and Skaalvik [17] discovered that self-concept in Mathematics, intrinsic motivation, and the usage of learning strategies, are related to students’ achievements through the school years and this is also true of adult students. The model MASRL postulates that motivation which generates the ability to perform tasks, helps to arise motivation, especially the intrinsic type, among students [2].

848 María Vallejos et al. / Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851 Maria Vallejos et al./ Procedia – Social and Behavioral Sciences 00 (2011) 000–000

3

1.3. Anxiety

Anxiety in the learning process becomes a barrier for the development of the cognitive process. Craske [18] suggests that cognitive processing becomes less developed because of concerns of possible future events, (anticipatory anxiety) when these events are imminent; fear or panic when danger or threat is present then prevents the cognitive activity from being developed. The conceptual demands, such as mathematic tasks and the complexity of the task, have also been shown to differentiate the performance and self-validation of the difficulty sensation state [19, 20, 21], contributing to increased anxiety in the development of necessary tasks in the learning process.

Pekrun, Elliot, and Maier [22] also discovered that emotions positively affect academic achievement and are involved in the effects of achievement goals and achieving results. This one probably makes use of academic strategies through the effect of emotion [23]. This evidence suggests that positive emotions may have a direct effect on metacognitive skills, as well as an indirect effect through motivation. Moreover, although there is growing research evidence on the interactions between metacognition, motivation, and how these affect the level of a person and its effects on school performance, these mechanisms that unites them with school performance are not clear [2], this being an important reason to develop the present study which proposes an instrument that allows measuring the constructs of motivation, attitudes towards science, and anxiety in the academic activities which will serve the teacher as a reference for handling educational processes and the timely adjustment of students' emotions.

2. Methodology

2.1. Participants

Two samples were formed in a first pilot study. In the first phase of this research, 81 students from two universities in Lima participated (private and community universities), ranging from 15 to 27 years of age with a mean of 18.6 years of age (SD = 2.7). This first pilot study was conducted with the purpose of testing and to debug the measurement instruments developed for this research. In a second study 866 students participated; 281 women, and 581men—freshmen and sophomores—of 9 different Engineering careers, and from 9 different universities in Peru’s three regions (coast, highlands, and jungle); with a range of 15 to 38 years of age; with a mean of 18.7 years age (SD = 3.0). Samples were collected by inviting all students to voluntarily participate in the study.

2.2. Instrument

A battery of instruments was adapted and developed for Predicting Performance in Engineering Careers (PPEC) involving the constructs of attitude toward science, motivation, learning and anxiety about academic activity. The battery consists of 16 items of attitude towards science, 12 items of motivation, 5 items of lack of motivation, and 15 items of anxiety items; each item is answered on a Likert scale of four points that assesses how the student feels, thinks or acts in different situations.

2.3. Procedure

The procedure was developed in two phases. The first one involved the construction of the instrument; and the second one, the analysis of the validity and reliability of the instrument.

2.3.1. First Phase The initial instrument consisted of 108 items, 9 items of general information, 34 items about attitude towards

science, 25 items about motivation, and 40 items about anxiety. The items considered for attitudes toward science were built taking in account the 2 to 3 questions for each indicator presented by Pozo and Gomez [24]; in the items considered to measure the motivation as an instrument proposed by Carratala [25] was adapted, which considered 25

849María Vallejos et al. / Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851 María Vallejos et al./ Procedia – Social and Behavioral Sciences 00 (2011) 000–000

4

items; the items considered to measure anxiety is based on the instrument proposed by Calvo [26]; and the State-Trait Anxiety Inventory (STAI, for adolescents and adults), developed by Spielberger et. al. [27].

After a thorough analysis of information provided by a group of experts and the results obtained from data provided by participants in the study of the pilot sample and the analysis of test-retest to see temporary stability of the instrument, corrections were made in the problem areas and some items that did not provide much information were removed. As a result we had a new improved version of the instrument consisting in 60 items.

2.3.2. Second Phase Once the measuring instrument has been debugged we proceeded to examine the reliability and validity of the

instrument in a large sample from around the country. The analysis of this second phase was recommended as the final instrument, this one was formed by 48 items, of which 16 items measure attitudes toward science, 12 items about motivation, 5 items measure lack of motivation, and 15 items about anxiety. This tetra-factorial structure shows a good fit to the theory and data based on the results of confirmatory factor analysis performed.

3. Results

3.1. Evidence of Content Validity

To guarantee that in the instrument both the structure and the content of the construct were properly reflected, it was submitted to an experts committee, consisting of seven judges, doctors of Educational Psychology, Food Engineering, Systems Engineering, and Master’s in Research and University Teaching, all of them with a well known academic experience at a university’s level, graduate school, and research. Furthermore, to the participants in the pilot stage, it was aimed to obtain information and feedback from the formulation of items and other formal aspects of the instrument. These results were evaluated using the coefficient of validity V [28].

3.2. Reliability analysis

The internal consistency of the PPEC was assessed by Cronbach's reaching a value of .771 indicating an acceptable reliability for the initial version, since the reliability indices above .70 are adequate if the purpose of the scale is the research, although our ultimate goal is the diagnosis or classification, therefore it was necessary to improve the instrument and set it above .80, which is a higher value [29]. On the other hand, the low Cronbach’s coefficients obtained in the attitude dimensions .441 and motivation .461, recommended a review of the instrument that only worked optimally for measuring anxiety, being alpha .807. Nevertheless, the internal consistency of the final instrument version reaches a value of .897 indicating good reliability and Cronbach's coefficients per dimension or scale are: attitude towards science .73; motivation .73; the new dimension of lack of motivation .75; and anxiety .89; values that indicate increased of reliability compared to the initial instrument and ultimately these were acceptable values.

Table 1. Descriptive statistics in the test and retest.

Statistics Attitude Motivation Anxiety Global

Test Retest Test Retest Test Retest Test Minimum 47.0 42.0 50.0 45.0 43.0 40.0 148.0 143.0 Maximum 72.0 74.0 81.0 80.0 78.0 78.0 225.0 230.0

Mean 58.9 59.1 67.1 64.9 61.9 60.5 187.8 184.5 Standard Dev. 5.8 5.9 6.6 7.0 8.1 8.7 17.1 18.1

To assess the temporal stability of the test a test-retest method was given by applying the instrument to each of

the subjects in two sessions, with an interval of three weeks. The scores obtained on the constructs: attitude toward science, motivation, and anxiety in academic performance on both evaluations are presented in Table 1. The descriptive statistics by construct applied to students in the study of test-retest reliability, underscore the normality

850 María Vallejos et al. / Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851 Maria Vallejos et al./ Procedia – Social and Behavioral Sciences 00 (2011) 000–000

5

of the studied population. Most of the results obtained by the subjects revolve around the mean, indicating scarce variability in the sample.

3.3. Evidence of construct validity

The evaluation of the factorial structure and PPEC construct has been performed using the LISREL structural equations program (International Scientific Software) and the fit indices obtained were: goodness of fit index (GFI) = 0.93, adjusted goodness of fit index (AGFI) = 0.92; and the Root Mean Square Error of Approximation (RMSEA) = 0.07 which indicates a good fit of the data to the model hypothesized. Regarding the relationship between the constructs, it showed a high positive relationship between attitude toward science and student motivation, and a negative relationship between, attitude toward science regarding anxiety and lack of motivation (the two latter ones strongly related); in this way the battery has adequate reliability and constructs validity.

4. Conclusions

Social-Cognitive Theory of Career Development (SCCT), proposed by Lent, Brown and Hackett [30] whose model considers three components: social cognitive approach, self-efficacy beliefs, and outcome expectations and goals. These 3 components have received much attention from researchers, including studies to verify their proposals about racial minorities, college and senior high schools students [31, 32, 33, 34]. Models of vocational and educational interests, career choice and academic performance have been supported by the literature [35].

Coleman's report [36], is another model that explains the academic performance by three factors: individual, school, and family / community [37]. The individual factor involves the motivation, values and the capacity or skill to focus and study.

In this investigation we have considered the importance of motivation when studying, the desire to succeed, tenacity and effort led to a goal that has to give satisfaction, social prestige and the possibility of academic achievement in students. Anxiety control when facing academic performance, and the attitude that the student has about academic activity, are important factors in the success of human activity and will determine the significant percentage of students who pursue a career in Engineering and complete their university degrees.

This study presents a psychometric instrument to predict academic performance in engineering careers considering the constructs: attitude toward science, motivation / lack of motivation, and anxiety. Using a Likert scale with scores of 1-4 to for answer options in each item; we evaluated the technical requirements of validity and reliability, and the results are directly related to the characteristics of the sample to which the test was given [38]. Both the results of the studies regarding the reliability and the validity of content, and the instrument construct, are acceptable to showing a good fit to the theoretical model proposed. Consequently, PPEC can be considered an instrument for obtaining profiles of attitudinal and personality factors that influence university academic achievement.

Acknowledgements

The researchers wish to thank Universidad Peruana Union and the Engineering and Architecture Department at Universidad Peruana Union and the proyect CCG10UCM/HUM-5158 from Universidad Complutense de Madrid (UCM & DGUI) of Spain. Without these supports, this study would have not been achieved.

References

[1]. Patrick, H. Anderman, L.H. Bruening, P.S. & Duffin L.C. (2011): The Role of Educational Psychology in Teacher Education: Three Challenges for Educational Psychologists, Educational Psychologist, 46(2), pp. 71-83

[2]. Efklides, A. (2011): Interactions of Metacognition With Motivation and Affect in Self-Regulated Learning: The MASRL Model, Educational Psychologist, 46(1), pp. 6-25

851María Vallejos et al. / Procedia - Social and Behavioral Sciences 31 (2012) 846 – 851 María Vallejos et al./ Procedia – Social and Behavioral Sciences 00 (2011) 000–000

6

[3]. Wayne, W. (2010). Psychology: Themes and Variations. (8th ed.) Las Vegas: Cengage Learning. (Chapter 9). [4]. Salonen, P., Vauras, M., & Efklides, A. (2005). Social interaction: What can it tell us about metacognition and co-regulation in learning?

European Psychologist, 10, pp. 199–208. [5]. Tapia A. (1997). Orientación Educativa, Teoría, Evaluación e intervención. (Educational Orientation, Theory, Evaluation and

Intervention) Madrid: Edit. Síntesis. [6]. Covington, M. (2000). Goal Theory, Motivation, and School Achievement: An Integrative Review, in Annual Review of Psychology, 51,

pp.171-200. [7]. Erez, A., & Isen, A. M. (2002). The influence of positive affect on the components of expectancy motivation. Journal of Applied

Psychology, 87, pp. 1055–1067. [8]. Efklides, A., & Petkaki, C. (2005). Effects of mood on students’ metacognitive experiences. Learning and Instruction, 15, pp. 415–431. [9]. Eysenck, M.W., Derakshan, N., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory.

Cognition and Emotion 7, pp. 336–353. [10]. Sarason, I. G. (1988). Anxiety, self-preoccupation and attention. Anxiety Research, 1, pp. 3–7. [11]. Teigen, K. H. & Keren, G. (2002). When are successes more surprising than failures? Cognition and Emotion, 16, pp. 245–268. [12]. Litman, J. A. (2005). Curiosity and the pleasure of learning: Wanting and liking new information. Cognition and Emotion, 19, pp. 793–

814. [13]. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-

being. American Psychologist, 55, pp. 68–78. [14]. Koriat, A. (2000). The feeling of knowing: Some metatheoretical implications for consciousness and control. Consciousness and Cognition

9, pp. 149–171. [15]. Metcalfe, J. (2000). Feelings and judgments of knowing: Is there a special Noetic state? Consciousness and Cognition 9, pp. 178–186. [16]. Efklides, A., Kourkoulou, A., Mitsiou, F., & Ziliaskopoulou, D. (2006). Effort regulation, effort perceptions, mood, and metacognitive

experiences: What determines the estimate of effort expenditure? Metacognition and Learning 1, pp. 33–49. [17]. Skaalvik, S., & Skaalvik, E. M. (2005). Self-concept, motivational orientation, and help-seeking behavior in mathematics: A study of

adults returning to high school. Social Psychology of Education 8, pp. 285–302. [18]. Craske, M. G. (1999). Anxiety disorders: Psychological approaches to theory and treatment. Boulder, CO: Westview Press. [19]. Efklides, A., Papadaki, M., Papantoniou, G., & Kiosseoglou, G. (1997). The effects of cognitive ability and affect on school mathematics

performance and feelings of difficulty. The American Journal of Psychology, 110, 225–258. [20]. Efklides, A., Papadaki, M., Papantoniou, G., & Kiosseoglou, G. (1998). Individual differences in feeling of difficulty. The case of school

mathematics. European Journal of Psychology of Education, 13 (2), 207-226. [21]. Efklides, A. (2002). The systemic nature of metacognitive experiences: Feelings, judgments, and their interrelations. In M. Izaute, P.

Chambres, & P.-J. Marescaux (Eds.), Metacognition: Process, function, and use (pp. 19–34). Dordrecht, The Netherlands: Kluwer. [22]. Pekrun, R., Elliot, A. J., & Maier, M. A. (2009). Achievement goals and achievement emotions: Testing a model of their joint relations

with academic performance. Journal of Educational Psychology, 101, pp. 115–135. [23]. Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A

program of qualitative and quantitative research. Educational Psychologist, 37, pp. 91–105. [24]. Pozo J. & Gomez A. (1998). Aprender y enseñar ciencia: Del conocimiento cotidiano al conocimiento científico. (Learning and Teaching

Science:From the Everyday Knowledge to Scientific Knowledge) Published by Morata. ISBN 8471124408, 9788471124401. [25]. Carratala, E. (2007). Análisis de la teoría de las metas de logro y de la autodeterminación en los planes de especialización deportiva de la

Generalitat Valenciana. (Theory Analysis of Achievement Goals and the Generalitat Valenciana Self-determination in Sport Specialization plans) España: Universidad de Valencia – Psychology Department. p 72.

[26]. Calvo, A. (2001). Estrategias para aprender a aprender. (Strategies for learning to learn) CISSPRAXIS S.A. España. [27]. Spielberger, C. D., Goursch, R. L. & Lushene, R. E. (1982). Cuestionario de Ansiedad Estado - Rasgo, STAI (State-Trait Anxiety

Inventory STAI) . Madrid. TEA Ediciones. [28]. Escurra, L. (1988). Cuantificación de la validez por contenido por criterio de jueces en Revista de Psicología. (Quantification of Validity

Content by Judges Criteria in a Psychology Journal) Universidad Católica del Perú, Año VI, Vol. IV, Nº 1-2. [29]. Nunnally, J. & Bernstein, I. (1994). Psychometric theory. (8th ed.). Michigan: McGraw Hill, (Chapter 9). [30]. Lent, R., Brown, S. & Hackett, G (1994). Toward a Unifying Social Cognitive Theory of Career and Academic Interest, Choice and

Performance. Journal of Vocational Behavior 45, pp. 79-122. [31]. Fouad, N.A., & Smith, P. L. (1996). A test of a social cognitive model for middle school student: Math and Science. Journal of Counseling

Psychology, 43, pp.338-346. [32]. Gainor, K. A., & Lent, R. W. (1998). Social cognitive expectations and racial identity attitudes in predicting the math enrollment intentions

of Black college student, Journal of Counseling Psychology, 45, pp. 403-413. [33]. Lapan, R. T., Shaughnessy, P. & Boggs, K. (1996). Efficacy expectations and vocational interests as mediators between sex and choice of

math/science college majors: A longitudinal study, Journal of Vocational Behavior, 49, pp. 277-291. [34]. López, E. G., Lent, R., W., Brown, S. D. & Gore, P. A. (1997). Role of social-cognitive expectations in high school students´ mathematics-

related interest and performance, Journal of Counseling Psychology, 44, pp. 44-52. [35]. Flores, L. & O´Brien, K. M. (2002). The Career Development of Mexican American Adolescent Women: A test of Social Cognitive

Career Theory, Journal of Counseling Psychology, 49(1), pp. 14-27. [36]. Plowden, C. (1967). Children and their Primary Schools. London: HMSO. [37]. Murrillo, J. (2005) La investigación sobre eficacia escolar. (School Effectiveness Research) Edic. Octaedro. S.L. Barcelona, España. [38]. AERA, APA y NCME, (1999). Standards for educational and psychological tests. Washington DC: American Psychological Association,

American Educational Research Association, National Council on Measurement in Education.