Electronic copy available at: http://ssrn.com/abstract=1578439

Rubrics as an evaluation tool in Economics By:

Jhon James Mora R.1 Department of Economics Icesi University, Colombia

Abstract: Critical thinking and writing skills are necessary competencies for effective communication, problem resolution, and analysis in economics. Although there are several different approaches to evaluate critical thinking and writing skills, rubrics is an evaluation tool which provides more information than numerical grades because it establishes a set of criteria which both professors and students know and are able to evaluate jointly. Meanwhile, rubrics provide students more effective feedback about critical aspects of a given course such as the identification of problems, the quality of evidence furnished by the students, accuracy of conclusions, and objective reflection, among others. Keywords: Teaching economics, rubrics, evaluation. JEL: A22, I21 1.- Introduction Changes in evaluation approaches stem not only from discussions about students' learning styles and developments in learning theories, but also from the latest technology developments and institutional needs with regard to keeping record of and evaluating competencies within the frame of national or international accreditation processes. Since the late 1990s economists have not only discussed the competencies expected from economists [Colander and Kramer (1987), Colander (2005 a,b)], but have also incorporated their teaching styles, evaluation methods, and strategies implemented for teaching economics in the classroom [Becker and Watts (1998, 2001) and Espinoza, Machicado, and Mackhlouf (2007)]. The way students are evaluated is undoubtedly one of the most complex challenges faced by economists. It is apparently simple to assign a grade and expect the resulting grade distribution to show the student's degree of progress. Nevertheless, what's the purpose of a grade if the associated feedback does not contribute a great deal to the mutual knowledge of professors and students at a given school? This paper discusses the most relevant aspects associated with the creation of rubrics as an economic evaluation tool and reviews previous experiences with the creation of a rubric in a Macroeconomic Theory course. The first findings of the use of this rubric show interesting results related to critical thinking and written communication skills, which were both evaluated using this approach at Icesi University. It also provides an introduction to a discussion of ranges of skills based

1 The author would like to thank Luisa Fernanda Bernat, Jose Hernando Bahamon and Hector Ochoa, for their comments, and those who attended the “I ENCUENTRO NACIONAL DE PROFESORES DE ECONOMÍA “EXPERIENCIAS PEDAGÓGICAS” in Asociacion de Facultades, Departamentos y Programas de Economia and Universidad de los Andes, Bogota 2009. I take full responsibility for any mistakes in this paper.Head of the Economics Department at Icesi University. Calle 18 No. 122-135. e-mail: jjmora@icesi.edu.co (Cali, Colombia).

Electronic copy available at: http://ssrn.com/abstract=1578439

on an analysis of the distribution of skills and, lastly, incorporates a matching-type methodology using control groups for discussing the results of mean values obtained with the aforementioned rubric. 2.- Teaching Economics Although economists have long engaged in a debate about teaching economics which dates back to the second half of last century, as reported by the American Economic Review, the National Council on Economic Education, Grinnel College, Stanford University, the Federal Reserve Bank of Minneapolis, the Teaching Training Program, several seminars held at the American Economic Association, and multiple publications in specialized economic journals such as the Journal of Economic Education and Education Economics, among others, it seems that the use of a blackboard and chalk in teaching economics continues to prevail to a large extent. This holds true, for example, in the United States where 83% of classroom time is dedicated to giving lectures, writing notes, and drawing charts on a board [Becker and Watts (1998, 2001) and Lora and Ñopo (2009)]. 2 Lora and Ñopo (2009) showed that Latin American economics students are highly dissatisfied with the teaching approaches used at several universities in Argentina, Bolivia, Chile, and México. They reported that this dissatisfaction is not due to a lack of interest in this program, lack of acknowledgment of or good rapport with the teachers, but rather to a lack of innovation, an inability to get students actively involved in the classroom, and the absence of current relevant contents. In general, the approaches to teaching economics in Latin America are fairly consistent among different teachers and universities in different countries where there is a lack of innovation in teaching [Lora y Ñopo (2009)] and a need for a more active use of available resources such as information technologies, diverse evaluation approaches, games, and experiments, among others [Becker and Watts (2001)]. Muñeton and Valencia (2004) reported that 73.7% of classroom time at 3 different universities in Antioquia, Colombia, is dedicated to giving lectures, writing notes, and drawing charts on a board. They also reported that, except for statistics and econometrics courses, in average, computer rooms are hardly ever used - although the use of these rooms is not very extensive, i.e. only 22%. Games and simulations are occasionally used, and experiments in the classroom are never conducted. Although students from these universities believe that other resources, such as computer-assisted presentations, facilitate learning, these kinds of other sources are not customarily employed. Meanwhile, it seems that teaching approaches are changing in Colombia as evidenced by the works on the use of information technology resources [Medina (2009)], cartoons [Villaveces (2009)], and experiments [Cardenas (2009)]. 3 It is worth noting that the use of resources is effective provided that students are involved in their learning process and, of course, if teaching approaches continue to be used where students are assumed to be recipients of knowledge transferred from teachers or books, the results are clearly not going to be optimal.

2 58% at the University of Murcia, Spain (Maria-Dolores 2009). 3 See Montenegro (2008) for a overview of the first Economics academic programs offered in Colombia.

In this respect, contributions from several theories of learning such as constructivism provide other tools for teachers to rethink their activities in the sense that they should acknowledge the fact that "students bring their own ideas with them to the classroom", and instead of passively adding the contents delivered by their teachers to their own ideas, students restructure this new information in order to link it with their own cognitive structures. Hence, students play a major role because they build their own knowledge and go beyond "copying" knowledge presented to them in lectures [Garfield (1995), Kennedy (1998)]. Consequently, students go to the university to "study and learn", while teachers design and manage learning experiences (by encouraging, guiding, questioning, and evaluating) that maximize the likelihood that students build their own knowledge” (Gonzales, 2009). 3.- Rubrics A rubric is an evaluation tool that is typically used in subjective assignments. In this way, a rubric is used to evaluate a student's performance based on the sum of a full range of criteria rather than a single numerical score. Rubrics are useful for generating a high objectivity level and providing a clear description of the expectations of a given school or department with regard to the students' learning performance. As a result, it clarifies expectations before allocating a grade and allows for feedback on the student's weaknesses after grading the assigments. Rubrics also help teachers evaluate performance and communicate it to the students without the need to go into extensive comments about the assignment.

Rubrics tell students what is expected from them, the grading criteria to be considered or excluded, and the way their assignments are graded. It provides objectivity to subjective grades, facilitates grading, and help students understand how their performance was graded, their strengths, and weaknesses.

Rubrics also provide students with an understanding of how a given score was arrived at. Thus, rubrics evaluate student performance based on the sum of an overall range of criteria instead of just a numerical score.

In the case of written assignments, the components of a rubric include the following: 1.- Title of the rubric: It is advisable to ensure clear communication with the students. 2.- Scale: Possible scores in each grading scale. 3.- Scope of evaluation: A set of skills to be evaluated 4.- Descriptors: Descriptors must be established for each performance level to make scores believable and unbiased. There are several websites of Economics departments showing different applications of rubrics such as Hauge et-al (2009) from the Economics Department at Ripon College. There is also the website of the Mihaylo College of Business and Economics at CSU-Fullerton, which shows a wide range of categories used by the University of Winona and various other universities worldwide.

4.- Rubrics at Icesi

Since 2007 the School of Business Management and Economics at Icesi University began to make adjustments to conform to the requirements of an international accreditation process with The Association to Advance Collegiate Schools of Business (AACSB). In the frame of this process, the professors who taught the course "Macroeconomic Theory I" got together and made adjustments to make the course syllabus focus on the pursuit of two learning goals that must be reinforced, i.e. improving critical thinking and oral/written communication skills. The starting point for establishing the measuring approach was previous reviews of the course, and the rubric methodology was selected as asessment tool to measure students’ work. With regard to the grading scales, the following four categories were agreed upon: no evidence, confusing, partial with minor inaccuracies, and totally using the numerical equivalent grades shown in Table 1 below. With regard to skills, it was agreed that critical thinking skill include: analysis based on economic theory using models, charts, and algorithms, and arguments for value judgments of reading passages (own comments). The following was agreed with regard to written communication skills: rationale for writing, grammatical construction, and spelling. With regard to the presentation of documents, the following was agreed: identification of sections and titles, page numbering, quotes, references, bibliography, and the incorporation of charts, tables, and figures. The students prepared a review about the report submitted by the Colombian Central Bank to the Colombian Congress. The deadline for submitting the review was 15 days. Specific information was gathered about two courses in the second semester of 2009. The students in these courses were provided with information about the rubric, including evaluation criteria, grading, and skills to be evaluated. The number of students in both courses was 50. In average, they earned a grade of 3.1 on a 0-to-5 scale as shown in the table below:

Table 1. Statistics of Variables.

Variable Remarks Mean Std. Dev. Minimum Maximum

Final grade 50 3.15 0.96 0.00 5.00

Critical thinking skills grade 50 3.20 1.17 0.00 5.00

Written communication skills grade 50 2.84 1.22 0.00 5.00

Document presentation grade 50 3.28 0.97 0.00 5.00

Same school 50 0.48 0.50 0.00 1.00

Age 50 22.28 3.39 18.00 31.00

Economics 50 0.30 0.46 0.00 1.00

Sex 50 0.52 0.50 0.00 1.00

Cumulative average 50 3.85 0.24 3.49 4.47 Source: Author calculations.

Out of the abilities evaluated, written communication was the skill in which the students earned the lowest average. It was also the skill with the highest standard deviation. Other data about the students include the fact that almost 50% of the students graduated from the same school, they are

mostly women, and the percentage of students who attends either of the Economics programs available at the University is 30%. One of the advantages of rubrics - what actually makes them important - is that they provide both professors and departments aggregate information about the development and evolution of competencies of a group of students. This allows for more comprehensive review and feedback about the programs. This is particularly shown in the following table. Table 2. Criteria and results of using the rubric.

Source: Author calculations.

The results in Table 2 above show that the aspect where positive results regarding the development of skills are most evident is critical thinking skills, i.e. 12% vs. 6% for written communication skills. However, these results also show that more progress is yet to be made in critical thinking and written skills. An additional consideration that should be taken into account in reading and analyzing the results is the threshold for each of the categories. This is shown in the following Kernel figure illustrating the distribution of skills. Figure 1. Distribution of skills

Source: Author calculations.

0

.1

.2

.3

.4

Den

sity

0 2 4 6 Critical thinking skill

Density Gaussian density

0

.1

.2

.3

.4

Den

sity

0 2 4 6 Written communication skill

Density Gaussian density

Not evidenced

Grade: (0 - 2)

Confusing unclear

Grade: (2,1 - 3,0)

Partial with minor inaccuracies Grade: (3,1- 4)

Totally

Grade: (4,1 - 5)

Final grade

Critical thinking skill 2,50 [ Med=3.20; Sd=1.17 ] 28% 18% 42% 12% 100% Written communication skill 1,25 [ Med=2.84; Sd=1.22 ] 32% 12% 50% 6% 100% Document presentation 1,25 [ Med=3.28; Sd=0.97 ] 24% 8% 64% 4% 100%

Aspect Highest grade

Evaluation of results

Silverman's (1981) technique was used for determining whether the data revealed the existence of one or more than one mode. Silverman's (1981) test is based on "smoothing" the critical bandwidth. Let us now consider the problem of testing the null hypothesis that density f has k modes versus the

alternative that f has more than k modes. Let kh be the critical bandwidth; i.e. the smallest value of h

so that );(ˆ hxf has k modes. A large value of kh compared to 1ˆkh shows that strong smoothing is

needed in order to create and estimate density having only k modes. Silverman's procedure (1981) employs the bootstrap technique [Efron (1979)] to evaluate the significance of an observed value of

ˆkh . It is based on the Achieved Significance Level (ASL), and this property of monotonicity is unique

when the core is Gaussian [Babaud J., A. Witkin, M. Baudin and R.O. Duda. (1986) and Minnotte and Scott (1993)]. Hence, if we use a Gaussian core, 100 replicas, and an average of 30 histogram changes, and select the lowest h in Silverman's test, the following was determined: 4 Table 3. Number of modes

Critical thinking skills Written communication skills

Number of modes P-value Number of modes P-value

1 0.06 1 0.16

Source: Author calculations. According to Martínez-Ramírez, Mayorga-Saucedo, and Salgado-Ugarte (1998), if the p-value is greater than 0.1, then the hypothesis that holds that the distribution is unimodal is acceptable. Therefore, based on table 3 above, it can be concluded that the distribution of critical thinking skills is not unimodal and the distribution of written communication skills is unimodal5. It will thus be necessary to review the thresholds in the future in order to improve identification of the results of a group. Lastly, another analysis was performed in order to test how different the results of a review were vs. those from the immediately preceding semester. Because the rubric cannot be used with the students who have previously completed the Macroeconomic Theory I course, but it can be used with students who will take this course for the first time, two different groups of students are considered. A control group consisting of students who completed the Macroeconomic Theory I and the other will be the treated group comprised of students who will attend the same course. There were 50 students in the groups in which the rubric was used. As mentioned above, the students earned a mean grade of 3.15 on a 0-to-5 grading scale (zero being the lowest and five being the highest grade) with a standard deviation of 0.957. The number of students in the control groups was 40. Their mean grade was 3.73 on a 0 to 5 grading scale (1.68 being the lowest and 4.8 being the highest grade). 6

Thus, a matching-type analysis was performed in order to determine the impact of a measurement on two different individuals. In particular, let z be a binary variable that indicates the application of a rubric or treatment in such a way that the students on which the rubric was used, z takes the value of

4 We also calculated the critical bandwidth using Least Squares CV, Biased CV, Smoothed CV, Jones, Marron and Park CV, Park and Marron plug-in and Sheater and Jones plug-in. The results are available from the authors on request. 5 See Mora and Nuñez (2009) for the test of the unimodality in the distribution. 6 We used in a control group two different teachers from those included n the treatment group.

one. Let y1 be the grade earned by those students on which the rubric y was used, and let y0 be the grade earned by those students on which the rubric was not used.

Then, let us define a random vector (y0,y1,x, z) for the population of interest, whereas x is a vector of explanatory variables in such a manner that an individual i will have (y0i,y1i ,xi, zi). Based on the above it is possible to calculate the average treatment effect (ATE), the average treatment effect for those students who were graded using the rubric (ATT), and the average treatment effect for those students who were graded without using the rubric (ATC). The results were as follows:7

Table 3. Effect of treatment.

Value p Value p

ATE -0.58 0.004 -0.40 0.053

ATT -0.48 0.019 -0.34 0.107

ATC -0.71 0.001 -0.47 0.049

Source: Author calculations. Note: The following variables were used as control variables: same school, age, cumulative,

average, sex, includes Economics; N=90.

The results in Table 3 show that the average grade dropped by 58 tenths in the rubric-based system, and if we consider the effect of having Economics students in these courses, this reduction equals 40 tenths on the grade earned by the students when the rubric was used. Meanwhile, the effect of the rubric on the results of those students who participated in the use of the rubric is statistically significant and, as in the previous case, is also negative. The statistical differences between the courses that used the rubric and those who did not suggest that, when it comes to grading, teachers probably do not take into account all of the competency evaluation factors described in the rubric. Therefore, they implicitly allocate different weights to the factors that they identify, which could account for these differences. 5.- Conclusions Rubrics are useful for defining and specifying the expectations of a given school or department with regard to their students' learning performance. Rubrics also provide objectivity to subjective grades, facilitate grading, and enable students to have a better understanding of their strengths and weaknesses. It is essential that rubrics have a title, a grading scale, a scope of evaluation, and descriptors. In order to accomplish optimal results, it is advisable not to return grades to the students without providing them qualitative results and allowing a certain period of time for them to react to these results. Rubrics supply information that can be used as feedback only about previous courses (about skills that should have been introduced and probably reinforced in previous courses), but also about students who are just beginning a course. Rubrics were selected as an evaluation method at Icesi because, first of all, it allows analyzing the general results of a group with regard to the skills that are going to be introduced, reinforced, and

7 For further explanation, see Wooldridge (2002).

evaluated, and secondly, in the past, students were asked to write essays, but they were not provided with proper feedback. The results on the Macroeconomic Theory I course demonstrate that the critical thinking skill is where there is the highest percentage of students in the "totally" category. The results also show that the skill that needs to be emphasized or stressed is written communication. An analysis of the distribution of skills reveals that the numerical evaluation of skills should be reviewed in the future in order to adjust these levels. Lastly, rubrics enable all faculty members in a given discipline to reach a consensus about the weight of each evaluation factor with regard to the final grade, which should be aligned with the learning objectives of each course. 6.- Bibliography Becker, W. (2000). “Teaching Economist in the 21st Century”, Journal of Economics Perspectives V.14 No. 1. Becker, W. y M. Watts. (1989). “Teaching economics: What was, is, and could be” en Becker y Watts (comp). Teaching Economics to Undergraduates: alternatives to chalk and talk. Edward Elgar Publishing Limited. Cardenas, J.C. 2009. Experimentos económicos para el aprendizaje interactivo en economía" en ENCUENTRO NACIONAL DE PROFESORES DE ECONOMÍA “EXPERIENCIAS PEDAGÓGICAS” http://www.afadeco.org.co/sites/default/files/AFADECO%20Bogota%202009%20experimentos.pdf Colander, D. y A. Klamer (1987). "The Making of an Economist", Journal of Economic Perspectives, 1 (2), pp. 95-111. Colander, D. (2005a) “The Making of an Economist Redux”, Journal of Economic Perspectives, 19 (1), pp. 175–198. Colander, D. (2005b) “What Economists Teach and What Economists Do”, Journal of Economic Education, 36 (3), pp. 249-261 Espinoza, L., Machicado, C.G., K. Makhlouf. (2007). “La Enseñanza de la economía en Bolivia y Chile, Serie de Documentos de trabajo sobre desarrollo No. 10, Instituto de Estudios avanzados en Desarrollo. Gonzales, H. (2009). El proyecto educativo de la universidad Icesi y el aprendizaje Activo. Cartilla Docente No 22. Decimo quinta edición, Publicaciones del CREA. Agosto. Hauge, S., Huang, R., Schoofs, P and Dima Zhosan, (2009). (Infrastructure investment and economies of scale in program assessmen. Economics Department, Ripon College http://bioquest.org/teagle/Ripon-economics-assessment-conference-poster.pdf

Lora, E y H. Ñopo. (2009). La formación de los economistas en América Latina. Documento de trabajo del BID No. 119 Maria-Dolores, R. (2009). “La enseñanza de teoría económica dentro del espacio europeo de educación superior: algunas reflexiones”, Revista d`innovació educativa no. 3, pag: 97-102. Martínez-Ramírez, J., E. Mayorga-Saucedo e I.H. Salgado-Ugarte. (1998). “ Estimación de la eficiencia de un método no paramétrico para probar la multimodalidad de datos univariados”. Memorias del XIII Foro Nacional de Estadística, I.I.M.A.S. UNAM, Asociación Mexicana de Estadística, INEGI,Centro Estudiantil, ITESM, Monterrey, México. 5 al 9 de octubre de 1998: 84-92. ISBN: 970-13-2111-1 Mora, J.J. and Nuñez, J.J. 2009. Economic Convergence of Income Distribution Worldwide from 1986 to 2000, Journal of Economic Studies 36, no. 6, p.p:675-691 Medina, C. (2009). Tecnología en la Enseñanza para el Mejoramiento del Aprendizaje-TEMA" en I ENCUENTRO NACIONAL DE PROFESORES DE ECONOMÍA “EXPERIENCIAS PEDAGÓGICAS” http://www.afadeco.org.co/sites/default/files/TEMA%20Catherine%20Medina.pdf Montenegro, A. (2008). Los primeros programas de economía en Colombia. W.P. No. Departamento de Economia, Facultad de Ciencias Económicas y Administrativas. Muñeton, G., Mauricio G. Valencia. (2004). Métodos de enseñanza en economía: alternativas a la tiza y el tablero. Oikos, No 18. Soren Hauge, Roc Huang, Paul Schoofs and Dima Zhosan, Economics Department, Ripon College (2009). “Infrastructure investment and economies of scale in program assessment http://bioquest.org/teagle/Ripon-economics-assessment-conference-poster.pdf Salgado-Ugarte, I.H. (2002). Métodos estadísticos no paramétricos de suavización para análisis de datos biológicos. FES Zaragoza y DGAPA (PAPIIT IN217596; PAPIME 192031), 139 p. (7capítulos), I.S.B.N: 970-32-0292-6. --------- I.H., M. Shimizu y T. Taniuchi, (1997). “Nonparametric assessment of multimodality for univariate data”. Stata Technical Bulletin, 38, p.p: 27-35. Silverman, B.W. (1981). “Using kernel density estimates to investigate multimodality”, Journal of the Royal Statistical Society, B, 43, p.p: 97-99. --------. (1986). Density estimation for statistics and data analysis. London: Chapman & Hall. Steven C. Myers & Michael A. Nelson & Richard W. Stratton, 2009. "Assessing An Economics Programme: Hansen Proficiencies, ePortfolio, and Undergraduate Research," International Review of Economics Education. vol. 8(1), pages 87-105. Villaveces, J. (2009). El uso de la caricatura para la enseñanza de la historia económica" en I ENCUENTRO NACIONAL DE PROFESORES DE ECONOMÍA “EXPERIENCIAS PEDAGÓGICAS”http://www.afadeco.org.co/sites/default/files/Caricatura%20para%20la%20enseñanza%20de%20la%20historia%20económica%20Juanita%20V.pdf

Wooldridge, J. (2002). Econometric Analysis of cross section and panel data. The MIT press, Cambridge Mass.