Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-1

A Complex Tutoring System for e-Learning: The New Evaluation Model

Rafael Pastor, Roberto Hernández, Salvador Ros, Timothy Read, Manuel Castro, and Rocael Hernández

Spanish University for Distance Education: rpastor@scc.uned.es, roberto@scc.uned.es, sros@scc.uned.es, tread@lsi.uned.es, mcastro@ieec.uned.es

Galileo University: roc@galileo.edu

Abstract - UNED (Spanish University for Distance Education) is the largest public distance education university in Spain with over 220,000 students, 1600 lecturers and 2000 administrative staff. Currently, the EHEA (European Higher Education Area) adapted new degrees have just started, so focusing on personal activities for the evaluation model is very important. In UNED, the number of students (near of 42000 students in the 120 subjects of its first course) makes the role of tutors who assist lecturers in the new evaluation model proposed by EHEA very important. Also, tutors must use the same e-learning platform tools as lecturers, in order to have a course evaluation control panel that follows a student’s learning process. In this paper, the course evaluation system will be presented and how it is implemented in order to enable tutors to access only the grades of their students. As a result, some indicators will be presented in order to validate the approach of the new evaluation system for EHEA degrees: tutors with “activity” in their personal subgroups, number of degrees undertaken by the tutors, etc. Index Terms – Course planning, educational practices, adaptation, postgraduate courses.

INTRODUCTION

The Spanish University for Distance Education (UNED), [1], has recently started with its new degrees (adapted to the European Higher Education Area, EHEA). Traditionally, each course (old degrees) has a set of tutors that help lecturers in the student’s evaluation using traditional learning techniques. With the start of the new degrees, this situation has changed because now, tutors must work actively in the evaluation of students using technological tools based on an e-Learning platform (owned by UNED). Each degree course in UNED has a large number of students (some subjects nearly 8500 students) so the tutor activity is very important in order to get a successful continuous evaluation system. This evaluation system is based on two types of assessments: tasks, a deliverable must be sent using the e-Learning platform, and evaluation questionnaires, auto-evaluated or evaluated by tutors. So, the tutors must

work in the e-Learning platform in order to grade the student’s activities. In UNED, tutors are grouped in nine geographical zones distributed throughout Spain, so the tutors’ location is distributed all over Spain (in some cases, even outside). In the case of the new degrees, UNED has about 3600 tutors involved in the tutoring system (only in the first semester of 2009-2010 course) so it is very important to provide automatic mechanisms to grant tutors access to their subjects and, of course, to the evaluation system in the e-learning platform. This issue is handled by using ICT (Information and Communication Technologies) structures [2],[3]. In this case, the use of ICT techniques is mandatory in order to give full support to tutors, lecturers and students to the EHEA evaluation model. To provide this support, UNED has selected its e-Learning platform aLF (active Learning Framework, a system being developed by researchers in the School of Computer Science) [4]. Up until to now, tutors did not have to have access to the e-Learning platform in order to help lecturers in the traditional evaluation model. Instead, they sent tutorial reports about the student evolution in the “face to face class” and the grades for the “distance task” provided for the lecturers assigned to the subject (in subjects with a big number of students, typically four or five professors). But now, with the new degrees all this information is obtained using the evaluation mechanics provided by the e-Learning platform, so a central repository of tasks, degrees and related comments (filtered by student or tutor) is provided to the lecturers in order to validate the tutors’ grades of their students’ work. Additionally, a sub-group of the students assigned to tutor is generated automatically in order to provide a “virtual” communication space between students and the tutor. This sub-group can be used as a “virtual class” and has all the tools available as the main course (material, planning, calendar, forums, chat, etc.). So, the idea is to move the “face to face” communication (tutoring time elapsed in the real classes) to the virtual space in order to maximize the tutors’ presence to their students (included those that are not able to go to the tutoring time). This feature is very important because tutors assigned to students could not be located in the same geographical area of students. This allows to share tutoring resources using the

Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-2

eLearning platform to sure the tutor’s availability to students.

THE EVALUATION MODEL FROM THE LECTURERS’ POINT OF VIEW

The EHEA directives declare the evaluation of personal and group activities as the main feature of an EHEA e-Learning space. So, a lecturer must generate a planning model of a course based on autonomous tasks and collaborative activities between students. The first feature consists of providing an organizing tool to declare the course model, so that from this tool the student has a clear view of his/her objectives and who is going to undertake collaborative work in order to achieve a good academic performance. In UNED’s platform (aLF, active Learning Framework), this feature is providing by the planning tool. The planning tool has a user-friendly interface for thematic or weekly blocks that presents the different resources of the e-learning platform in an organized way. The first step in the course planning is the writing of a teaching guide based on activity items (assessments, tasks, forums discussion, etc.). This guide has been produced by lecturers in order to describe the overall work that students have to do in the course. To support that, an organization and planning system have been created using blocks where information (resources and activities) can be added by lecturers from many sources and ordered in any way. The blocks (organizational units) can have a topic or weekly format, and a summary is available for each block to describe its contents. For each block new and already created resources and activities can be included, so reusable activities can be defined in order to help teachers in every term (one planning, several terms). This tool [5] allows students to review the course evolution and the planning of their own available time to achieve the course’s objectives. It must be kept in mind that the typical profile of a student in the UNED, is a person who works and has limited time for learning activities. However, this is not an impediment to prevent continuous evaluation of the student, with a schedule of activities according to a course’s structure. This is possible because the programming of the course includes appropriate milestones (individual activities) to undertake the continuous assessment (with deadlines for completion and grading). In addition to individual assessment, it is possible to add collaborative activities to the course planning, based on specific working sub-groups or through a structured Wiki. The lecturer includes the evaluation tasks inside the planning page to produce a course plan in order for the students to know the course structure and evolution. So, the lecturer is the course manager and her/his responsibilities are basically: • Course content management.

• Definition of tasks for continuous evaluation • Attend to students in thematic or general forums • Preparation of final exams

Due to UNED’s structure, tutors are part of the learning process and they are the “nearest part” of this process to the students. Normally, students are helped “face to face” by tutors in the study centres, so the coordination between lectures and tutors is essential in order to have clear directives about course learning objectives. Tutors must transfer lecturers’ directives to students in the “face to face” sessions and “virtual” spaces. Additionally, associated to the evaluation model, new responsibilities for tutors have been defined: grading students (due to mandatory continuous evaluation) and “virtual” attendence for all their students. In the next sections how the e-Learning platform provides support for these new responsibilities is presented.

TUTORING GROUPS

As mention before, the tutors’ responsibilities must be fully supported by the e-Learning platform. The first is the grading capacity inside the platform, which is not really obvious, since the tutors’ role must be restricted in order to not overlap with the lecturers’ responsibilities. In reality, the tutor will have privileges to grade in the same way that lecturer do, but with some additional “filters”. Due to the large number of students in the majority of subjects, the tutor role is very important to achieve the final objective of correcting all the tasks defined by lecturers. The lecturers delegate the evaluation of tasks to the tutors and then, review the grades and accept or correct the final evaluation for students. In order to organize the work of the tutors, the platform provides some mechanisms to: • Coordinate the tutor/lecturer relation. • Automatically create the students group for tutor, so it

can be possible to have a separate space for communication between tutors and students.

• Allow access to the evaluation tool interface, only for the students assigned in the previous step.

In the next sections, a detailed explanation of platform mechanisms will be shown in order to have a more clear idea about “virtual” course complexity.

I. Tutorial coordination group

The platform structure allows sub-group creation in a general way in order to provide reduced spaces (in terms of users involved, not in tools availability). In the case of the coordination space between lecturers and tutors, one specific

Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-3

are automatically created with all tutors (permissions restricted over the tools) and lecturers (with all permissions over the space tools) involved. In this case, the coordination space (named as Tutorial subgroup) has a default tools template that provides: • Forum administration for lecturers and read/write

permissions for tutors. • A repository documents area. As shown in figure 1, this

is the file storage tool in the platform that allows the sharing of documentation between lecturers and tutors. UNED has a mandatory guide (tutor orientations) that lecturers must write and locate in this area for tutors.

FIGURE 1

FILE STORAGE FOR THE TUTORIAL COORDINATION SUBGROUP. • News. Lecturers can publish on the top page of

subgroup to communicate relevant information to tutors. • Chat rooms. Useful to resolve doubts about tutorial

practice between tutors or lecturers-tutors in real time. This sub-group provides a simple and effective way to coordinate, communicate and document the relationship between tutors and lectures. Of course, the enrolling of tutors and lectures is automated, and the subgroup is created at the start of course (really, one month before in order to temporize the coordination adequately).

II. Student attention group: Study centre sub-group.

As a requisite for tutors, not only the “face to face” sessions exist as a communication path between tutors and students, but also the “virtual” attendance is required for students in the corresponding study centre. For this purpose, a new set of sub-groups are created at the beginning of the course. These sub-groups correspond to each study centre where the student is assigned, and the enrollment for students and tutors is automatic. The sub-group has a similar structure to the tutorial coordination group, but in this case, the tutors have administrative privileges in the sub-group (as lecturers in the coordination sub-group). The sub-group, by default, has similar tools but other important (planning tool for

tutorial sessions, evaluation tool for subgroup, etc.) can be instanced and used. In figure 2 a news item is published at the top of the page in the study centre’s sub-group to provide useful information about a “face to face” session in the study centre.

FIGURE 2

A NEWS ITEM FROM A TUTOR OF A STUDY CENTRE.

GRADING TOOL

All the last spaces (sub-groups) are defined and configured “outside” main course (managed by lecturers). This means that the sub-group access is from inside the main course but that the spaces are formally separated. In the case of the grading tool, this is located inside the main course due to lecturers having to have full access to the grades provide for tutors. These two sub-groups allows tutors to do the activities related to the firsts two points referenced in the Tutoring groups section, but the last one must be provided by the eLearning platform as well. This is achieved by the use of the grading tool. The platform’s evaluation tool (grading tool) requires lecturer level privileges for its use, so initially tutors really do not have access to it. In order to provide access to tutors, a previous stage is undertaken: for each set of students enrolled in the study centre group, an “evaluation group” is created. This evaluation group is used as a data source for the evaluation tool that tests if the user (student) in the main course has a corresponding evaluation group. Using this simple approach, the tutor is granted access to the same evaluation tool as lecturers, but only with their corresponding evaluation group. In figure 3, an example of some evaluation groups is shown. These groups, obviously, are created and managed automatically with no intervention from lecturers or tutors. The name column of the table shown in figure 3 corresponds to the name of the study centre, and the member column indicates the total number of students assigned in this evaluation group. It is possible to delete the evaluation group only for administrative purposes, but normally it is not

Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-4

necessary to do so, due to the automatic process of group synchronization.

FIGURE 3

EVALUATION GROUPS FOR TUTORS ORGANISED BY STUDY CENTRE

TUTOR EVALUATION IN GRADING TOOL

Once the evaluation group is created, the tutor can start the grading process (always taking into account that tasks are date based normally). Now, the tutor only has access to their students, assigned in her/his group, in the evaluation tool, and hence, in subjects with a lot of students, this filter facilitates the evaluation process for the tutors. The interface for the grading of students is similar to that provided to lecturers and is related to a single gradable task. The interface (see figure 4) is organized by tabs, categorizing the student task state: • Task info. Provides general information about task

definition: instructions, grade, estimated hours to complete, start and finish for submission (if it is applicable).

• Graded. Indicates the total number of students graded. • Not yet graded. Indicates the total number of students

not yet graded but with submissions done. • Not yet submitted. Indicates the total number of

students that do not submit the task. • Evaluate with sheet. Allows having a sheet (CSV

format) in order to do an off-line grading.

FIGURE 4

EVALUATION TOOL INTERFACE FOR A GRADABLE TASK FOR A TUTOR

The tutor must review all the students in the “Not yet graded” tab in order to grade all the students assigned in the evaluation group mentioned before. It is also possible to

grade students with no submission, in situations where the submission is not suitable for electronic format (for example, technical drawing with special papers requirements). The process of grading is quite simple and two methods are allowed: • Simple grading: Tutors can use the grade and comment

text boxes (see Figure 4) to produce a short feedback to the student. It can be used from the own tab interface.

• Advanced grading. When special comments (HTML based or extended comments) are needed, tutors can use the advanced evaluation card located in the right side of student submission line (Advanced Evaluation link). On the advanced evaluation card (see figure 5) it is possible to use an integrated HTML editor to get more extended comments and attach a particular solution to the student (correction file). The advanced evaluation card allows tutors to navigate through their students, so it is not necessary to go back to the corresponding evaluation tab.

FIGURE 5

ADVANCED EVALUATION CARD FOR GRADING AN STUDENT Finally, when the tutors’ grading process has finished, the lecturers responsibility is to judge and validate (or not) the grades provided by the tutors.

JUDGING GRADES FROM TUTORS

In subjects with a lot of students (around 8000 students) it is very difficult to validate one by one all the grades provided by tutors. So, from a lecturer’s point of view it is important to have a set of “filters” to get localized data from students and tutors. In this case, the platform provides a similar interface for lecturers (see figure 6) but with two additional selection elements:

• Selection by study centre. This filter allows lecturers to

see the grades from students assigned in a particular study centre. The interface provides statistics data: mean; median and mode that can be used to detect

Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-5

discordant grades respect to the rest of study centres. Furthermore, the lectures can review all students from a study centre and modify or validate every grade entered by the tutors.

• Selection by tutor. This filter allows lecturers to see the

grades from students graded by a particular tutor (one study centre can have several tutors, depending on the amount of students enrolled in the subject and in the study centre). The interface provides statistics data to allow a basic detection of discordant grades from the rest of tutors (even in the same centre). Again, a lecture can review all students graded by a tutor and modify or validate every grade done by tutor

FIGURE 6

EVALUATION TOOL INTERFACE FOR A GRADABLE TASK FOR A LECTURER

The validation process does not require anything in particular, no action to be taken. This simplifies the validation process and delegates the review grade process to the students by request.

SOME RESULTS FROM FIRST 2010 SEMESTER

In order to validate the adaptation to the EHEA evaluation model, some indicators about tutoring grading (March 1, 2010) have been obtained and are presented in table 1.

TABLE 1 INDICATORS

Value Total number of grades provided by tutors Total submissions Grades without submissions Students not graded by tutors

61711 48427 13284 1371

For obvious reasons, detailed data is not included in this paper, but some interesting conclusions can be drawn: • The total number of grades provided by tutors is quite

high; taking in account the fact that this is the first year of EHEA degrees. Of course, if we count all students enrolled in every subject of the first semester of 2010: 121547 in total, then this is a logical conclusion.

• Approximately, 21% of total grades are not related to a submission in the platform. It is not a surprising result

since the majority of tutors do not have experience with learning platforms, so they decide to undertake the task in their “face to face” sessions and not to grade in electronic format. However, they have decided to grade the students inside the platform to allow lectures to have the results (not the submission) in electronic format.

• Only 2% of submissions were not undertaken by tutors. Basically, the grading of these students were made by lecturers in order to be sure that all the submissions have been graded.

CONCLUSIONS

The evaluation model required by the EHEA implies a structural change in the whole university. This adaptation to the continuous evaluation model proposed implies new responsibilities for tutors in the UNED, including the grading of electronic submissions in the e-Learning platform of UNED (called aLF). In this paper the communication and coordination mechanism between tutors and lecturers has been discusses together with the way in which subgroups for students are created to give support to the “virtual” attendance in the study centres. Furthermore, the evaluation tool interface has been described, which provides a similar interface for tutors and lecturers, and the evaluation methods provides both simple and advanced grading. This integration of interfaces allows simple training for the tool in a population of 1600 lecturers and 6900 tutors. As a final conclusion it can be concluded that the complete evaluation process has worked well with some exceptions due to the particular structure of UNED. We can conclude that it has been successful taking in account the complexity of the university’s structure.

ACKNOWLEDGMENT

The authors wish to thank the Spanish Ministry of Science and Innovation and the National Plan R&D TIN2008-06083-C01/TSI and TIN2008-06083-C03/TSI “s-Labs: Integration of open services for remote and distributed virtual laboratories, reusable and safe”. Also authors would like to acknowledge the support of the Project 142788-2008-BG-LEONARDO-LMP mPSS – mobile Performance Support for Vocational Education and Training Project and IPLECS Project – Internet-based Performance-centered Learning Environment for Curricula Support Project ERASMUS 141944-LLP-2008-1-ES-ERASMUS-ECDSP. Finally, the authors want to acknowledge the support provided by e-Madrid Project, S2009/TIC-1650, “Investigación y Desarrollo de tecnologías para el e-learning en la Comunidad de Madrid”.

Session S3J

978-1-4244-6262-9/10/$26.00 ©2010 IEEE October 27 - 30, 2010, Washington, DC 40th ASEE/IEEE Frontiers in Education Conference S3J-6

REFERENCES

[1] National Distance Learning University of Spain, http://www.uned.es

[2] T. Read, S. Ros, C. Rodrigo, R. Pastor and R. Hernández, “ The UNED ICT Architecture for “Virtual Attendance””, 23rd ICDE World Conference on Open Learning and Distance Education, Maastricht, Netherlands, 2009.

[3] R. Pastor, T. Read, S. Ros, R. Hernández and R. Hernández, “Virtual communities adapted to the EHEA in an enterprise distance e-learing based environment”. HCI-2009 International Conference on Human Computer Interaction, San Diego, California, USA, 2009.

[4] R. Pastor., S. Ros, R. Hernández., J.G. Boticario and T. Read, “Open source and e-learning in UNED”. International Open Software Conference, Badajoz, Spain (2007)

[5] R. Pastor, A. Rodriguez, V. Aguilar and R. Hernández, “Blocks organizer for .LRN”. 7TH Openacs / .LRN Conference, Valencia, 2008.

AUTHOR INFORMATION

Rafael Pastor, senior lecturer at the Spanish University for Distance Education. IEEE Member, rpastor@scc.uned.es Roberto Hernández, senior lecturer at the Spanish University for Distance Education. IEEE Senior Member. roberto@scc.uned.es Salvador Ros, senior lecturer at the Spanish University for Distance Education. IEEE Senior Member. sros@scc.uned.es Timothy Read, senior lecturer at the Spanish University for Distance Education, tread@lsi.uned.es. Manuel Castro, Professor at the Spanish University for Distance Education, IEEE Senior Member. mcastro@ieec.uned.es. Rocael Hernández, General Manager GES, Galileo University, roc@galileo.edu.