Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-1

A Video-Message Evaluation Tool Integrated in the

UNED E-learning Platform

Rafael Pastor, Roberto Hernández, Salvador Ros, Antonio Robles-Gómez, Agustín Caminero, Manuel Castro, and

Rocael Hernández Spanish University for Distance Education: rpastor@scc.uned.es, roberto@scc.uned.es, sros@scc.uned.es,

arobles@scc.uned.es, accamineros@scc.uned.es, mcastro@ieec.uned.es

Galileo University: roc@galileo.edu

Abstract – Use of multimedia services has become very

important in learning process. This is due to the

traditional approach of document based assignments not

provide any feature which allows to evaluate some basic

competences (oral communications skill, for example).

In this paper, the UNED (Spanish University for

Distance Education) solution will be presented focusing

on user’s interaction: lecturers and students. In the first

case, the lecturer has to define a task and propose a

solution (available only at the end of task availability).

This solution is also video/audio based, so the student

can learn about it and compare it with his/her solution.

The student has several attempts, in order to try better

responses and decides which of these attempts will be

selected as final response. The whole interaction schema

will be presented and how is integrated in the e-learning

platform as a usual task (like document based tasks).

This integration permits the use of all features of the

platform evaluation tool, simplifying the evaluation

process and tasks grading (no special procedures are

required).

Index Terms – Video assignments, e-learning, evaluation

process, service integration.

INTRODUCCTION

The oral and communications competences and skills are

features not so much supported in the lecturer planning of

subjects and its evaluation. This is due to, normally, e-

learning platforms have several tools to include recordings

of video/audio but not integrated with assignment or tasks.

This issue makes difficult the inclusion of multimedia based

activities, basically because the relation between tools and

evaluation is not clear and also, non ICT management

procedures must be used in order to add the grading to

evaluation tools of platforms (some much extra work for

lecturers, depending of student number).

Nowadays, Internet access is an usual facility that

students has incorporated in their learning process, basically

by means of virtual learning environments, social networks

(Face Book, Twitter, Linked it, etc.) , multimedia services

(like YouTube) or video conferences systems. So it is

expected that e-learning platforms will be capable not only

of incorporate these technologies in a basic way (link based)

but adding it as evaluation tools transparently to lecturers

and students.

UNED [1] has its own conferencing software: AVIP

(Audio-Video over Internet Protocol) [2]. This tool was

developed to permit a wide range of online video-

conferencing techniques to be combined with smart-board

equipped classes and digital repositories to provide

interactive taught classes to a large number of students

(those actually in the class, those connecting online, and

those who access recordings of the sessions).

UNED CONFERENCING TOOL

As Internet access has improved, it has become easier to

access audio and video recordings online While this is an

effective way to expand our knowledge, albeit somewhat

“passively”, it does nothing for the development of

production and interaction competences and skills, and the

communicative functions involved (defence, argumentation,

persuasion, reasoning, justification, etc.) in real-time and

real-world scenarios. In order to fulfill these specifications,

the AVIP tool provides several services available to UNED

users (lecturers, tutors, students, etc.):

AVIP Classrooms. Physical classroom with

incorporated videoconferencing (MCU based and IP

based) systems and digital boards. These classrooms

provide full service (board, chat, video, audio,

document presentation, desktop sharing, etc.) for tutors,

connecting several AVIP classrooms in order to get

“virtual classroom” between students connected over

Internet or located in the same/another AVIP

Classroom.

Campus Broadcast. Full service for real time

broadcasting of UNED events (seminars, tutors activity,

etc.), including the storage facilities and Internet access

(IP classrooms).

E-learning Platform Integration Service. The UNED

e-learning platform aLF/.LRN [3], [4], provides a full

integration with AVIP. This integration allows to

courses administrator managing of “virtual classes”

based on AVIP classrooms (access control for users,

date, time assigned, password protected, etc.). The

interface for a virtual classroom is shown in Figure 1.

Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-2

The basic features of AVIP Classrooms (IP based) are

similar to other commercial tools like Elluminate [5], and

Adobe Connect [6] or open source as Kaltura [7] or

BigBlueButton [8]: real time video/audio, whiteboard, chat,

desktop sharing, etc. These features are useful in order to

provide virtual classes or virtual training [9], replacing the

traditional concept of class. But really, these tools only

provides a type of activity to be used as collaborative

systems in real time, with lecturers/administrators doing

teleconferences or moderating the audio/video access.

FIGURE 1

AVIP CLASROOM

Really, the uses of these tools are focusing in video/audio

recording of sessions but not other transparent integration is

provided (as it is shown in this paper). For example,

Elluminate’s Moodle integration provides the concept of

live sessions [10], and also it is possible to get live/recorded

sessions, integrated as course content objects inside Moodle.

But there is no a straight way to include this content objects

as response for assignments, only using hyperlinks as

responses. Of course, the lecturers must define a “forever

live session” in order to let student record responses and

“remember” the hyperlink to include as a response for

assignment. This procedure have several problems

associated: shared use of session (students cant not use the

same live session at the same time), complex managed

procedure for lecturers (defining session, instruct students

about tool use and adding response, and moderating and

coordination of session use) and technological competence

of students (not all students have skills to manage

hyperlinks in a simple way).

In the case of Adobe Connect [11], Kaltura and

BigBlueButton, the e-learning platforms integration

modules are similar (activity/session based) and they

provides simple integration with platforms as Moodle, Sakai

and Blackboard (and many others). Kaltura have

development plans to add video extensions: add/edit video,

assignment type integration at course level and addition of

grading in video assignment uploads, but for now, it is only

an announce [12].

No one of tools mentioned before allows a full integration

with other specific tool like evaluation/grading of video

based assignment, very useful to get asynchronous video

recordings associated to course activities. The basic

difference between the last tools and AVIP is the

evaluation/grading tool integration, so for aLF users

becomes transparent the using of video responses for

assignments (in the same way as usual documents

responses) and students/lecturers have not to manage

copy&paste operations of recoding URLs in order to

integrate video/audio recordings. It is very important not to

force to lecturers to learn new tools outside of virtual

courses, so the impact of using could be negative. In this

case, the integration is transparent and the evaluation

process uses the same mechanics that traditional

assignment, so the learning curve is very short.

AVIP VIRTUAL CLASSROOM.

Based on AVIP and to face the problem mention before,

new features were added (AVIP-PRO) to enable video

recordings to be made online without having to be in a

moderated AVIP session (external tool) and is integrated in

the UNED e-Learning platform. It has been designed as a

didactic tool for multiple functions, namely the training of

specialized oral discourse (of particular relevance in

disciplines like Law and Business), real-time oral evaluation

(fairly common in face-to-face institutions with a reasonable

student-teacher ratio) and training and evaluating second

language oral competences online. Language learning tasks

can be prepared where students undertake any kind of online

oral communication and training, combining the flexibility

of the AVIP framework together with the task-based

structure of aLF. Institutionally, the tool is scalable and can

be managed from within aLF as a standard task type

enabling large-scale training and evaluation to be

undertaken. In next sections, basic properties of task-based

structure will be presented focusing in the video task

features, showing the lecturers/students point of view and

how responses are managed in the same way as it is done

with “normal” responses (document/hyperlink based).

DEFINING VIDEO TASKS FEATURES

The assignments tool of aLF/.LRN allows delivering some

common formats (document based) but in this case, a

specific type of assignment has been included to support

multimedia integration in deliverables. An assignment in

aLF has several general properties, but the more important

are:

Task Name. It defines the task name that will appear in

the assignment dashboard and the grading dashboard.

How will the task be answered?. Type of format

supported by assignment. There are four types of

formats: Submit online (document based), Associate

with Forums (to grade forums participation in courses),

No submission (to support not deliverables tasks) and

Multimedia recording (AVIP-Pro).

Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-3

Time assigned. Only applicable for AVIP-PRO tasks.

It defines the maximum recording time for

lecturers/students solutions

Description. HTML formatted text to describe the

objectives of assignment. It can be complemented with

one attach (document) to include other relevant

information.

Start and due date. Availability times for assignment

Perfect score. Maximum grade for assignment.

Attempts number. Maximum number of recordings

for lecturers/students.

The defining process of a video assignment is the same as

the rest of supported formats, so, in this way, is fully

integrated with the assignment/grading (evaluation) tool in a

transparent way. In the case of a multimedia task, the two

more important properties that must be defined are Time

assigned and Attempts number. These two parameters

must be clearly set before start date of assignment, so the

students have a clear knowledgement of time required and

attempts available. In fact, these parameters cannot be

changed after task start date.

Once the assignment is created, it will be available in the

assignment dashboard (portlet of assignment tool). In Figure

2, part of an assignment dashboard it is presented. These

assignments are included in a subject of a master degree,

developed by the Communications and Control Systems

Department of UNED. In this case, the video task is

oriented to be complementary to the final practice in the

subject (Web standards design). The main objective is to do

a brief presentation (ten minutes) in which the students have

to explain clearly: development methods, standards used,

usability issues and professional usage (if it is applicable).

As it is shown in Figure 2, a camera icon indicates the video

assignment, in order to clearly distinguishing it of the rest of

assignments. For every assignment, in the corresponding

dashboard, it s possible to do some basic operations:

(View assignment properties), (Edit assignment

properties) and (Delete assignment). Finally, a global

option called Upload solution can be used to add lecturer’s

solutions to tasks. In case of deliverable document based

tasks, a solution document can be included to provide

lecturer feedback to students.

RECORDING TASK SOLUTIONS: LECTURER POINT OF

VIEW

In order to provide an assignment solution, lecturers have

the option of recording a video scene using their own

camera (usually, it is incorporated to the same

laptop/computer used by lecturer). To do this, there is an

option in the assignments dashboard called

that will show the recording dashboard for lecturers, as

shown in Figure 3.

FIGURE 2

TASKS LIST, VIDEO TASK SELECTED.

FIGURE 3

SOLUTIONS DASHBOARD WITH RECORDINGS

The solutions dashboard shows all the recordings have been

made by lecturers. There are two main actions associated, in

order to work with the recordings dashboard:

. This button allows starting a solution’s

recording, presenting the record window (see Figure

4).

. When the solution board has one or

more recording sessions, it is possible for lecturers to

select the assignment solution. This solution will be

visible for students at the end of task availability (due

date), so students can compare their solutions with the

lecturer proposal.

For all recordings, there are two sub-actions to manage each

recording: View recording in the recorded sessions player

( icon) and delete recording ( icon).

The recording window (see Figure 4) has all the information

needed to work with the solution and it has two panels:

A) Information panel (left and bottom space of recording

window). It provides the indications about video

recording (time assigned and attempts available), task

name and description (HTML based and document

attached to the task)

B) Recording panel (upper right space of recording

windows). It provides the video/audio feedback of

recording and options available (recording modes and

recording publish options).

VIDEO TASK RECORDING WINDOW.

From this window, the lecturer can record a solution simply,

using the record button located in the recording panel, see

Figure 5 (a). Once the record button is clicked, a new

window, see Figure 5 (b), it is shown in order to write the

name of solution (name used in the recording dashboards)

and also to select the type of recording (video/audio, only

audio). See Table 1 for icons meaning. Once the name of

Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-4

solution is selected, the lecturer only has to select the type

clicking the corresponding icon and the recording begins. In

order to stop the recording, the record button (a square while

it is recording) must be clicked and the solution recording is

added to the lecturer’s recordings dashboard.

FIGURE 4

RECORD WINDOW WITH STUDY MATERIAL

(a) (b)

FIGURE 5

SOLUTIONS DASHBOARD WITH NO RECORDINGS

In Table I, another icon has been included in order to

manage the public settings associated with recordings.

These recordings can be produced using AVIP-PRO or

AVIP conferencing tool integrated in aLF. In Figure 6, a full

list of recordings (some omitted in order to clarify options)

is presented (when the lecturer clicks the third icon

presented in Table I). This list contains all the recordings

(not only the recordings associated with video task) in order

to manage them. There are several options: public

publishing (to all the users of UNED space), edit recording

or delete recording, and provides a full dashboard to manage

all the recording information provide for every user

(lecturers, students, tutor, etc.) of AVIP system. TABLE I

RECORDING MODES AND OPTIONS

Icon Mode/Option

Provides a full recording supporting audio and video

modes defined in the preferences/options panel

Provides only audio recording

Hiperlink to recordings sessions management site

Finally, the lecturer can change basic parameters or

recording using the tools panel located in the recording

panel (left of image panel):

. Switch audio off/on.

. Freeze/Resume video (only in full recording

mode).

. Video/Audio options. They are very useful to get

a good user experience because it is possible to change

size of image, frames per seconds and image quality.

Also it is possible to modify the band width to adapt the

net connection to the current internet profile. In Figure

7 it is shown the options panel.

FIGURE 6

AVIP RECORDINGS DASHBOARDS

FIGURE 7

CONFIGURABLE RECORDING SETTINGS.

When the lecturer has finished a recording, he/she can

preview the video/audio file using the integrated player

provided (see Figure 8). For every recording, at the start of

previewing, basic information is presented to lecturer:

assignment associated, course in which the assignment is

defined, author, name of recording and start date. This

information is very relevant for lecturers in order to grade

students, as it will be seen later.

Finally, the lecturer only has to select the correct solution in

the recordings dashboards and click the Assign Solution

button.

RECORDING TASK SOLUTIONS: STUDENT POINT OF VIEW

The student actions to provide answers (solutions) to the

video assignment are very similar to the lecturer’s process.

In fact, the recording window and actions are the same, so

the previous section describes all the features which a

student has in his/her recording sessions.

Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-5

FIGURE 8

VIDEO/AUDIO SOLUTION.

In the case of a student, the assignment dashboard is lightly

different that lecturer’s one but it is very similar. All the

administrative actions are not available for a student

(logically) but the Submit answer option is located exactly

in the same place that the “Upload solution” option provided

for a lecturer. In Figure 9, the video assignment of the

master degree’s subject is shown.

FIGURE 9

ASSIGMENTS DASHBOARD FOR A STUDENT

The student has only to click the submit answer option to

access to his/her recordings dashboard (very similar to

lecturer’s one). In this case, the options available for student

are Record (the same features which are provide to a

lecturer) and “Assign record to task” to select the

corresponding recording as student solution to the

assignment. In Figure 10 (a), a selected recording is

assigned as answer. Also, once an answer is selected, a

student can change the answer election or view current

answer, as it is presented in Figure 10 (b). These actions are

only possible before the due date.

EVALUATION OF STUDENTS

The final stage that a lecturer will have to do it is the

grading of answer’s assignment. The grading process is the

same as the rest of types of tasks, so the same tool is used to

evaluate the performance of students. The answer is located

in the Submission column in Figure 11, and clicking the

View submission shows the recordings player with the

student’s video (similar to Figure 8, depending of recording

type: video/audio or only audio). Once the students answer

is played, lecturers must assign a grade to the submission

and the process of evaluation will finish.

(a)

(b)

FIGURE 10

MANAGE OF ANSWERS OF A VIDEO ASSIGMENT

FIGURE 11

EVALUATION OF STUDENTS ANSWERS FOR A VIDEO ASSIGMENT.

Another role which participates in the evaluation process is

tutor. The tutor participation [13] is very important in

UNED degree/master degree subjects because the number of

students (some subjects with near of 14000 students). Tutors

can evaluate to her/his students assigned (usually, by

geographical aggregations) using the same tool used by

lecturers. The only differences between roles corresponds to

the number of students in the different list/tabs in the

grading tool (graded, not yet graded, not yet submitted) due

to, usually, a tutor has assigned a student’s group of 50

students (maximum).

CONCLUSIONS

Web meetings tools provide useful features in order to have

virtual training or virtual classes. But the LMS integration

modules available, like in Elluminate, Connect, Kaltura or

BigBlueButton, only have features about manage sessions

and do recordings by lecturers and students. This

management procedure is complex for lecturers, especially

in courses with a big number of students (UNED has a

medium number of students per subject about 2000), forcing

lecturers to do extra work in order to coordinate the

evaluation/grading of task. Also, from the student point of

view, they are force to manage recording URLs, in order to

get their responses as video messages, and including it in

the grading tool of LMS. This can be a big problem with a

big number of students, like UNED.

To solve this issues, AVIP PRO provides a simple full

integration with evaluation/grading tool of UNED learning

platform (not only for recording sessions in available

rooms), allowing lectures manage video assignment in the

same way as document based assignment. The task

properties are the same and only two new properties are

applicable to these types of assignments (recording

Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD

41st ASEE/IEEE Frontiers in Education Conference

F3C-6

maximum time, attempts available). This transparent

integration facilitates the use of this tool, avoiding the

problems mentioned before.

In the paper, the grading/evaluation process has been

presented to demonstrate: 1) Task management process is

the same as document based task: defining of task and

delivering of responses by students; 2) There is no negative

impact on lecturer traditional works. This is due to the

evaluation process is not modified and only the specific

recording interface details will have to be learned.

Using multimedia tools provide a new way of

evaluation/grading in order to have a good approach to the

face to face exams (oral presentations, very useful on

language learning). Some competences which are not very

easy to take in account in document’s assignment can be

evaluated using AVIP PRO. In this case, oral competences

are applied in the context of final projects in a master degree

subject, so it can be evaluated competences like project

understanding, synthesis capacity or oral presentation skill.

ACKNOWLEDGMENT

The authors wish to thank the Spanish Ministry of Science

and Innovation and the National Plan R&D TIN2008-

06083-C01/TSI and TIN2008-06083-C03/TSI “s-Labs:

Integration of open services for remote and distributed

virtual laboratories, reusable and safe”.

Also authors would like to acknowledge the support of the

Project 142788-2008-BG-LEONARDO-LMP mPSS –

mobile Performance Support for Vocational Education and

Training Project and IPLECS Project – Internet-based

Performance-centered Learning Environment for Curricula

Support Project ERASMUS 141944-LLP-2008-1-ES-

ERASMUS-ECDSP.

Finally, the authors want to acknowledge the support

provided by e-Madrid Project, S2009/TIC-1650,

“Investigación y Desarrollo de tecnologías para el e-learning

en la Comunidad de Madrid”.

REFERENCES

[1] National Distance Learning University of Spain,

http://www.uned.es

[2] C. Rodrigo, J. L. Delgado, J. Vega, "Using Interactive

Videoconference to Promote Active Learning in a

Blended Learning Environment," ICALT, pp.658-662,

2010 10th IEEE International Conference on Advanced

Learning Technologies, 2010.

[3] R. Pastor, R. Hernández, S. Ros, T. Read, M. Castro,

and Rocael Hernández, "An enterprise e-learning

solution: The UNED practical case in the EHEA".

EDUCON 2010, IEEE Engineering Education 2010 –

The Future of Global Learning in Engineering

Education, Madrid, Spain, 2010

[4] R. Pastor, T. Read, S. Ros, R. Hernández and R.

Hernández, “Virtual communities adapted to the EHEA

in an enterprise distance e-learing based environment”.

HCI-2009 International Conference on Human

Computer Interaction, San Diego, California, USA,

2009.

[5] Elluminate, http://www.elluminate.com/

[6] Adobe Connect, http://www.adobe.com/products/adobeconnect.html

[7] Kaltura, http://www.kaltura.org/

[8] BigBlueButton, http://www.bigbluebutton.org/

[9] B. Fallery, S. Gerbaix and R. Ologeanu, 'Videotraining:

A comparison between “virtual class” and “remote

class”'. CTS 2008, International Symposium on

Collaborative Technologies and Systems, Irvine,

Canada, 2008.

[10] Ellumnite Integration Module:

http://www.elluminate.com/Products/Elluminate_Learn

ing_Suite/Elluminate_Bridges/Moodle_/?id=178

[11] Adobe Documentation about Connect Integration in

LMS:

http://www.adobe.com/education/elearning/pdfs/connec

t_lms_intgrtn_sb_ue.pdf

[12] Kaltura development plan announce:

http://www.remote-learner.net/kaltura_story

[13] R. Pastor, R. Hernández, S. Ros, T. Read, M. Castro,

and Rocael Hernández, "A Complex Tutoring System

for e-Learning: The New Evaluation Model", FIE 2010,

Frontiers in Education Annual Conference, Washintong

D.C., USA, 2010

AUTHOR INFORMATION

Rafael Pastor, senior lecturer at the Spanish University for

Distance Education. IEEE Member, rpastor@scc.uned.es

Roberto Hernández, senior lecturer at the Spanish

University for Distance Education. IEEE Senior Member.

roberto@scc.uned.es

Salvador Ros, senior lecturer at the Spanish University for

Distance Education. IEEE Senior Member.

sros@scc.uned.es

Antonio Robles-Gómez, Assistant Lecturer at the Spanish

University for Distance Education. arobles@scc.uned.es.

Agustín Caminero, Assistant Lecturer at the Spanish

University for Distance Education,

accamineros@scc.uned.es.

Manuel Castro, Professor at the Spanish University for

Distance Education, IEEE Senior Member.

mcastro@ieec.uned.es.

Rocael Hernández, General Manager GES, Galileo

University, roc@galileo.edu.