1

Easy, economic, expedient - An effective training evaluation model for SMEs Sajid Khan, Phil L. Ramsey School of Management, Massey University, Palmerston North, New Zealand Skhan2@massey.ac.nz, p.l.ramsey@massey.ac.nz

ABSTRACT Firm overall performance highly depends on its human capital, a pre-eminent part of the Intellectual Capital (IC). This potential therefore needs to be maintained and improved by various development strategies. Training in this regard is esteemed to be the most effective measurement to reach this aim. However, the effectiveness of training highly depends on training design and its evaluation. Especially SMEs hesitate to invest in training due to financial restrictions and concerns of training benefits. Therefore there is a need for a training evaluation model which not only is simple and affordable but also appreciates the fact that training is a complex web of relationships between trainer, trainee, content, work environment and organisational goals. To formulate such a model a three-step process was adopted which facilitated filtering out the appropriate assessments. The resulting evaluation model is capable of taking into account factors lying inside as well as outside the training realm while still playing a crucial role in the effectiveness of a training program. Thus, besides assessing the effectiveness of training it even works as a navigation tool for designing future training programs. Finally, the proposed training evaluation model was applied on a training program in a small-scale service oriented organisation and the analysis of the results showed that the model successfully attained the anticipated aims. Key words: Human Capital development, Training evaluation model, SME training, Training efficiency Introduction IC shows manifold facets depending on the viewing angle. IC can be seen as collective knowledge (Kong & Thomson, 2009), intellectual material (Stewart, 1997), subset of intangible assets (Hunter, Webster & Wyatt, 2005), knowledge based resources (Ordonez de Pablos, 2005) and many other nuances are expressed in the literature (Hamza & Isa, 2010; Sujan & Abeysekera, 2007). IC is predominantly perceived as a tripartite framework (Stewart, 1997; Yaghoubi, et al., 2010). This perspective views IC as a composition of three primary interrelated components - internal capital, external capital and human capital (Johanson, 2005; Ordonez de Pablos, 2003). Whereby internal capital refers to all the intelligence embodied in the structure and processes of a firm (Petty & Cuganesan, 2005) similarly, external capital encompasses all the relationships - market relationships, power relationships and cooperation – a firm has with outside stakeholders (Petty & Cugansen, 2005). Lastly, human capital refers to knowledge, skill, capabilities, experience, attitude, commitment and personal characteristics of a firm’s employees (Yang & Lin, 2009). Hence, IC refers to a firm’s repository of all the intelligence embedded in its organisational structure, network relationships and personnel (Choo & Bontis, 2002). Since the components of IC act and interact facilitating a firm’s decisive competitive advantage they all are considered vital for the survival and success for modern day firms (Bontis, Crossan & Hulland, 2002; Ordonez de Pablos; 2003; Hamza & Isa, 2010). However, it is worth pointing out that human capital is viewed rather more significant than its counterparts (Kong & Thomson, 2009). It is because human capital not only is generatively intangible (Ahonen, 2000) but rare and hard-to-imitate (Stiles & Kulvisaechana, 2003) and has relatively more potential than other constituents of IC in terms of influencing its counterparts (Huang & Hsueh, 2007). The attainment of adequate human capital is therefore purported being central to firm success in the existing global economy (Roslender & Fincham, 2001). It is then of utmost importance that human capital attains its potential through organisational development strategies that stimulate and improve the skills, knowledge and learning capacities of the individuals that make up an organisation (Pfeffer, 1998). Training in respect of achieving this goal is considered the most important agent since it facilitates firms enhancing their human capital value (Hamza & Isa, 2010). Modern day firms therefore funnel a significant amount of their budgets into training to upgrade the skills and knowledge level of their workforce which eventually translate into the improvement of firms’ bottom-line performance (OECD, 2002; Bassi & McMurrer, 2004). Despite the fact that training is deemed essential for all types of organisations, yet small and medium-sized enterprises (SMEs) unlike large businesses invest quite meagrely in it (Westhead & Storey, 1997). Some of the major reasons for this phenomenon are SMEs’ limited financial abilities (OECD, 2002), unawareness of owners regarding the benefits of training and owners' concern that costs incurred on training may surpass its returns (Huang & Hsueh, 2007).

2

Two of these three above mentioned reasons – owner ignorance of training benefits as well as their underestimation – are to a great extent related to the effectiveness of trainings within the context of SMEs. Although there is a huge body of literature discussing a multitude of training design and evaluation models, very limited attention is given to the effectiveness of training in the context of SMEs (Huang & Hsueh, 2007). Consequently, most of the training evaluation models discussed in the literature are complex, labour intensive and time consuming (Stevenson, 2012). All this necessitates the need for a simple, less costly and workable training evaluation model for SMEs. Not only should the model be able to assess the effectiveness of an existing training program but also provide a roadmap for designing future training programs. The formulation of the proposed evaluation model There is an ample literature describing and proposing various models for the evaluation of a training program (Wang & Wilcox, 2006). However, it is important to note that every evaluation model has its own assumptions, purpose, rational for inclusion and exclusion of variables and intended audience which not only differentiate it from others but confine its effectiveness to certain specific contexts (Mathison, 1994, p. 469). Wang and Wilcox (2006) for that reason heralded the identification of the right evaluation approach which can effectively correspond to the specific needs of the situation to be the core challenge for those theorists and HRD practitioners who plan to evaluate the training program. This paper aims to propose a training evaluation model for SMEs with this in mind. Apparently, the main objective for firms to undertake trainings is to bring positive change in the behaviour by means of improving knowledge and skill levels of their employees (Kirkpatrick, 1998). The learning of skills and knowledge, however, largely depends upon the conduciveness of a learning environment (Tam, 2000). Alvarez et al. (2004), Kraiger (2002) and Kirkpatric (1998) hold that trainees’ satisfaction with the training program is not only positively correlated with their learning of skills and knowledge but also with their behaviour at workplace. Keeping forgoing in view, the basic premises set out for the development of the proposed evaluation model were to have a simple and less resource-intensive framework that could evaluate training in its existing shape. The model should be able to effectively assess the actual and desired accomplishments of a training program in terms of trainees’ satisfaction with the training, learning improvements and change in trainees’ work behaviours. Furthermore the model should also be able to determine whether the parameters, trainees, facilitators, processes and the concerned management followed the right course of action in facilitating the training to achieve its anticipated outcomes. However, if the training deviated at some points then the model should not only detect those gaps but also provide guidance for rectifying the identified issues. To formulate such a model the following three-step process was adopted.

Step 1: Selecting the most suitable evaluation approach from various available approaches - formative, summative, confirmative and meta-evaluation (Fraser, 2008).

Step 2: Identifying a range of potential assessments for the proposed model after assessing the appropriateness of assessments proposed in some of the popular training evaluation models

Step 3: Comparing the shortlisted assessments to select the most suitable ones and aggregating selected assessments into a single implementable training evaluation model.

3

The forgoing brief descriptions of the four broad evaluation approaches disclose that three of them - formative, confirmative and meta-evaluation are incongruent with the main objectives of the model. Formative evaluation as well as meta-evaluation are quite complex in nature and also require a considerable amount of resources for their applications. Moreover, formative evaluation demands interference with the implementation of the training program and meta-evaluation requires previous evaluation studies for its use. As far as confirmative evaluation is concerned, although it is relatively simple and neither does require meddling with the intervention nor does it need previous evaluation studies for its execution but it does also require substantial resources in terms of labour and time. Thus, all the three evaluation approaches fail to be used as yardsticks to critically evaluate the appropriateness of individual assessment levels of various evaluation models. Summative evaluations unlike its counterparts perfectly match the requirements of this study. Thus, it is used as a basic standard along with the other presumed aims of the model to assess the suitability of each assessment proposed in various selected training evaluation models. In this step some of the popular and effective training evaluation models are randomly selected. Each stage of the selected model is assessed for its evaluation approach and intended assessment objectives. Those assessments are shortlisted for the formulation of the model that are using the relevant evaluation approach – the summative approach - and whose objectives match any of the objectives set for the desired model.

Measuring the trainees' perceived effectiveness of training The “Reaction” level/stage/phase proposed in Kirkpatrick’s model (1967), Philip’s model (1996), Kaufman’s model (1995), Alvarez et al.’s IMTEE model (2004) and CIRO model (1970) seek to measure trainees’ satisfaction with the training programme. These assessments were not only similar in their evaluation approach but also in terms of their methodologies and assessment objectives. As they all correspond perfectly

4

to one of the objectives - to effectively assess the satisfaction level of the trainees with the training - they are set out for the proposed model. Hence they all were shortlisted for the proposed model. Although these assessments belong to different evaluation models and have different names still they are basically not different from each other. Consequently, the need to compare the shortlisted assessments to identify the best assessment rendered itself unnecessary. The assessment was selected as the level 1 of the proposed model. This level of the proposed model seeks to assess the perception of trainees about the three main dimensions of a training program - training delivery, training content and training environment. Training delivery here refers to the process employed by a trainer to deliver training contents to trainees (Lather, 2009) whereas training content implies the knowledge or intellectual property of the training program (Kirkpatrick, 1998). Lastly, training environment means the physical environment in which training takes place (Busari, Verhagen & Muskiet, 2008). Measuring the learning improvement To assess the improvement in learning the “Learning” level/stage of Kirkpatrick’s model (1967), Philip’s model (1996), Kaufman’s model (1995) and Brinkerhoff’s model (2003) are shortlisted. Like with the reaction assessment, all the shortlisted assessments were the same in terms of their approach, objectives and methodologies and could adequately serve one of the other main objectives - measuring the actual state of benefits in terms of skill and knowledge - of the proposed model. Consequently, the level 2 of the proposed model intends to evaluate the improvement in the two major dimensions of learning - knowledge and skills. Measuring the improvements in behaviour Literature identifies two main conditions for the change in trainees’ behaviour to adequately manifest itself (Kirkpatrick, 2006). First, the trainees must be able to utilise their newly learned skill and knowledge. Second, the work environment must be conducive for the application of newly learned behaviours. As behaviour assessment takes place after the learning assessment, understandably, if the learning evaluation indicates some improvement while behaviour assessment fails to show the same level of improvement then it may be because the climate of the organisation was unsupportive of the new behaviours. Consequently, the cause/s of this dilemma needs to be searched in the climate of the organisation rather than in the training environment. For the proposed model the “Behaviour” level of Kirkpatrick’s model (1967), Philip’s model (1996) and Kaufman’s model (1995), the “Usage & Endurance of Learning Stage” of Brinkerhoff’s model (2003) and Behaviour Gap Model (2001) as a whole were shortlisted. However, contrary to the first two assessments, there was a huge variation between the objectives and methodologies of these shortlisted assessments. To select the most appropriate assessment for the proposed model, the shortlisted assessments were divided into two groups. The first group was comprised of the assessments proposed to assess trainees’ behaviours in Kirkpatrick’s model (1967), Philip’s model (1996) and Kaufman’s model (1995). These assessments do not take the climate of an organisation into consideration while assessing trainees’ behaviours. Consequently, they neither can comprehensively assess the change in trainees’ work behaviours nor can they identify the factors that lie outside the training realm and can harm the effectiveness of a training program.

The remaining shortlisted assessments take rather a broader perspective while assessing trainees’ behaviour. Brinkerhoff’s model (2003) in its “Usage & endurance of learning stage” takes into consideration all the important events that take place from training space to actual work environment to identify when, where, how well, and how often which trainees use which part of their learning (p. 130-133). This process enables training evaluators to measure how and to which extent the effect of training lasted in terms of improving behaviours. Similarly, the Behaviour Gap Model (2008) views the actual work environment to be made up of a myriad different variables most of which lie beyond the realms of training but can strongly affect workplace behaviours of a trainee (Ramsey, Macky & Mason, 2008). This model proposes a two-step process for trainees’ behaviour assessment. In the first step information is collected to identify behaviour at four different dimensions - organisation-desired behaviours, trainer-targeted behaviours, trainee-intended behaviours and trainee workplace behaviours. In the second step a series of gaps are identified by comparing 1. Organisation-desired behaviours to trainee workplace behaviours, 2. Organisation-desired behaviours to trainer-targeted behaviours, 3. Trainer-targeted behaviour to trainee-intended behaviours, and 4. Trainee-intended behaviours to trainee workplace behaviours (Ramsey, Macky & Mason 2008). This assessment not only provides enough information about the actual improvement in the workplace behaviour of trainees but also ascertains the causes of deviation of the training from achieving the required behavioural objectives. Moreover, it also assists in developing effective behavioural objectives for future training programs.

5

The forgoing description makes it quite clear that the Behaviour Gap Model (2008) can address the objectives of the desired model exceedingly far better than “Usage & endurance of learning stage” of Brinkerhoff’s model (2003). Consequently, it becomes level 3 of the proposed model.

The application of the proposed training evaluation model The research was conducted in one of the financial advisory service organisations based in Palmerston North, New Zealand. The organisation - ANL (Advisory Network Limited) provides professional advice in various types of financial matters such as insurances, long-term and short-term financial investments and retirement plans. ANL conducted orientation training for its newly hired staff. To evaluate the effectiveness of the training program, the ex post facto study was undertaken which began in August, 2010 and ended in December of the same year. The span of 4 months provided the researchers with the opportunity to not only measure the trainees’ perception and learning but their behaviour after having been working for more than 3 months in the actual work environment. The sample of this study was comprised of 12 research participants out of whom 8 were ANL staff members who received the training. The remaining participants included 2 trainers who designed and delivered the training and 2 members from the management of the company. All three level assessments of the proposed evaluation model were undertaken after the completion of the training program. For the first two levels - reaction and learning levels - a questionnaire was employed as sole investigation instrument. For the behavioural level, the questionnaire was complemented by interview, observation and relevant literature review. To develop valid and reliable research instruments for this research, all major dimensions of reaction, learning and behaviour levels proposed in the model were taken into consideration. Instruments development Literature pertaining to HRD considers the use of questionnaires as sole research instrument as an adequate tool for assessing trainees perceived effectiveness of a training program (Kraiger, 2002; Alvarez et al., 2004). According to Gliem and Gliem (2003), a questionnaire with Likert scale is relatively more reliable than single-scale or ordered-category items. Keeping the above-mentioned in view, a well-structured questionnaire with a Likert scale was developed. The Likert scale of the instrument offered the responses - excellent to poor, high to low and, strongly agree to strongly disagree. The internal consistency of the instrument was measured through Cronbach’s alpha whose value arrived at 0.8343 which rendered the questionnaire to be a highly reliable research instrument. The instrument was composed of 13 items. The first three questions intended to extract the respondents’ perception about the usefulness of the training in terms of its delivery mechanism, course contents and training environment. These questions sought to obtain a quantitative reply as well as a subjective answer in the form of comments. The subjective responses were solicited to obtain data for devising recommendations. Question 4 to 9 of the questionnaire sought to measure the enhancement in the state of knowledge of the trainees. The remaining last five questions intended to find out the tangible improvement in the skill levels of the trainees. For behaviour level assessment various data extraction methods – literature review, interviews, survey and observation – were employed to answer the following four questions.

1. What behaviours put trainers emphasis on? 2. What behaviours do trainees plan to adopt for application at the workplace?

6

3. What actual behaviours are rendered by the trainees at the workplace? 4. What are the organisation-desired behavioural objectives?

Once these behaviours were identified, a simple subjective evaluation was undertaken to identify the series of

gaps between the aimed objectives and the actual performance resulting from the training.

Data collection 1) For reaction and learning assessments A total of 8 questionnaires were distributed and all of them were returned making a response rate of 100%. Questions 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12 and 13, are positively worded, whereas, 9 and 14 are negatively worded. The responses received on negatively worded question were reversed. There was no blank response so the data acquired for these questions was easily tabulated and analysed through Statistical Analysis System (SAS) for descriptive statistical analysis. The subjective responses of the questions 1, 2 and 3 helped the researchers in devising recommendations for improving the upcoming orientation programs. 2) For behaviour assessment

a) Establishing ANL desired behaviours

To establish desired behavioural objectives for ANL, the literature published by Pelham and Kravitz (2008),

Lambert, Ohai and Kerkhoff (2009) and Sujan, Weitz, and Kumar (1994) was thoroughly reviewed to understand the prevailing standard sales practices. Then, ANL’s mission, vision, policies and needs were keenly understood through reviewing relevant documents and interviewing the GM (General Manager) and the MO (Manager Operations) of the firm. The aforementioned assisted the researcher to identify behaviours for employees which were vital to meet the existing and upcoming performance requirements. The behaviours were then discussed with the trainers and the management. Finally, ANL desired behaviours for trainees were established.

b) Establishing trainer- intended behaviours To establish trainer-intended behaviours, one of the researchers participated in the orientation program as an observer. The researchers then thoroughly reviewed the training contents and interviewed the trainers. This enabled the researcher not only to fully appreciate the reasons and logic of the trainers behind the training contents but also the intentions of the trainers regarding behaviours they anticipated from trainees to be demonstrated in the actual environment.

c) Establishing trainees-intended behaviours To establish trainees-intended behaviours, the trainees were asked via questionnaires and interviews. The questionnaire contained three questions asking trainees to reveal the behaviours they had learned, the behaviours they wanted to implement and the behaviours they learned but they presumed could not be implemented in the actual environment. The same questions were also asked through interview to ensure the validity of the data. All these activities were carried out immediately after the training had finished.

d) Establishing actual work behaviours To determine trainees' actual work behaviours a variety of instruments were used. In the first step, questionnaires were used asking trainees regarding each of those behaviours which the trainers anticipated of them. The trainees, GM and MO were then individually interviewed. Lastly, some of the trainees were also observed while delivering their job responsibilities in their actual work environment. All these activities were initiated three months after the training ended. Results and discussion The analysis and findings of each evaluation level are separately given as follows 1) Reaction level Table 3 shows the responses of the trainees along with the summary of the descriptive statistics for the reaction level.

7

The individual mean scores reveal that trainees perceived the training content, which received the highest mean (M) score of 4.5, to be highly effective as compared to the training delivery methods and conduciveness of the training environment which received 4.1 and 3.8 scores respectively. The overall reaction level attained M = 4.1 with Min = 3.6 and Max = 4.6 and SD (Standard deviation) σ = 0.25. The values of M and SD clearly imply that though there is a low-level disparity among the views of the trainees, as a whole the training is viewed as an excellent effort in terms of its relevance to their actual job, content delivery methods and conduciveness of environment for learning. The Min and Max values basically show the lowest and highest points in the data. In the existing case, the perceived effectiveness of training at its lowest point amounts to 72% and at its highest it reaches 92%. 2) Learning level Table 4 shows trainees' responses pertaining to learning in terms of perceived improvement in their knowledge and skill. The table also contains the summary of the descriptive analysis of the learning level.

The mean value (M = 4.1) suggests that the trainees perceived training to be highly successful in adding value to their existing knowledge and skills. The value of SD (σ = 0.22) indicates that there is not much disparity among the perceptions of the trainees. The Min and Max value (Min = 3.7, Max = 4.3) suggest that the minimum improvement perceived by a trainee amounts to 74% and the maximum reaches 86%.

8

In the existing case the correlation amounts to 0.80 at a statistically significant margin with p-value 0.0161. This implies that there exists a strong positive linear correlation between the two levels. In other words, trainees’ perception about training in terms of its content delivery methods, the relevance of training contents to their job, and conduciveness of the training environment for learning, is strongly linked with trainees' learning of the knowledge and skills that are presented during the training program. 4. Behaviour level

Gap 1: For this gap, ANL desired behaviours were compared with trainers' intended behaviours. The

comparison revealed that trainers only emphasized 40-45% of the organisational desired behaviours. This huge

gap between ideal behaviours and trainers' intended behaviours pointed toward some significant issues. First,

ambiguity and vagueness on the part of the organisation with regards to the outcomes it desired to seek from

the training. Second, insufficient and inadequate understanding of the trainers regarding the best sales

practices and organisational requirements. Last, the lack of communication between ANL management and

the trainers. All these accounted for a training content which was seriously deficient in emphasizing the

requisite behaviours. To address this gap, ANL needed to take into consideration its desired behaviour and

make radical changes to the existing training contents i.e. add new contents.

Gap 2: To identify this gap, trainers' intended behaviours were compared with trainees' intended behaviours. The result revealed that there was a high level of consistency between these two sets of behaviours. This indicated that the strategies that were employed by the trainers to deliver training were quite effective. Consequently, training methodologies for delivering training were deemed appropriate and no considerable changes were suggested.

Gap 3: This is the third level of analysis in which trainee-intended behaviours were compared with their actual post-training work behaviours. There was a small variation between these two sets of behaviours. This variation appeared due to two apparent contradictions between the trainees’ intentions and their actual work behaviours. The first contradiction was, the trainees intended that in the actual work environment they might not use some of the techniques. Conversely, the trainees were frequently using those techniques during their work environment because these techniques were essential for a successful completion of certain jobs. The second contradiction was, the trainees intended that they would use written scripts during tele-selling. However, most of the trainees did not use such scripts during tele-selling. The reason being that they gained enough experience due to which they could perform better without using scripts for tele-selling. Beside these two contradictions, trainees were found to be using all those behaviours which are intended by them at the end of the training session. This analysis suggested that there was no major hillock that was impeding the transfer of learning from training space to the actual work environment. Gap 4: This is the last step of behavioural analysis in which ANL desired behaviours were compared with the actual post-training behaviours of the trainees. There was a huge variation between these two sets of behaviours. As stated earlier, the desired behaviours are established after the training ended. Most of the identified behaviours are not conveyed to the trainers beforehand so they have not been focused initially during the training design phase, and later on, consequently, in the delivery phase of the training. Therefore, the training only achieved 40-45% of the desired performance objectives. Although the training failed to achieve most of the organisation desired objectives, this failure could not be attributed to some specific stake holders or aspects of the training. Rather many factors within and outside the training environment contributed to this phenomenon. These factors included ambiguity and non-clarity on the part of management about the organisation desired objectives, lack of communication between trainers and management, trainees’ unawareness regarding best sales practices and organisation desired objectives etc.

Conclusion The evaluation model proposed in this paper is based on a system perspective which views training as a complex web of relationships between trainer, trainee, content, work environment and organisational goals (Stevenson, 2012). The broad perspective enables the model to provide a relatively deeper understanding in terms of identifying those factors that exist outside the realm of training but play a crucial role in the effectiveness of a training program. In the case at hand, the implementation of the proposed model resulted in not only pointing out but also assisting in providing bases for designing more effective training programs.

9

The findings of this study entail some important implications for researchers, managers and trainers. This study shows that there is a strong link between training environment and participants learning. Moreover, the effectiveness of training not only is linked with effective inculcation of skills, knowledge and behaviours but is also connected with the requirements of the organisation. The researchers are therefore required to take these linkages in consideration while conducting research studies on training evaluation. For managers, this study suggests that they must be fully aware of the needs of their organisations along with the prevailing best practices. This will enable them to provide the requisite guidance to the trainings undertaken by their organisations. Finally, the trainers along with expertise in pedagogy and content knowledge also require sufficient interaction with the management to identify the actual performance objectives for which the training is undertaken.

This study shows that the model successfully delivered the anticipated outcomes. However, it is important to note that the conclusion drawn here is mere suggestive because of the given limitations of the study. The study was conducted in only one organisation. Moreover, the size of the sample was rather small. Consequently, the outcomes may not be applicable to other small and medium organisations. The second limitation is that the study lacked rigorous statistical techniques such as control group and longitudinal data. This would have helped to report that the organisation who implemented the proposed evaluation model to their trainings have actually improved their organisation performance.

This research contributes to the human capital literature by having developed a training evaluation model for SMEs. Future studies can analyse the same model on other organisations to test its effectiveness. This model can be an excellent tool for analysing the training effectiveness as well as planning future trainings. References: Ahonen, G. (2000) “Generative and Commercially Exploitable Intangible Assets”, In: Classification of

intangibles, ed. J. E. Grojer and H. Stolowy, pp. 206-214. Alvarez, K., Salas, E. and Garofano, C.M. (2004) “An integrated model of training evaluation and effectiveness”,

Human Resource Development Review, Vol. 3, No. 4, pp. 385-416. Bassi, L. and McMurrer, D. (2004) “How are Your Return on People?”, Harvard Business Review, March, pp. 18. Bontis, N., Crossan, M. and Hulland, J. (2002) "Managing an Organizational Learning System by Aligning Stocks

and Flows", Journal of Management Studies, Vol. 39, No.4,pp. 437–469. Brinkerhoff, R. O. (2003) The success case method, San Francisco: Berrett-Koehler. Bhola, H. S. (1990) Evaluating "Literacy for Development": Projects, Programs and Campaigns, Hamburg:

UNESCO Institute of Education (UIE) / Bonn: German Foundation for International Development. Busari, J. O., Verhagen, E. and Muskiet, F. D. (2008) “The influence of the cultural climate of the training

environment on physicians' self-perception of competence and preparedness for practice”, BMC Medical Education, Vol. 8, No. 51

Choo, C. W. and Bontis, N. (2002) The Strategic Management of Intellectual Capital and Organizational Knowledge, New York: Oxford University Press.

Dessinger, J. C. and Moseley, J. L. (2004) Confirmative evaluation: Practical strategies for valuing continuous improvement. San Francisco: Pfeiffer.

Fraser, J. (2008) “Developing a Framework to Evaluate Training Programs Provided by WHO: The Feasibility of Incorporating Social Justice, Cultural Competency and Return on Investment”, International Journal of Learning, Vol. 14, No. 9, pp 103-109.

Gliem, J. A. & Gliem, R. R. (2003) “Calculating, Interpreting and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-type Scales”, Proceedings of The Midwest Research to Practice Conference in Adult, Continuing, and Community Education, Ohio State University, Columbus, OH.

Hamza, N. and Isa, R. M. (2010) “Intellectual and Social Capital Development: A Case in Malaysian’s ICT Companies”, International Journal of Business and Management, Vol. 5, pp 53-60.

Hellebrandt, J. and Russell, J. D. (1993) ”Confirmative evaluation of instructional materials and learners”, Performance & Instruction, Vol. 32, No. 6, pp 22-27.

Huang, C. and Hsueh, S. (2007) “A study on the relationship between intellectual capital and business performance in the engineering consulting industry: A path analysis” ,Journal of Civil Engineering and Management, Vol XIII, No. 4, pp. 265-271.

Hunter, L., Webster, E. and Wyatt, A. (2005) “Measuring Intangible Capital: A Review of Current Practice”, Australian Accounting Review, Vol. 15, No. 2, pp. 4-21.

10

Johanson, U. (2005), “A human resource perspective on intellectual capital”, in Marr, B. (Ed.), Perspectives on Intellectual Capital, Elsevier Butterworth-Heinemann, Burlington, MA, pp. 96-105. Boston: Elsevier.

Kaufman, R., Keller, J., & Watkins, R. (1995) “What works and what doesn’t: Evaluation beyond Kirkpatrick”, Performance and Instruction, Vol. 35, No.2,pp. 8-12.

Kirkpatrick, D. (1967) “Evaluation”, In R. L. Craig & L., R. Bittel (Eds.), Training and Development Handbook, American Society for Training and Development, New York: McGraw-Hill

Kirkpatrick, D. L. (1998) Another look at evaluating training programs, Alexandria, VA: American Society for Training & Development.

Kirkpatrick, D. L. (2006) Improving employee performance through appraisal and coaching, NY: AMACOM. Kong, E. and Thomson, S. B. (2009) “An intellectual capital perspective of human resource strategies and

practices”, Knowledge Management Research and Practice, Vol. 7, No. 4, pp. 356-364. Kraiger K. (2002) Decision-based evaluation In Creating, Implementing, and Maintaining Effective Training and

Development: State-of-the-Art Lessons for Practice, ed. K. Kraiger, pp. 331–75. San Francisco, CA: Jossey-Bass.

Lambert, B., Ohai, T. and Kerkhoff, E. (2009) World-class selling: New sales competencies. Alexendira, VA: ASTD Press.

Lather, A. S. (2009) “Role and Importance of Training Climate for Effective Training Programs”, Enterprise Risk Management, Vol. 1, No. 1, pp. 72-82.

Mathison, S. (1994) Evaluation. In Encyclopaedia of English Studies Language Arts, Champaign, IL: NCTE and Scholastic, Inc.

Misanchuk, E.R. (1978) “Uses and Abuses of Evaluation in Continuing Education Programs: On the Frequent Futility of Formative, Summative, and Justificative Evaluation”, Adult Education Research Conference, April, San Antonio, Texas.

OECD (2002) Management Training in SMEs, www.oecd.org/pdf (Accessed on 13 Dec, 2012) Ordonez de Pablos, P. (2005) “Western and Eastern views on social networks”, Learn Organ, Vol. 12, No. 5, pp.

436–456. Pelham, A. M. and Kravitz, P. (2008) “An Exploratory Study of the Influence of Sales Training Content and

Salesperson Evaluation on Salesperson Adaptive Selling, Customer Orientation, Listening, and Consulting Behaviors”, Journal of Strategic Marketing, Vol. 16, No.5, pp. 413-435.

Petty, R., and Cuganesan, S. (2005), "Voluntary Disclosure of Intellectual Capital by Hong Kong Companies: Examining Size, Industry and Growth Effects", Australian Accounting Review, Vol. 15, No. 36, pp. 40-50.

Pfeffer, J. (1998) The Human Equation: Building Profits by Putting People First, Boston: Harvard Business School Press.

Phillips, J. J. (1996) “Measuring ROI: The Fifth Level of Evaluation”,Technical and Skills Training, April, pp. 10-13.

Ramsey, P. L., Mackie, B. & Mason, R. (2008) “Evaluation by means, Learning and Performance Matter”, In Kumar,P. and Ramsey, P. L. (Eds.), Learning and Performance Matter, pp. 157-170. Singapore: World Scientific Publishing Co.

Raupach, T., Brown, J., Anders, S., Hasenfuss, G. and Harendza, S. (2013) “Summative assessments are more powerful drivers of student learning than resource intensive teaching formats”, Journal of medical Internet research, Vol. 15, No. 3, e50.

Roslender, R., and Fincham, R. (2001) “Thinking critically about intellectual capital accounting”, Accounting, auditing and accountability journal, Vol. 14, No. 3, pp. 383-398.

Sinjindawong, S., Lawthong, N. and Kanjanawasee, S. (2010) “The Development and application of the Meta-evaluation Standards for Thai Higher Education Institutions”, Research in Higher Education Journal, pp. 1-9.

Stevenson, P. (2012) “Learning and teaching in action”, Health Information and Library Journal, Vol. 29, pp. 81-86.

Stewart, T. A. (1997) Intellectual Capital: The new wealth of organisations, Doubleday, New York. Kulvisaechana, S. & Stiles, P. (2003) Human Capital and Performance: A Literature Review, Task Force on

Human Capital Management Reporting, Department of Trade and Industry, UK. Stufflebeam, D. L. (1966) “A depth study of the evaluation requirement”, Theory Into Practice, Vol. 5, No.1, pp.

121–34 Sujan, A., & Abeysekera, I. (2007) “Intellectual Capital Reporting Practices of the Top Australian Firms”,

Australian Accounting Review, Vol. 17, No. 2, pp. 71-83.

11

Sujan, H., Weitz, B. and Kumar, N. (1994) ”Learning Orientation, Working smart and effective selling”, Journal of Marketing, Vol. 58, No. 3, pp. 39-52.

Thiagarajan, S. (1991) “Formative evaluation in performance technology”, Performance Improvement Quarterly, Vol. 4, pp. 22-34.

Vantiem, D., Moseley, J. and Dessinger, J. (2000) Fundamentals of performance technology: A guide to improving people, processes, and performance, Washington DC: International Society for Performance Improvement.

Wang, G. and Wilcox, D. (2006) “Evaluation of Systematic Training: Knowing more than is practiced”, Advances in Developing Human Resources, Vol. 8, No. 3, pp. 528-539.

Warr, P., Bird, M., and Rackham, N. (1970) Evaluation of management training, London: Gower Press Westhead, P. and Storey, D. (1997) “Training Provision and Development of Small and Medium–Sized

Enterprises”, Research Report No. 26, London: DfEE. Yaghoubi, N., Salarzehi, H. Moloudi, J. and Yaghoubi, E. (2010) “Review of Relationship between Intellectual

Capital and Organizational Justice”, Asian Social Science, Vol. 6, No. 10, pp.120-126. Yang, C. C. and Lin, C. Y. Y. (2009), “Does intellectual capital mediate the relationship between HRM and

organizational performance? Perspective of a healthcare industry in Taiwan”, International Journal of Human Resource Management, Vol. 20, No. 9, pp. 1965-84.