O

ALC

gL

si(

acmi

h

a

fhci

sjpciaw

wwot

n

M

PMbH

6tsdw

sz

7

734

A

RIGINAL ARTICLE

Generalizability Study of the Instrumental Activities of Dailyiving Profile

arolina Bottari, PhD, Clément Dassa, PhD, Constant Rainville, PhD, Élisabeth Dutil, MSc

PssocctrpcEctcad

PtIdtaicAfittivipiqewat

c

ABSTRACT. Bottari C, Dassa C, Rainville C, Dutil É. Aeneralizability study of the Instrumental Activities of Dailyiving Profile. Arch Phys Med Rehabil 2010;91:734-42.

Objective: To establish generalizability estimates of the In-trumental Activities of Daily Living (IADL) Profile when admin-stered to persons with a moderate or severe traumatic brain injuryTBI) within their home and community environment.

Design: Generalizability theory was used to estimate gener-lizability and dependability coefficients as well as the relativeontribution of identified sources of measurement error to totaleasurement error. Decision studies were used to enable the

nvestigators to determine the optimal measurement design.Setting: The IADL Profile was administered in subjects’

omes and community environments.Participants: Convenience sample of adults with a moder-

te or severe TBI (N�30, aged 16–65y).Interventions: Not applicable.Main Outcome Measures: IADL Profile scores include 6

actor scores (going to the grocery store/shopping for groceries,aving a meal with guests/cleaning up, putting on outdoorlothing, obtaining information, making a budget, and prepar-ng a hot meal) and 1 total score.

Results: The greatest sources of measurement error were theubject-item interactions (3-random-facet design) and the sub-ect-rater interactions (2-random-facet design). One hundredercent of generalizability coefficients of factor scores indi-ated acceptable to excellent reliability. Indices of dependabil-ty confirmed that 1 evaluator could reliably score the tool on

single occasion after having received a 3-day trainingorkshop.Conclusions: The IADL Profile administered to persons

ith a moderate or severe TBI provides occupational therapistsith a reliable set of measures of IADL independence capablef both capturing and analyzing the complex interactions be-ween personal and environmental factors.

Key Words: Activities of daily living; Brain injuries; Cog-ition; Psychometrics; Rehabilitation.© 2010 by the American Congress of Rehabilitationedicine

From the School of Rehabilitation (Bottari, Dutil), the Departments of Social andreventative Medicine (Dassa) and Psychology (Rainville), Université de Montréal,ontreal, Quebec, Canada; and the Center for Interdisciplinary Research in Reha-

ilitation (Bottari, Rainville, Dutil), and the Interdisciplinary Research Group inealth (Dassa), Montreal, Quebec, Canada.Supported by the Fonds de la recherche en santé du Québec (FRSQ) (grant no.

623). The study was financed pursuant to a jointly cooperative arrangement betweenhe FRSQ, the Association québécoise d’établissements de santé et de servicesociaux, the Association des établissements de réadaptation du Québec, the Société’assurance automobile du Québec, and the Quebec Rehabilitation Research Net-ork. Also supported by the Canadian Occupational Therapy Foundation.No commercial party having a direct financial interest in the results of the research

upporting this article has or will confer a benefit on the authors or on any organi-ation with which the authors are associated.

Reprint requests to Carolina Bottari, PhD, École de réadaptation, Université de Montréal,077 Parc, Montréal, Québec, Canada, e-mail: carolina.bottari@umontreal.ca.

0003-9993/10/9105-00804$36.00/0doi:10.1016/j.apmr.2009.12.023

rch Phys Med Rehabil Vol 91, May 2010

ERSONS WHO SUSTAIN a moderate or severe TBI areknown to be at increased risk of lifelong disability affecting

ocial participation, particularly independence in IADL, lei-ure, and work skills.1-6 Understanding the underlying causesf these complex situations requires the use of psychometri-ally sound measurement tools of IADL independence that areapable of both capturing and analyzing the complex interac-ions between personal and environmental factors. A number ofelated studies have provided clear indications that the princi-al cause of impaired independence in TBI is the range ofomplex behavioral and cognitive disturbances associated withF.7-12 EFs are broadly defined as the capacity to plan andarry out complex goal-directed behavior.13-16 Executive func-ion deficits have been shown to be more severe in moreomplex17 and novel tasks,18 in minimally structured tasks,19

nd in the presence of distracting stimuli in complex andynamic environments (natural contexts).20,21

Performance-based methods of assessing independence inADL and in IADL22,23 are increasingly used by occupational

herapists to complement information obtained from PADL andADL questionnaires, particularly with persons with cognitiveeficits. Rogers24 defines IADL independence as resulting fromhe competence of people to do things for themselves in inter-ction with the environment in which they live. Competencemplies the ability to decide what one wants to do, to plan aourse of action, to do the task, and to assess the outcomes.24

lternately, EFs involve such components as problem identi-cation, goal setting, planning, carrying out the task, evaluating

he final product, and detecting/correcting errors.16,25 Based onhis visible overlap between the concepts of EFs and IADLndependence, we expect that interdisciplinary treatment inter-entions that rely on results obtained from IADL measurementnstruments, which fail to consider EFs and the person’s actualerformance in a real-world environment, are inevitably rely-ng upon inaccurate estimations of the severity of the conse-uence of the TBI on IADL. Perhaps more importantly, wexpect that such instruments may also be providing the teamith a poor analysis of essential causes of reduced participation

nd equivalently flawed recommendations regarding essentialreatment interventions.

To date, few researchers have addressed the methodologicalhallenges involved in developing and validating performance-

List of Abbreviations

ADL activities of daily livingD coefficients dependability coefficientsD-study dependability studyEFs executive functionsG coefficients generalizability coefficientsG-study generalizability studyIADL instrumental activities of daily livingPADL personal activities of daily livingt1 time 1t2 time 2

TBI traumatic brain injury

bseeattadCCPFtgoesiorritAmcfpAroctir

wsrtPieftttsrdsr(pdoccaia2uli

fnsg(bimdscstt(wTWrsmosiset

tatia“dtimidsb

P

tpTeptaTrfohrsaotc

735GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

ased IADL tests, that consider EFs and are administered inuch complex, highly individualized, and multidimensionalnvironments as a subject’s home and community.26-29 How-ver, this challenge was clearly addressed by related work onn instrument called the ADL Profile,22,30 an analytic observa-ion-based measure of ADL independence that also documentshe repercussions of executive function deficits on 17 PADLsnd IADLs. Theoretic and conceptual frameworks used in theevelopment of the ADL Profile include Luria’s Model oferebral Functioning16 and the Disability Creation Process.31

oncepts from Luria’s model were integrated into the ADLrofile to analyze the manner in which a task is performed.our operations that guide task analysis were thus identified on

he basis of the work of Luria16 and Lezak13: formulating aoal, planning, carrying out the task, and verifying attainmentf the initial goal. Also, based on the work of these authors, thevaluation approach suggested in the performance-based as-essment consists of providing the least possible structure andnstructions to the person in order to obtain a maximum amountf information on the person’s abilities, in particular thoseelated to EFs. The Disability Creation Process31 suggests thateduced participation is a situational state resulting from annteractive process between personal and environmental fac-ors. Hence, unlike many traditional ADL assessments, theDL Profile is administered in the person’s home and com-unity environment, as it is within this environment that one

an most clearly understand the influence of environmentalacilitators and obstacles on performance. A recent review oferformance-based PADL and IADL assessments ranked theDL Profile first among standardized ADL measures with

egard to its correspondence to constructs of importance forccupational therapists, such as holistic perspective and client-enteredness.32 The psychometric properties of this test (con-ent validity, trait validity, factorial validity, interrater reliabil-ty, test-retest reliability, internal consistency) have beeneported elsewhere.22,33-37

The present study was designed to expand upon this previousork as we estimated the generalizability of an alternate ver-

ion of the ADL Profile,22 the IADL Profile, which considersecent advances in research regarding EFs, more specifically inhe area of the ecological assessment of EFs.27 The IADLrofile is an ecological measure of independence administered

n a subject’s home and community environment that aims tostablish whether the subject’s main difficulties pertain toormulating goals, planning, carrying out the task, and/or at-aining the initial task goal—all important components of EFshat can be impaired subsequent to a TBI. It uses a nonstruc-ured evaluation approach to maximize the possibility of ob-erving deficits related to goal formulation and planning. Whenequired, assistance can be provided by the examiner. This isone in a hierarchical and graded fashion and is based onpecific guidelines. Each of the tool’s 29 items is scored by theater/therapist using a 5-level ordinal scale of independenceindependence [4], independence with difficulty [3], verbal orhysical assistance [2], verbal and physical assistance [1], andependence [0]) (see Bottari et al38-40 for a detailed explanationf the scale). Verbal assistance includes such things as explicitues to formulate goals, plan, initiate the task, detect andorrect errors, and verify attainment of the goal, while physicalssistance includes such things as physical support when walk-ng outdoors or to prevent a fall, and assistance to manipulatepot or carry objects. An important distinction between thesetypes of assistance is that the former is provided because of

nderlying cognitive deficits and the latter because of under-ying physical deficits. In a previous study39 based on 100

ndividuals with a moderate or severe TBI, principal axis t

actoring and confirmatory factor analysis provided prelimi-ary support for 6 correlated factors: (F1) going to grocerytore/shopping for groceries (8 items), (F2) having a meal withuests/cleaning up (7 items), (F3) putting on outdoor clothing4 items), (F4) obtaining information (3 items), (F5) making audget (3 items), and (F6) preparing a hot meal for guests (4tems). Each factor provides a factor score, which is a unidi-ensional score that reflects an individual’s level of indepen-

ence on a single task or a combination of 2 tasks. Factorcores are obtained by calculating the average score of all itemsomposing each factor. Total explained variance of the 6-factorolution was 73.6%. Cronbach’s alpha analysis revealed higho very high internal consistency for all scales ranging from .81o .98. Internal consistency of the total scale was very high.95). Subjects tested with the IADL Profile were also testedith 3 measures of EFs (Stroop Color and Word Test,41,42

ower of London,43 the Working Memory Index of theeschler Memory Scale III44). Results showed moderate cor-

elations between the IADL Profile factor scores and the totalcore with measures of injury severity (particularly posttrau-atic amnesia), as well as moderate correlations with measures

f planning and working memory.45,46 Results of multipletepwise regressions showed posttraumatic amnesia and work-ng memory to be the major determinants of IADL Profilecores. Together, indices of injury severity, measures of EFs,ducation, age, and environmental factors accounted for 12%o 28% of the variance in IADL Profile scores.38

The present study is a tool design study that sought to estimatehe generalizability of the IADL Profile when administered todults with a TBI in their home and community environment. Inhe context of generalizability theory, the main focus was tonvestigate the degree to which subjects’ observed scores “gener-lized to a more extensive set of measurements” or universe scoresunder a specifiable set of measurement conditions”47(p188) (seeata analysis section for further explanation).47 The main advan-age of generalizability theory over the classical reliability theorys its simultaneous consideration of multiple sources of measure-ent error in a single analysis (eg, occasions, raters, items),48,49 an

ssue of importance in the context of observational proce-ures.50,51 Use of this theory permits the identification of the exactource and magnitude of measurement errors and the interactionetween different sources of error.49

METHODS

articipantsSubjects were recruited from 8 postacute TBI programs in

he province of Quebec, Canada. Criteria for inclusion were (1)atient age between 16 and 65 years, (2) moderate or severeBI, and (3) proficiency in French. The severity of the TBI wasstimated with the Glasgow Coma Scale52 and duration ofosttraumatic amnesia. Exclusion criteria were (1) disorienta-ion, (2) severe mobility deficits, (3) severe language deficits,nd (4) a history of hospitalization for psychiatric disorders.his study used multiple designs: that is, 1 random facet, 2

andom facets, 2 random facets and 1 fixed facet, and 3 randomacets. As it was the first study to document the generalizabilityf a new measure of independence administered in a subject’some and community environment, we used the benchmarkeference of Donner and Eliasziw53 to estimate the requiredample size. These authors suggest that 30 subjects is andequate sample size for a 1-facet design—one of the designsf our study. Although this reference does not formerly extendo the other designs of our study (2- and 3-facet designs), forompletion, we analyzed these data in this same sample. Before

he evaluation, subjects were given verbal and written infor-

Arch Phys Med Rehabil Vol 91, May 2010

mTt

I

Pshfhabvta“igvfaptstcoctfiri

R

tmTuswcssPtiEccsi

pmitca(iot

P

OwDhtwecsho

bfssuTsAammnies

D

wgpmuptuGoctfsac(gfptcacHafaouac

736 GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

A

ation on the study and signed statements of informed consent.he study was approved by the ethical review boards of par-

icipating centers.

nstrumentationInitial instructions for the first series of 6 tasks of the IADL

rofile (ie, putting on outdoor clothing, going to the grocerytore, shopping for groceries, preparing a hot meal for guests,aving a meal with guests, cleaning up after the meal) are asollows: “Without knowing it, you invited my assistant and I toave lunch with you. Please get ready to receive us. We willssume any incurred expenses for a maximum of $20.” Foroth this series of tasks and subsequent tasks, the examiner firsterifies that the subject clearly understands the instructions andhen adds the following information to document the subject’sbility to formulate goals and plan the series of related tasks:Now, please tell me what you are going to do.” With thesenstructions, it is expected that subjects will actually go to therocery store and shop for groceries so that real-world obser-ations in a community setting can be made. This will beollowed by the actual preparation of the meal within the homend a sharing of the meal with the evaluator. The concept ofreparing a meal for guests has an added level of complexity tohat of preparing a meal for oneself because there are increasedocial expectations and additional attention that must be giveno meal selection and organization, social perception, andleaning. All of these combine to make this situation a samplef EFs tasks and an adequate reflection of independence in aomplex situation. The last 2 tasks in the tool are more struc-ured. The task of obtaining information requires the subject tond a bus schedule to a distant site, and making a budgetevolves around the task of planning to buy a car on a fixedncome.39 Administration time is approximately 3 hours.

atersThe 4 occupational therapists who participated as raters in

his study had been practicing occupational therapists for aean of 5.25 years (range, 1–11y), and had been working inBI programs for a mean of 4 years (range, 1–8y). Ratersnderwent a standardized training regimen, a 3-day trainingession given by the primary investigator (rater 4). Here ratersere introduced to the architecture of the test (objectives,

onceptual frameworks, variables, administration procedure,coring, and interpretation) and practiced task analysis andcoring supported by videotaped administrations of the tool.articular emphasis was placed on teaching the occupational

herapists to analyze and score observable task-related behav-ors according to the tool’s underlying 4 operations related toFs. Finally, to ensure the training of raters to a definedriterion regarding accuracy of rating, each rater scored theomplete evaluations of 3 subjects before the onset of thetudy. Results were reviewed and corrected by the primarynvestigator to ensure rating quality.

All subjects were evaluated with the IADL Profile by therimary investigator in their home and community environ-ents within a single session. The primary investigator was not

nvolved clinically with the clients whom she met solely foresting purposes. The evaluation was recorded with a videoamera by a research assistant, and the video recording of thessessment was subsequently analyzed by the 3 other ratersraters 1–3) who completed 2 independent ratings at a 1-monthnterval of each other. Raters worked independently of eachther and were told not to discuss ratings among themselves

hroughout the duration of the project. w

rch Phys Med Rehabil Vol 91, May 2010

roceduresData collection took place between February 2005 and

ctober 2005. Administration of the IADL Profile occurredithin a single session and lasted approximately 3 hours.ressing to go outdoors, preparing a hot meal for guests,aving a meal, cleaning up after the meal, obtaining informa-ion, and making a budget were evaluated within the home,hile going to the grocery store and shopping for groceries was

valuated in the person’s community environment. Tasks wereonsistently tested in the same order. First we administered theeries of 6 tasks linked to the overarching goal of preparing aot meal for guests. This was then followed by the tasks ofbtaining information and making a budget.Repeated testing was not feasible because it would have

een unreasonable to ask subjects with known problems ofatigue to repeat the evaluation, and having 4 raters in theubject’s home at any one time would have been overly intru-ive and generally not possible. Hence, video recordings weresed to allow raters to score the test on 2 separate occasions.his procedure has been used in other interrater reliabilitytudies of performance-based measurement instruments.54,55

lthough use of video recordings is not directly equivalent ton in-person observation by all 4 raters, this methodology didinimize the error associated with changes in IADL perfor-ance secondary to the task becoming more automatic and less

ovel for the subject when testing is repeated (particularlymportant in relation to EFs, as tests are most sensitive toxecutive function deficits during initial testing when the testituation is most novel).56

ata AnalysisTo examine the reliability of the instrument, G coefficients

ere computed. When compared with classical test theory, thereater interest of generalizability theory is that it provides aowerful way to disentangle and identify sources of measure-ent error (eg, occasions, raters, and items) that can then be

sed to more clearly understand a given assessment tool androcess. Generalizability theory (G-study) is “concerned withhe extent to which a sample of measurements generalizes to aniverse of measurements,”47(p158) and the computed statistic, a

coefficient, estimates the reliability of a measurement. Inrder to do so, the universe is defined in terms of measurementonditions called facets that capture relevant sources of varia-ions considered as errors of measurement. In this study, 3acets were examined: raters, items within a score, and occa-ions. Two types of G coefficients were computed: relative andbsolute.57 In keeping with the development of a new tool, weonsidered the item facet as random in all measurement designs2 random facets, 3 random facets), implying that results can beeneralized to a much larger pool of equivalent items, exceptor one design (2 random facets and 1 fixed facet). However, inractice, it is not intended that results be generalized beyondhe fixed content of the specific 29 items in the tool, in whichase the item facet is fixed. In fact, in the IADL Profile, itemslways refer to the same 4 operations: formulate a goal, plan,arry out the task, and verify attainment of the initial goal.ence, for completion, the data were analyzed twice, once withrandom-item facet (3-random-facet design and 2 random

acet design) and once with a fixed-item facet (2 random facetsnd 1 fixed facet design). When facets are set as random, aspposed to fixed, the uncertainties about generalization to theniverse score increases. Thus, the generalizability coefficientsre typically lower. For the purposes of this article, the mostonservative results (ie, the lower generalizability coefficients)

ill be presented. For coefficients of generalizability inferior to

0o

dtidsa(alT

2tw(aciDsArta

Ac ing a* o .70

Ac

737GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

.7 with a random-item facet, the equivalent coefficients basedn a fixed-item facet will also be presented.For the factor scores (6 factors) and the total score, 2 crossed

esigns were used: a single-facet design with 4 raters (the 3rained raters [raters 1, 2, and 3] at time 1 and the primarynvestigator [rater 4]) assessing all subjects, and a 2-facetesign with 3 trained raters assessing all subjects on 2 occa-ions (t1, t2). For each design, the G coefficients are firstveraged over the exact raters and occasions as defined abovethe so-called G-study), then for a number of relevant designsimed at optimizing decision-making (D-study). We ran simi-ar analyses using the items considered in each factor score.wo crossed designs were also used in this approach. First, a

Table 1: G-Study and D-Study Coefficients

Factor No.

G-Study

Relative Absolute

1 Facet (raters)F1 .996 .996F2 .971 .970F3 .999 .998F4 .988 .986F5 .990 .989F6 .985 .983Total score .996 .994

2 Facets (raters, items)F1 .959 .951F2 .809 .780F3 .995 .993F4 .876 .816F5 .956 .940F6 .728 .714Total score .924 .885

bbreviations: F1, going to grocery store/shopping for groceries; F2,lothing; F4, obtaining information; F5, making a budget; F6, preparCoefficients in bold represent generalizability coefficients inferior t

Table 2: G-Study and D-Study Coefficients

Factor No.

G-Study

Relative Absolute R

2 Facets (raters, occasions)F1 .997 .997F2 .970 .970F3 .999 .998F4 .987 .987F5 .990 .990F6 .986 .986Total score .997 .997

3 Facets (raters, occasions, items)F1 .961 .954F2 .808 .782F3 .996 .995F4 .879 .819F5 .950 .938F6 .746 .733Total score .928 .892

bbreviations: F1, going to grocery store/shopping for groceries; F2, havinlothing; F4, obtaining information; F5, making a budget; F6, preparing a

-random-facet design in which the variability associated withhe 4 raters (raters 1, 2, 3, and 4) at time 1 and that associatedith the items (variable according to the factor) is considered

table 1). Second, a 3-random-facet design in which the vari-bility associated with 3 raters (raters 1, 2, and 3), that asso-iated with the occasion (t1, t2), and that associated with thetems is considered (table 2). The D-study was used to calculate

coefficients that reflect clinical reality; that is, in the clinicaletting a single therapist assesses each subject on 1 occasion.dded to this, we calculated D coefficients to examine the

eliability of the tool in instances where 2 examiners would ratehe test on 1 occasion. Finally, we considered a situation wheresingle therapist scores a subject’s performance on 2 occasions

Raters, 1 Occasion, 29 Items, 28 Subjects

D-Study

2 Raters 1 Rater

Relative Absolute Relative Absolute

.993 .991 .986 .983

.944 .943 .895 .891

.997 .997 .994 .993

.976 .973 .952 .947

.980 .979 .961 .959

.970 .967 .941 .936

.991 .988 .983 .976

.956 .947 .948 .939

.787 .758 .745 .718

.994 .992 .991 .988

.866 .806 .845 .786

.945 .930 .924 .909

.717 .702 .696* .680*

.920 .880 .912 .870

g a meal with guests/cleaning up after meal; F3, putting on outdoorhot meal for guests.

.

Raters, 2 Occasions, 29 Items, 28 Subjects

D-Study

ers/1 Occasion 1 Rater/2 Occasions 1 Rater/1 Occasion

e Absolute Relative Absolute Relative Absolute

.994 .993 .993 .989 .989

.943 .915 .915 .892 .892

.997 .996 .995 .995 .994

.971 .963 .963 .945 .944

.977 .974 .974 .959 .959

.974 .961 .961 .950 .950

.994 .992 .991 .988 .987

.951 .956 .948 .953 .945

.760 .762 .738 .743 .720

.994 .994 .992 .993 .991

.806 .857 .800 .839 .785

.922 .927 .915 .912 .899

.722 .724 .712 .715 .703

.889 .923 .887 .919 .883

for 4

havin

for 3

2 Rat

elativ

.995

.943

.998

.972

.977

.974

.994

.958

.786

.995

.863

.934

.735

.925

g a meal with guests/cleaning up after meal; F3, putting on outdoorhot meal for guests.

Arch Phys Med Rehabil Vol 91, May 2010

(fu

asafato

taeowmrrwg3ipfstwt

fras(dfcsp(snD2s2swedf(r

tGdfrfct

ifistd

1cebrtaeit1eetesddtc

stsda(tat

wsoop4rwrw(a

raraGvstcPu

g

738 GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

A

see tables 1 and 2). All generalizability analyses were per-ormed using EduG (3.0).58,a Other analyses were performedsing SPSS (15.0).b

G and D coefficients can be interpreted in the same manners any reliability coefficient: values of .70 or higher are con-idered acceptable for basic research purposes,59 and values oft least .80 are considered satisfactory and usually necessaryor high-stakes decisions.48,60 In the context of this study, Gnd D coefficients less than 0.6 indicate that generalization tohe universe score is not acceptable, and G and D coefficientsf 0.9 or higher indicate an excellent generalizability.

RESULTSThe average age of the subjects was 40.9 years, and most of

hem were men (77%). Sixty-three percent had a severe TBI,nd the average time postinjury was 12.9 months. Averageducation was 11.6�3.4 years. Most subjects (76.7%) walkedutdoors with or without an orthosis, and only 2% used aheelchair. Most of our subjects (73%) lived with others, andost lived in an urban setting (63%). With a score of 4

epresenting independence without difficulty and a score of 0epresenting dependence, mean scores on the IADL Profileere as follows: (F1) going to grocery store/shopping forroceries, 2.94; (F2) having a meal with guests/cleaning up,.57; (F3) putting on outdoor clothing, 3.61; (F4) obtainingnformation, 2.22; (F5) making a budget, 1.30; and (F6) pre-aring a hot meal for guests, 3.23. Two subjects were removedrom the final analysis because of missing data. For 1 of theseubjects, the presence of extremely icy winter conditions madehe task of going to the grocery store too risky and dangerous,hereas the other subject had to interrupt the evaluation to go

o a medical appointment.Table 1 gives the G and D coefficients (relative and absolute)

or the 6 factor scores as well as for the total score (with 4aters at t1) for 2 measurement designs: 1 random facet (raters)nd 2 random facets (raters, items). The results were veryimilar whether we took into account only the raters’ variations1-random-facet analysis) or the raters’ variations and the in-ividual items’ variations within a given factor (2-random-acet analysis), except for factor 6. For this factor, the Goefficient obtained from the item scores composing the factorcore (2-random-facet analysis) was acceptable for basic researchurposes, whereas the one obtained from the factor scores1-random-facet analysis) was acceptable for high-stakes deci-ion-making, as it was well above .90. Little difference wasoted between absolute and relative coefficients. Results of the-study indicated that reducing the number of raters (from 4 toor 1) did not significantly affect the reliability of the factor

cores or the total score, except for factor 6. In the latter case,G coefficients that were slightly superior to .70 became

lightly inferior to .70 in the D-study with 1 rater, but onlyhen item score variations within a given factor were consid-

red (2-random-facet analysis), and not when factor scoresefined by averaging the items were considered (1-random-acet analysis). However, when the item facet was set as fixed2-random and 1-fixed-facet analysis), D coefficients with 1ater increased to .94, indicating excellent reliability.

When the ratings of raters 1, 2, and 3 were analyzed at t1 and2, one more facet was considered (occasion). Table 2 gives the

and D coefficients (relative and absolute) for 2 measurementesigns: 2 random facets (raters, occasions) and 3 randomacets (raters, items, occasions) for factor and total scores. Theesults were essentially parallel to those presented previouslyor the 4 raters. Table 3 provides estimates of the varianceomponents and the error variances of total scores for each of

he following measurement designs: 3 random facets (raters, i

rch Phys Med Rehabil Vol 91, May 2010

tems, occasions), 2 random facets (raters, occasions) and 1xed facet (items), and finally 2 random facets (raters, occa-ions). In the first 2 designs, we analyzed the data by includinghe 29 items composing the total scores, whereas in the thirdesign we analyzed only the total scores.Overall results of the generalizability study showed that

00% of the generalizability coefficients of factor scores indi-ated acceptable to excellent reliability, with most (75%) co-fficients values being larger than .90. Absolute coefficientsased on observed mean scores were almost as reliable aselative coefficients based on deviation scores. Hence in prac-ice, assessing a patient’s independence per se is as reliable asssessing it in relation to other patients. The greatest source ofrror variance in the 3-random-facet design (raters, occasions,tems) was the subject-item interactions, explaining 61.7% ofhe error (see table 3). In the 2-random (raters, occasions) and-fixed (item)-facet design, the greatest source of measurementrror was the subject-rater interactions, explaining 50.7% of therror variance, followed by the subject-rater-occasion interac-ions (44.8%) (see table 3). In the 2-random-facet design (rat-rs, occasions), the greatest sources of measurement error werehown to be exactly the same as the 2-random and 1-fixed-facetesign, which is in line with the meaning of fixed-item-facetesigns. Although the similar results that were found pertainingo all factor scores were not reported here, these were fairlyonsistent with total score results.

In summary, because the factor and total scores are thecores that will be used clinically, it is worthwhile noting thathe corresponding design in table 3 (2 random facets for totalcores) showed 2 things: 98.7% of variance components wereue to differences among subjects, leaving only 2.3% of vari-nce to be explained by subject-rater-occasion interactions0.8%) and subject-rater interactions (0.4%). The latter 2 werehe basis of error variances, which were respectively reporteds 44.8% and 50.7%. Hence, the errors were small and essen-ially due to the raters.

When the item facet was random (3-random-facet design),hich expresses percentage of error of the observed mean

core when item scores are generalized to a hypothetical poolf equivalent alternate IADL items, the coefficient of variationf the standard error of the mean was below 6%. However, theercentage of error of the observed mean score decreased to% when restricted to the 29 items of this instrument (2-andom-facet and 1-fixed-facet design); that is, when the itemsere considered as a fixed facet. When the item facet was

emoved from the analysis (2-random-facet design), resultsere the same as those obtained when the item facet was fixed

2-random-facet and 1-fixed-facet design); that is, the percent-ge of error of the observed mean score was 4%.

Results of the D-study indicated that reducing the number ofaters (from 3 to 2 or 1) did not significantly affect the reli-bility of the factor scores or of the total score. Moreover,esults showed that the tool was just as reliable when scored onsingle occasion as on 2 occasions. Hence, results of both the-study and the D-study indicate 2 things of particular rele-ance to clinical practice. First, trained single raters based on aingle observation can reliably report both the factor scores andhe total score of the IADL Profile. Second, the training re-eived by the raters in this study is intrinsic to the IADLrofile’s reliability. Hence, we will recommend that subsequentsers of the tool participate in a similar training workshop.

DISCUSSIONThe present study was prompted by the need to establish the

eneralizability of a new performance-based measure of IADL

ndependence (the IADL Profile) administered in the home and

cT1crtitttoisocwfirBaeom

dbwpt

eti(

E

optidwigilavvsfsedieti

A S, su

739GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

ommunity environments of adults with moderate and severeBI. Overall, results of the generalizability study showed that00% of the generalizability coefficients of factor scores indi-ated acceptable to excellent reliability. The strength of theseesults can be partly explained by the training received by theherapists and the detailed scoring guide that was provided. Themportance of observer training and well-defined scoring cri-eria is well known to contribute to the reliability of observa-ional assessments.61 Moreover, the scoring procedure requiredherapists to observe subjects in numerous situations (8 tasks),ver an extended period (about 3h), and to summarize behav-ors related to each item before assigning a score on the ratingcale, an element thought to contribute to the reliability ofbservational assessment tools. Finally, factor scores werealculated by averaging results of the various item scores ofhich they are comprised. It is known that aggregated scores

rom multiple observations are more reliable than individualtem scores, and scores from longer observations are moreeliable than scores from brief observations.61 According torentnall and Bundy,61 some sources of measurement errorssociated with each individual observation are averaged orven “cancelled out” when scores are aggregated from multiplebservations, thereby increasing the reliability of the instru-ent.To our knowledge, this study is one of the first to have

ocumented the generalizability of an ecological, performance-ased measure of independence in IADL that determineshether subjects’ difficulties are related to goal formulation,lanning, carrying out, or attaining the initial task goal. Despite

Table 3: Variance Components in Percentage of the Total VarianTotal Error Variance for IADL Profile Total Sco

Source of Variation

3 Random Facets (R,I,O

VarianceComponent

Error Varfor

MeasuremDesig

Item FacetRandom

Item FaRando

S 21.1 NAR 0.0 0.1I 26.4 35.6O 0.0 0S � R 0.0 0.1S � I 45.7 61.7S � O 0.0 0R � I 0.1 0R � O 0.0 0I � O 0.0 0S � R � I 2.9 1.3S � R � O 0.1 .3S � I � O 0.0 0R � I � O 0.0 0S � R � I � O 3.8 0.9

Item Facet RandomGrand mean for levels used 3.115Variance error of the mean for all

levels used 0.034SEM 0.184Coefficient of variation of SEM with

respect to the grand mean 0.059

bbreviations: I, items; NA, not applicable; O, occasions; R, raters;

he overall quality of the results, the following discussion will t

xamine 2 sources of measurement error, one speculated andhe other demonstrated: (1) the evaluation context and itsnfluence on the scoring of the items of the IADL Profile, and2) the raters.

valuation ContextReal-world assessments are increasingly considered as the

ptimal manner in which to document the interplay betweenersons’ neuropsychologic deficits and the requirements ofheir daily lives for a better appreciation of everyday function-ng.62-64 However, evaluations that are administered withinynamic and complex real-world environments are subject to aide variety of influences that are not controlled by the exam-

ner. For instance, the scoring of items related to the tasks ofoing to the grocery store and shopping for groceries wasnfluenced by the environmental contexts in which subjectsived. In this study, 63% of subjects lived within urban settingsnd 37% within rural settings. Distances to the grocery storearied greatly between these 2 settings and led to a largeariability in the spectrum of behaviors to be analyzed andcored. For instance, in rural settings, subjects frequently livedar from the grocery store. In the context of the test protocol,ome subjects then had to be driven to the grocery store by thexaminer. For most subjects, this was due to their loss ofriving privileges secondary to their TBI combined with theirnability to find a suitable alternative (ie, one not requiring thexaminer’s intervention). Scoring guidelines developed withinhe context of our studies on the IADL Profile state that in suchnstances, subjects are to be rated as requiring assistance for the

d Error Variance for Measurement Design in Percentage of the3 Raters, 2 Occasions, 29 Items, 28 Subjects)

2 Random Facets (R,O) and 1Fixed Facet (I) 2 Random Facets (R,O)

VarianceComponent

Error Variancefor

MeasurementDesign

VarianceComponent

ErrorVariance for

MeasurementDesign

Item FacetFixed

Item FacetFixed

No ItemFacet

No ItemFacet

22.9 NA 98.7 NA0.0 4.5 0.0 4.5

25.7 0.0 NA NA0.0 0.0 0.0 00.1 50.7 0.4 50.7

44.5 0.0 NA NA0.0 0.0 0.0 00.1 0.0 NA NA0.0 0.0 0.0 00.0 0.0 NA NA2.8 0.0 NA NA0.2 44.8 0.8 44.80.0 0.0 NA NA0.0 0.0 NA NA3.7 0.0 NA NA

Item Facet Fixed No Item Facet3.115 90.339

0.016 13.4080.126 3.662

0.040 0.040

bject; SEM, standard error of the grand mean.

ce anres (

)

iance

entncetm

ask of going to the grocery store. This need for assistance is

Arch Phys Med Rehabil Vol 91, May 2010

jipmlt

R

thesG.msmwtv

fltfworlgitftc

S

ardawss((otacndi

ootoHtspr

n“

tmalacTt

tlsaPp

estctmiTmohpfbt

afTcrp

LDptM

740 GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

A

udged to be a direct repercussion of their TBI on their level ofndependence. These results highlight the extent to which 2ersons with identical deficits living in 2 different environ-ents (rural, urban) can function at 2 completely different

evels (independence, assistance), depending on their proximityo services such as a grocery store.

atersVia the D-study, we have shown that a single occupational

herapist can reliably score the test within a single session afteraving participated in a 3-day workshop, scoring 3 completevaluations, and adhering to stringent scoring guidelines pre-ented in the IADL Profile manual. Although results of the-study showed a relative G coefficient for total scores to be

997, suggesting a very small portion of error in the measure-ent, results of the 2-random-facet (raters, occasions) mea-

urement design showed subject-rater interactions to be theain source of variation to consider. Thus, as in practice weill be using factor and total scores, which were considered in

his 2-random-facet measurement design, the main source ofariation to consider will mainly be the raters.Reliability was shown to be at its weakest for the scoring of

actor 6 (preparing a hot meal for guests). These are particu-arly interesting results when one considers that meal prepara-ion is likely one of the tasks that is most frequently usedor testing purposes by occupational therapists, particularlyithin the context of discharge planning. The frequency of usef this task in occupational therapy may in part explain theseesults, as therapists over time develop their own set of guide-ines to score this task. In so doing, therapists may forego testuidelines, allowing their professional judgment to supersedenstructions provided in the test manual, and implicitly reducehe test’s reliability. This problem will need to be addressed inuture training sessions as therapists are taught to respect aool’s rating guidelines over and above their own subjectivelinical reasoning.

tudy LimitationsSome potential limitations of the present study merit consider-

tion. One limitation concerns its small sample size. Although itespected the requirements for a single-facet generalizability studyesign, the current sample size was nonetheless small for the 2-nd 3-facet study designs. However, a post hoc power analysisith the significance level set at .05, and power set at .80,

howed that when a G coefficient value was estimated, in thistudy (with our sample) to be .95, we are reasonably assuredpower of 80%) that our design would detect a real coefficientin the population) of .83. Similarly, an estimated G coefficientf .80 would tend to detect a real coefficient of .68. Both ofhese values are large enough to indicate acceptable generaliz-bility.48,59,60 Alternately, the strength of the generalizabilityoefficients that we found may in part be explained by the largeumber of rater-based observations collated, and the use of aetailed scoring manual that illustrated how observed behav-ors were to be scored.

Another potential limitation concerns the generalizability ofur findings. Our study focused primarily on therapists’ ratingsf video recordings of the test, administered to all subjects byhe primary investigator. This sampling limits the applicabilityf our findings to different therapists “administering” the test.ence, the effect of a therapist’s patience and ability to observe

he subjects without intervening unnecessarily was not mea-ured, potentially reducing the error attributed to raters. This, inart, explains why the conclusion states that 1 evaluator can

eliably “score” the tool on a single occasion. Future studies are

rch Phys Med Rehabil Vol 91, May 2010

eeded to show whether more than 1 evaluator can reliablyadminister” and “score” the tool on a single occasion.

Finally, because raters scored the same video on 2 occasions,he effects of patient learning on repeated testing were re-oved, though voluntarily, as executive dysfunction is most

pparent in novel and complex tasks. Repeat testing has theikely potential of removing the novelty of the test situationnd, in the presence of executive dysfunction, completelyhanging the manner in which the person performs the task.his is a well-known limit of all tests of executive dysfunc-

ion.56

Future studies are required to verify the reproducibility ofhe results presented here on the 2- and 3-facet designs on aarger sample. Future studies are also required to address thepecific needs of persons with a mild traumatic brain injury, asdministration of the complete series of tasks of the IADLrofile may not be required for this higher-functioning group ofeople.

CONCLUSIONSThis study provides occupational therapists a much-needed

cological measure of everyday functioning administered in aubject’s home and community environment that documentshe interplay between the individual’s neuropsychologic defi-its and the requirements of his or her daily life. Studies relatedo the IADL Profile are among the first to have addressed theethodological challenges involved in developing and validat-

ng performance-based IADL tests for individuals with aBI, administered in such complex, highly individualized, andultidimensional environments. With its emphasis on directly

bserving the ability of persons to carry out tasks in theirome and community environment, the IADL Profile has theotential for providing specific task-based recommendationsor treatment planning that will be easily understandable byoth the family and other members of the interdisciplinaryeam.

In summary, the IADL Profile can be used with confidence,s it demonstrates highly acceptable generalizability estimatesor its intended use with persons with a moderate or severeBI. We expect that use of this measurement instrument inlinical and research settings may contribute new knowledgeegarding the influences of real-world environments on IADLerformance in TBI.

Acknowledgments: We thank Mariama Touré, BSc(OT), Emilieemay-Brault, BSc(OT), Réjean Prévost, MSc, and Julie Lamoureux,MD, for their contribution to the study. We also thank all partici-ating rehabilitation centers that recruited subjects for this study, andhe Center for Interdisciplinary Research in Rehabilitation of Greater

ontreal for its overall support.

References1. Brzuzy S, Corrigan JD. Predictors of living independently after

moderate to severe traumatic brain injury: a comparison study.J Head Trauma Rehabil 1996;11(3):74-83.

2. Colantino A, Ratcliff G, Chase S, Kelsey S, Escobar M, VernichL. Long term outcomes after moderate to severe traumatic braininjury. Disabil Rehabil 2004;26:253-61.

3. Dawson DR, Chipman M. The disablement experienced by trau-matically brain-injured adults living in the community. Brain Inj1995;9:339-53.

4. Gordon WA, Zafonte R, Cicerone K, et al. Traumatic brain injuryrehabilitation: state of the science. Am J Phys Med Rehabil2006;85:343-82.

5. Kozlowski O, Pollez B, Thevenon A, Dhellemmes P, Rousseaux

M. Devenir et qualité de vie à trois ans dans une cohorte de

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

5

741GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

patients traumatisés crâniens graves. Ann Readap Med Phys2002;45:466-73.

6. Whiteneck G, Brooks CA, Mellick D, Harrison-Felix C, SendroyTerrill M, Noble K. Population-based estimates of outcomes afterhospitalization for traumatic brain injury in Colorado. Arch PhysMed Rehabil 2004;85(Suppl 2):S73-S81.

7. Chevignard M, Pillon B, Pradat-Diehl P, et al. An ecologicalapproach to planning dysfunction: script execution. Cortex 2000;36:649-69.

8. Eslinger PJ, Damasio AR. Severe disturbance of higher cognitionafter bilateral frontal lobe ablation: patient EVR. Neurology 1985;35:1731-41.

9. Fortin S, Godbout L, Braun CM. Cognitive structure of executivedeficits in frontally lesioned head trauma patients performingactivities of daily living. Cortex 2003;39:273-91.

0. Shallice T, Burgess PW. Deficits in strategy application followingfrontal lobe damage in man. Brain 1991;114:727-41.

1. Stuss DT, Benson DF. The frontal lobes. New York: Raven Press;1986.

2. von Cramon DY, Matthes-von Cramon G. Back to work with achronic dysexecutive syndrome? (a case report). NeuropsycholRehabil 1994;4:399-417.

3. Lezak MD. Assessment of psychosocial dysfunctions resultingfrom head trauma. In: Lezak MD, editor. Assessment of thebehavioural consequences of head trauma. New York: Alan R.Liss; 1989. p 113-43.

4. Lezak MD, Howieson DB, Loring DW. Neuropsychological as-sessment. 4th ed. New York: Oxford University Press; 2004.

5. Luria AR. Higher cortical functions in man. 2nd ed. New York:Basic Books; 1966.

6. Luria AR. The working brain: an introduction to neuropsychol-ogy. New York: Basic Books; 1973.

7. Rainville C, Passini R. Communication, résolution de problème etdémence. In: Michel BF, Verdureau F, Combet P, editors. Com-munication et démence. Marseille: Solal, éditeur; 2005. p 137-59.

8. Burgess PW. Strategy application disorder: the role of the frontallobes in human multitasking. Psychol Res 2000;63:279-88.

9. Le Thiec F, Jokic C, Enot-Joyeux F, Durand M, Lechevalier B,Eustache F. Évaluation écologique des fonctions exécutives chezles traumatisés crâniens graves: pour une meilleure approche duhandicap. Ann Readap Med Phys 1999;42:1-18.

0. Humphreys GW, Forde EM. Disordered action schema and actiondisorganisation syndrome. Cogn Neuropsychol 1998;15:771-811.

1. Zalla T, Plassiart C, Pillon B, Grafman J, Sirigu A. Actionplanning in a virtual context after prefrontal cortex damage. Neu-ropsychologia 2001;39:759-70.

2. Dutil E, Bottari C, Vanier M, Gaudreault C. ADL Profile: descrip-tion of the instrument. 4th ed. Montreal: Les Éditions Émersion;2005.

3. Fisher AG. Assessment of motor and process skills. Volume 1:development, standardization and administration manual. 5th ed.Fort Collins: Three Star Press; 2003.

4. Rogers JC. The spirit of independence: the evolution of a philos-ophy. Am J Occup Ther 1982;36:709-15.

5. Lezak MD. Neuropsychological assessment. 2nd ed. New York:Oxford University Press; 1983.

6. Bottari C, Dutil E, Dassa C, Rainville C. Choosing the mostappropriate environment to evaluate independence in everydayactivities: home or clinic? Aust Occup Ther J 2006;53:98-106.

7. Burgess PW, Alderman N, Forbes C, et al. The case for thedevelopment and use of “ecologically valid” measures of execu-tive function in experimental and clinical neuropsychology. J IntNeuropsychol Soc 2006;12:194-209.

8. Gitlin LN. Conducting research on home environments: lessons

learned and new directions. Gerontologist 2003;43:628-37.

9. Rempfer MV, Hamera EK, Brown CE, Cromwell RL. The rela-tions between cognition and the independent living skill of shop-ping in people with schizophrenia. Psychiatry Res 2003:103-12.

0. Dutil E, Forget A, Vanier M, Gaudreault C. Development of theADL Profile: an evaluation for adults with severe head injury.Occup Ther Health Care 1990;7:7-22.

1. Fougeyrollas P, Noreau L, Bergeron H, Cloutier R, Dion SA,St-Michel G. Social consequences of long term impairments anddisabilities: conceptual approach and assessment of handicap. IntJ Rehabil Res 1998;21:127-41.

2. Klein S, Barlow I, Hollis V. Evaluating ADL measures from anoccupational therapy perspective. Can J Occup Ther 2008;75:69-81.

3. Bottari C. Perception of experts on criteria for the assessment ofthe “dysexecutive syndrome” in everyday activities. Montreal:Université de Montréal; 2001.

4. Gervais N. Comparaison du profil des AVQ et de la mesured’indépendance fonctionnelle: validité de trait. Montréal: Univer-sité de Montréal; 1995.

5. Kasindi G. Comparaison de deux outils servant à évaluerl’autonomie de la personne traumatisée crânio-encéphalique; leProfil des AVQ et l’handicapomètre. Montréal: Université deMontréal; 1998.

6. Rousseau J, Dutil E, Lambert J. Fidélité inter-examinateurs du“Profil des AVQ- Mise en situation” chez la personne traumatiséecranio-cérébrale. Étude de la cote globale. Partie 1. Can J OccupTher 1994;61:149-58.

7. Rousseau J, Dutil E, Lambert J. Fidélité inter-examinateurs du“Profil des AVQ- Mise en situation” chez la personne traumatiséecranio-cérébrale. Étude sur la cote des opérations. Partie II. Can JOccup Ther 1994;61:159-67.

8. Bottari C, Dassa C, Rainville C, Dutil E. The criterion-relatedvalidity of the IADL Profile with measures of executive functions,indices of trauma severity and sociodemographic characteristics.Brain Inj 2009;23:1-14.

9. Bottari C, Dassa C, Rainville C, Dutil E. The factorial validity andinternal consistency of the Instrumental Activities of Daily LivingProfile in individuals with a traumatic brain injury. NeuropsycholRehabil 2009;19:177-207.

0. Bottari C, Dassa C, Rainville C, Dutil E. The IADL Profile:development, content validity, intra and interrater agreement. CanJ Occup Ther 2010;77:90-100.

1. Golden CJ. Stroop Color and Word Test. Wood Dale: StoeltingCo; 1978.

2. Stroop JR. Studies of interference in serial verbal reactions. J ExpPsychol 1935;18:643-62.

3. Shallice T. Specific impairments of planning. Philos Trans R SocLond 1982;298:199-209.

4. Wechsler D. Wechsler Memory Scale—third edition manual. SanAntonio: The Psychological Corp; 1997.

5. Bottari C, Dutil E, Dassa C, Rainville C. The influence of exec-utive function deficits on independence in instrumental activitiesof daily living. Brain Inj 2008;22(Suppl 1):34.

6. Bottari C, Dutil E, Dassa C, Rainville C. Relationship oftraumatic brain injury severity and socio-demographic charac-teristics to independence in everyday activities. Brain Inj 2008;22(Suppl 1):33.

7. Crocker L, Algina J. Introduction to classical and modern testtheory. Belmont: Wadsworth Group/Thomas Learning; 1986.

8. Bain D, Pini G. Pour évaluer vos évaluations: la générasibilité,mode d’emploi. Genève: CRPP; 1996. p 125.

9. Vangeneugden T, Laenen A, Geys H, Renard D, Molenberghs G.Applying concepts of generalizability theory on clinical trial datato investigate sources of variation and their impact on reliability.Biometrics 2005;61:295-304.

0. Cronbach LJ, Glerser GC, Nanda H, Rajaratnam N. The multifacet

concept of observational procedures. The dependability of behav-

Arch Phys Med Rehabil Vol 91, May 2010

5

5

5

5

5

5

5

5

5

6

6

6

6

6

a

b

742 GENERALIZABILITY OF INSTRUMENTAL ACTIVITIES OF DAILY LIVING PROFILE, Bottari

A

ioral measurements: theory of generalizability for scores andprofiles. New York: Wiley; 1972. p 1-33.

1. Pedhazur EJ, Schmelkin LP. Measurements, design, and analysis:an integrated approach. Hillsdale: Lawrence Erlbaum Associates;1991.

2. Teasdale G, Jennett B. Assessment of coma and impaired con-sciousness: a practical scale. Lancet 1974;2:81-4.

3. Donner A, Eliasziw M. Sample size requirements for reliabilitystudies. Stat Med 1987;6:441-8.

4. Bao S, Howard N, Spielholz P, Silverstein B, Polissar N. Interraterreliability of posture observations. Hum Factors 2009;51:292-309.

5. Martin K, Hoover D, Wagoner E, et al. Development and reliabil-ity of an observational gait analysis for children with Downsyndrome. Pediatr Phys Ther 2009;21:261-8.

6. Rabbitt P. Introduction: methodologies and models in the study ofexecutive function. In: Rabbitt P, editor. Methodology of frontaland executive function. Hove: Psychology Press; 1997. p 1-38.

7. Brennan RL. Generalizability theory. New York: Springer-Verlag;2001.

8. Cardinet J, Tourneur Y. Assurer la mesure. Peter Lang:Berne;

1985.

rch Phys Med Rehabil Vol 91, May 2010

9. Nunnally JC. Psychometric theory. 2nd ed. New York: McGrawHill; 1978.

0. Walsh WB, Betz NE. Tests and assessment. 3rd ed. EnglewoodCliffs: Prentice Hall; 1995.

1. Brentnall J, Bundy AC. The concept of reliability in the context ofobservational assessments. Occup Ther J Res 2009;29:63-71.

2. Knight C, Alderman N, Burgess PW. Development of a simplifiedversion of the multiple errands test for use in hospital settings.Neuropsychol Rehabil 2002;12:231-56.

3. Ponsford J, Sloan S, Snow P. Traumatic brain injury: rehabilita-tion for everyday adaptive living. Hillsdale: Lawrence ErlbaumAssociates; 1995.

4. Schwartz MF, Segal M, Veramonti T, Feraro M, Buxbaum LJ.The Naturalistic Action Test: a standardised assessment foreveryday action impairment. Neuropsychol Rehabil 2002;12:311-39.

Suppliers. EDUG: Institut de recherche et de documentation pédagogique, 43

FB, de l’Hôpital, CP 556 CH-2002, Neuchâtel, Switzerland.. Version 15.0; SPSS Inc, 233 S Wacker Dr, 11th Fl, Chicago, IL

60606.