Copyright © 2020 pubrica. All rights reserved 1

How to Develop and Evaluate a PRO Instrument?

Dr. Nancy Agens, Head,

Technical Operations, Pubrica

sales@pubrica.com

I. WHAT IS PRO?

PRO or Patient-reported outcome

is a relatively novel method to evaluate the

outcome of any treatment by clinician or

researcher (in clinical trial) purely based

on the inputs got from the patient alone.

The United States Food and Drug

Administration (US-FDA) defines PRO as

‘any report of the patient’s health

condition that comes directly from the

patient, without interpretation of the

patient’s response by a clinician or anyone

else.’

II. ASK THE DOCTOR, WHY

ASK THE PATIENT?

Objective data regarding outcome

can be given by the treating physician

(cure/survival/disability limitation), by

investigations (return of deranged

biochemistry to normal) and by the

caregiver (return to functional normality).

These are Observer-reported outcomes

(OROs) and medicine has long relied on

these for evidence-based management.

Any illness followed by its

treatment weighs physically, mentally,

emotionally, economically on the patient.

Patients perceive the outcome of the

treatment in comparison to their

expectations based on prior knowledge.

Patient response and compliance to

treatment are intricately linked to the

ultimate treatment outcome.

Since long, clinicians have

recorded patient symptoms in the daily

clinical notes and modified their treatment

accordingly. PRO goes beyond just

recording symptoms. Central to this is the

concept of Health-related Quality of Life

(HRQOL), a multi-dimensional approach

that records patient’s perceived effect of

the illness and treatment on quality of life

in his/her physical, mental, emotional,

economical, and social space. By

standardizing the questions and

quantitating them as far as possible, data

can be analysed, interpreted, and

compared. PRO data is used to inform and

guide patient-centered care, to assess risk

and benefit from treatment, to decide on

treatment protocols and to guide health

policy makers.

III. WHAT INFORMATION DO I

WANT FROM THE PATIENT? –

THE PRO INSTRUMENT

The outcome of interest that we

want to study is the PRO Concept, while

the PRO instrument is the means to collect

the data related to the PRO concept, in the

form of questionnaire and the information

and documentation that supports its use.

In other words, PRO instrument is

a questionnaire that compiles data related

to the PRO concept of interest which can

be summarized as a score.

Content validity is the extent to

which an instrument measures the concept

of interest that is desired from both the

developer’s and user’s perspective.

IV. DESIGNING A VALID PRO

INSTRUMENT

Steps involved are: -

Copyright © 2020 pubrica. All rights reserved 2

1. Defining the concept – concept should

be clearly defined based on review and

literature gap analysis of existing PRO

instruments as well as taking inputs

from scientific research services,

domain specialists, developers and

most importantly the patients or the

users and should focus on the core

characteristics.

2. Conceptual Framework – Breakdown

the concept to sub-concepts and finally

to items or the questions that will be

included in the questionnaire. Each

item should be crisp and clear with

focus on only one aspect. It should not

unnecessarily cause strain to the

patient in recalling.

3. End-point Model– Endpoint is the final

measured outcome of the treatment

that will be compared, and an endpoint

model summarises all the endpoint

based on OROs and PROs.

4. Review of Preliminary Instrument –

Literacy review is done to understand

the proportion of people who may or

may not have the literacy level to take

the questionnaire. Translation review is

done to ensure conceptual equivalence

when translated to another language.

Cognitive interview is the process

where the user is made to take the

questionnaire by thinking aloud to

understand the approach of the user to

the questions and that it follows the

intention of the item. This is done until

saturation is reached on the

information received for reviewing the

instrument.

5. Reliability Testing – Measure to

estimate reproducibility and consistent

true estimation of outcomes under

Copyright © 2020 pubrica. All rights reserved 3

consideration. It is further described

below.

6. Validation –

It is an ongoing never-

ending process of improving the

instrument by repeated feedback

mechanism.

V. EVALUATING PRO

INSTRUMENT

As clinical validation data

accumulates, a PRO instrument undergoes

review and revision to make them robust.

It includes the evaluating for the

following: -

1. Validity– is the extent to which the

scores and their interpretations,

supported by theoretical evidence,

correlate with the proposed use of the

instrument. It needs to be tested for the

entire range of the proposed

application.

a. Content Validity- It is a measure of

how much the instrument measures the

concept of interest. US-FDA proposes

using qualitative and quantitative

methods.

Qualitative estimation – include

items involved, their recall period,

response options, scoring pattern,

method of conducting the

questionnaire etc.

Quantitative estimation – e.g.

Item response theory (IRT) – A

powerful tool to assess whether

the items used measure the

entire spectrum or continuum

of the patient responses.

Rasch analysis - Only IRT tool

to use the total of all the items

in the PRO instrument making

it a simple model.

b. Construct Validity- It is based on

comparison with convergent and

divergent groups with known validity

and showing that the instrument under

study conforms to similar

interpretations.

c. Criterion Validity- Comparison with

known criteria or known gold standard

for this concept.

2. Reliability-

a. Reproducibility or test-retest

reliability- It is assessed under

same conditions in same user in

a close span of time to assess

any variations in recording

results that can be attributed to

the instrument. It is calculated

by intra-class correlation

coefficient (ICC).

b. Internal consistency- The items

should deliver same scores in

similar situations consistently.

It is determined by Cronbach’s

alpha or coefficient of

reliability.

c. Floor and ceiling effect – i.e.

the instrument’s ability to

measure extremes of construct

helps choose the proper

instrument for the study.

d. Inter-interviewer reliability-

there should not be significant

difference in scores when

interviewed by two different

interviewers.

e. Variability –

Instrument should have ability

to detect changes when

outcomes are different, else it is

no good.

Adhering to good practices while

designing and validating a PRO instrument

is primary for it to be effective and receive

regulatory approvals. Readiness to

continuously revisit and revise items is

needed to do better justice to the concept

of interest in the long run.

REFERENCES:

1. FDA Guidance for Industry. Patient-Reported

Outcome Measures: Use in Medical Product

Development to Support Labeling Claims. 2009.

Available at:

http://www.fda.gov/downloads/Drugs/Guidances/U

CM193282.pdf.

2. Patrick DL, Burke LB, Gwaltney CJ, et al. Content

validity--establishing and reporting the evidence in

Copyright © 2020 pubrica. All rights reserved 3

newly developed patient-reported outcomes (PRO)

instruments for medical product evaluation: ISPOR

PRO good research practices task force report: part

1--eliciting concepts for a new PRO

instrument. Value Health. 2011;14(8):967‐977.

doi:10.1016/j.jval.2011.06.014.

3. Rothrock, N & Kaiser, Karen & Cella, David.

(2011). Developing a Valid Patient-Reported

Outcome Measure. Clinical pharmacology and

therapeutics. 90. 737-42. 10.1038/clpt.2011.195.

4. Deshpande, P. R., Rajan, S., Sudeepthi, B. L., &

Abdul Nazir, C. P. (2011). Patient-reported

outcomes: A new era in clinical

research. Perspectives in clinical research, 2(4),

137–144. https://doi.org/10.4103/2229-3485.86879