Service User Involvement: How do we know it's a reality?

Service User Involvement: How do

we know it’s a reality?

Clive Willis

MBA in Health Services Management

Greenwich School of Management

Dissertation

14th Fedruary 2011

Greenwich School of Management

Dissertation: Master of Business Administration in Health Services

Management

Title: Service User Involvement: How do we know it’s a reality?

Student: Clive Willis

Student Ref: H52017PE

Submission date: 14th February 2011

Dissertation Supervisor: David Schofield

Content: Page

Abstract 1

Chapter 1 Introduction 1

Chapter 2 Literature Review

2.1 Service User Involvement (SUI) in Substance Misuse Delivery 8

2.2 Service User Involvement (SUI) in Health & Social Care 10

2.3 Evaluating User Involvement 12

2.4 Summary 17

Chapter 3 Methodology 18

Chapter 4 Data Collection 24

Chapter 5 Findings & Analysis 27

5.1 Drug Action Team (DAT) 1, Semi-structured Interviews 28

5.1.1 Efficacy of the Tool to Support, Plan and Monitor User Involvement 35

5.1.2 Amendments to Process and/or Assessment Tool 39

5.2 Drug Action Team (DAT) 2, Semi-structured Interviews 41

5.2.1 Efficacy of the Tool to Support, Plan and Monitor User Involvement 48

5.2.2 Workshop 51

Chapter 6 Conclusions 57

Chapter 7 Recommendations 60

References 62

1

Service User Involvement: How do we know it’s a reality?

Abstract

Service User Involvement (SUI) is a requirement in the planning, commissioning and

delivery of health and social care services and leads to improved choices and control for

people who use them. However, the ability to both measure and monitor this activity at a

national level has continued to elude us.

This study seeks to support the future planning, delivery and monitoring of SUI through the

development and piloting of an assessment tool for local drug partnerships (or equivalent

structure i.e. Drug Action Teams (DAT)). It highlights how the identification of baseline

activity of SUI in planning and development processes provides a platform from which these

partnerships can better understand and embed user participation into decision making. It is

only by understanding the why, where and when of user involvement, that we can appreciate

and comprehend the HOW.

Chapter 1. Introduction

Total estimates for problematic drug use (PDU1, opiate and/or crack cocaine use) in England

are around 350,000 across the whole population; this number does not account for other drug

use such as cannabis, ecstasy, benzodiazepines, etc. The complexity of need of these

1 Problem drug use (PDU) is defined as ‘injecting drug use or long-duration/regular use of opioids, cocaine and/or amphetamines’.

2

individuals presents a significant challenge nationally, regionally and locally in terms of

engagement and treatment. Current policy and legislation (Department of Health: The

National Health Services Act, 2006a; Our health, our care our say, 2006b; NHS Constitution,

2010) places a requirement on organisations providing or commissioning NHS or local

authority care to ensure that the local community and service users are involved in the

planning, development and decision-making of their services. Guidance from the National

Treatment Agency (NTA2) reflects these requirements in the commissioning and provision of

substance misuse services (NTA, 2006, 2009, 2010). However, there is little evidence to

support the view that SUI is sufficiently integrated into policy and planning across health and

social care, or specifically within substance misuse provision (Carr, 2004; Patterson et al,

2007; DH, 2008a).

Whilst community participation has been a key principle of NHS delivery since its inception

in 1948, the ability to performance assure organisations and services against this activity has

continued to prove difficult. Some of the difficulties associated with evaluating performance

have been how participation is defined; the absence of validated and jointly owned

frameworks or models for measurement and the consistency of outputs being measured.

Within the substance misuse field this is exacerbated by the fact that the cohort of individuals

affected by substance misuse is variable in terms of the substance used, the level of usage

(from recreational use to PDU) and demographics. This variability requires a multitude of

approaches in ensuring effective engagement when assessing the need of a community and

future planning.

2 A Special Health Authority whose role is to performance assure delivery of local drug partnerships (or equivalent i.e. Drug Action Teams)

3

Although guidance on how and when to involve users is abundant (GLADA, 2005; DH,

2008b; Welsh Assembly, 2008; Jürgens 2008), there is little evidence to support the

effectiveness of involvement methods (Nicholson, 2005). In addition, the evaluation models

currently being used lack the ability, either due to the limitations of the model or through

their implementation, to be replicated across different programs (Rowe & Frewer, 2004).

Existing evaluations have also been limited to specific intervention programs or

organisations. Consequently, it is not currently possible to develop a consistent understanding

of SUI at a national level. In order to inform national guidance to support the ongoing

development and integration of user participation, we first need to establish a baseline of

activity in relation to process (Rifkin et al, 1988; Tritter & McCallum, 2006), from which

organisations can begin to explore what works best in terms of engagement in a given

locality.

This is not suggesting that there is not some excellent and innovative service user

engagement going on, but the inability to use comparative analysis across organisations

impedes benchmarking and future development through research and national guidance.

This study will seek to support future planning, delivery and monitoring of SUI through the

development and piloting of an assessment tool for local drug partnerships. This will be

limited to establishing the extent to which user involvement is embedded into these activities,

rather than the success of user involvement. The way in which users are engaged to

participate in the activity being evaluated is a separate issue and has purposely not been

included on the basis that what works in any given locality is dependent on the demographics

of the community (or subgroup) seeking to be served and the outcomes being sought. It is

envisaged that the identification of baseline activity in policy and planning will provide a

4

platform from which organisations can better understand and embed user participation into

decision making, which in turn will support and drive future evaluation of engagement

methods or tools at a local level.

The objectives of the study are to:

1. Evaluate how accurately the assessment is able to identify baseline activity.

Responses to the assessment questions will be analysed and then reviewed in a focus

group to measure individual perception of activity against actual delivery (Delphi

technique, Fiander & Burns, 2000).

2. Evaluate how the assessment process informs future delivery.

The assessment tools ability to identify deficits in user involvement and subsequent

action planning (Robinson, 2004) will be used as a measurement of how processes

inform future delivery.

3. Evaluate the efficacy of the tool to support, plan and monitor user involvement.

This will be undertaken through participant questionnaires which will capture how

they perceived the tool’s ease-of-use and ability to support planning and monitoring

of user involvement.

5

An adaptation of the model proposed by Rifkin et al (1988) was developed to map the current

level of user participation within local drug partnerships. The model focuses on key factors of

delivery within an organisation and where SUI is encompassed within them. This enables an

understanding of where within the organisation, user participation is predominant.

Adaptations of the model centred on the factors originally stated by Rifkin et al (1988) and

their relevance to local drug partnerships.

The five factors being measured in this study were modified to the following:

• Needs Assessment – how the engagement and participation of service users inform

and support the identification and prioritisation of ‘need’ locally.

• Leadership – the influence of Leadership on engagement and implementation of user

participation.

• Programme/Policy Development – the ability of service users to inform both the

context and content of a new initiative or programme.

• Programme/Policy Implementation – the ability of service users to inform and

support programme implementation.

• Resource Mobilisation – the organisations ability to effectively utilise user

participation.

For each factor stated above a continuum was developed with wide SUI (users plan;

implement and evaluate activity, with professionals used as a resource) at one end and narrow

SUI (professionals plan; implement and evaluate activity, with no SUI) at the other. The

continuum was then divided into a series of points. By placing a mark at the point that most

describes activity in the local drug partnership, process indicators were defined for SUI in the

local drug partnership as the width of SUI on the continuum of each factor. Placing a mark on

6

the continuum, and connecting those marks in a spoke arrangement, provided the base for

where SUI is most narrow.

In order to plot the marks on the continuum, it is necessary to establish a consensus as to

where the organisation sees its activities for each factor in relation to SUI. The Delphi

technique (Fiander & Burns, 2000) was used to collect this information as it allowed for

individual views and/or perceptions to be captured through questionnaires, alongside a group

process via a workshop to establish consensus on where SUI should be placed on the

continuum for each factor. An assessment tool questionnaire was developed to capture

individual team member perceptions on where SUI should be placed on the continuum for

each factor.

Questionnaires were completed as part of a semi structured interview. The use of semi-

structured interviews allowed for the interviewer to note any difficulties experienced by the

respondent in completing the questionnaire and ask additional questions to ascertain the

tool’s ease-of-use and ability to support planning and monitoring of user involvement.

Data from the completed assessment tool questionnaires were collated and analysed to

produce an initial outline of perceived activity and feedback to the team in a workshop. The

initial outline was used as the basis for discussion of current activity to inform a final

decision by the group on where marks should be placed on the continuum for each factor.

This enabled staff to evaluate activity based on a collective understanding rather than

individual perceptions. In addition, this process allowed for a wider discussion on gaps in

delivery to inform action planning.

7

The study has shown that by focusing on organisational processes and where users’ are

involved in its planning, development and decision-making processes, we are able to:

a) recognise gaps in delivery and identify actions to improve and/or sustain user

involvement

b) support individuals and teams to understand and consider activity to support effective

SUI

c) effectively plan and monitor SUI

However, further testing of the assessment tool should be undertaken in organisations with a

range of user involvement experience and organisational support for user involvement. In

addition, wider testing of the assessment tool should also consider the independence of the

facilitator (interviewer) and co-facilitation by a representative of the local user group.

Key to any further evaluation of the assessment tools efficacy is the support and/or

involvement of those with responsibility for the oversight and governance of the

organisations (local drug partnerships, or equivalent i.e. DAT) being assessed.

Chapter 2

Literature Review

8

Chapter 2. Literature Review

2.1 Service User Involvement (SUI) in Substance Misuse Delivery

SUI in health and social care is today commonplace and perceived as much as a part of

normal business as the delivery of the services themselves. However, SUI within the

substance misuse field has been slower to develop, largely as a result of the myriad of

government departments and organisations tasked with responding to this multi dimensional

agenda. The NTA was created in 2001 as part of the Government’s overall Drug Strategy

(Home Office, 1998) and presented the first opportunity to develop user involvement in a

consistent and meaningful manner. As a consequence, few studies to-date have attempted to

provide insight into SUI in substance misuse provision at a national level, given the infancy

of its development as a national platform.

The two most referenced studies were considered, each focusing on distinct areas of user

involvement. The Department of Psychological Medicine, Imperial College London study

(Patterson et al, 2007) explores user involvement in planning, commissioning and delivery of

drug treatment services, whereas, the Joseph Rowntree Foundation study (Fischer et al, 2007)

explores SUI in treatment decisions. It is important to note the difference in focus of these

studies as this highlights an important question in relation to which studies are pertinent to

this literature review. Whilst user involvement in making treatment decisions is integral to the

wider SUI agenda i.e. personalisation and individual budgets (Department of Health, 2008a),

those decisions are limited within the context of what services are available locally. This

literature review will predominantly focus on SUI at a policy and planning level, therefore

9

impacting the availability of choice, and draw from wider studies (e.g. Fischer et al 2007)

where a reference to policy-making and planning is made.

Patterson et al (2007) surveyed service commissioners, providers and users in a

representative 50 of the 149 local Drug Action Teams (DAT) and conducted in-depth case

studies in six DAT areas. Their findings concluded that there was significant variability in the

level, type and quality of SUI nationally. Key findings included:

• “wide variation in the degree to which UI systems and structures had been

established.

• disparity between and within DATs in terms of models and mechanisms used and the

degree to which user involvement was integrated in planning, commissioning and

development of services.

• fewer than half (47%) had formal user representation at commissioning level.

• findings indicated a lack of strategic UI planning in many DATs.

• processes to enable user input to contribute systematically to commissioning and

service level development are often not well defined.

• a significant minority of DATs (n=16; 36%) evidenced no user involvement at

strategic levels with fragmented and inconsistent UI within services.” (Patterson et

al, 2007, p2)

In addition, Patterson et al (2007, p1) found only 22% of DATs surveyed had “user

involvement workers”. This might be seen as a missed opportunity by DATs given Fischer et

al (2007, p1) noted “user involvement co-ordinators” in DATs as a positive development.

10

The Patterson et al study (2007), although surveying a broad spectrum of participants, was

limited in the breadth of partnerships it included and raises questions on whether national

comparisons can be made. However, a recent report on how well local drug partnerships

planned and commissioned services for diverse populations made the following statements as

priorities for action.

• “Commissioners should ensure that all contracts include requirements for

compliance with all relevant equality and diversity legislation.

• Local drug partnerships should focus on ensuring that services are accessible and

relevant to crack users, to increase the proportion accessing services.

• Service providers and local drug partnerships should consult more broadly with the

communities they service, including with diverse groups and communities

(particularly those not currently accessing drug treatment) and involve them in

service planning.” (Commission for Healthcare Audit and Inspection, 2009, p8)

These indicate wider consultation and involvement is required and goes to support some of

the findings of Patterson et al (2007).

2.2 Service User Involvement (SUI) in Health & Social Care

Studies on SUI tend to fall into the following categories: those that evaluate methods or tools

in engaging service users; those that evaluate the usefulness of SUI, and those that evaluate

the evaluations. Few however attempt to clarify a national position of existing levels of

participation or involvement. The Department of Health (DH) acknowledges the lack of

evidence to support a view that SUI is embedded in NHS day-to-day delivery and goes as far

as to state:

11

“There is scant evidence to show that involvement activity is stitched into all the strands of

NHS organisations’ work, including their decision-making processes......” (DH, 2008a, p10)

There are numerous guidance documents available on service user involvement such as: Real

Involvement (DH, 2008b), User Involvement Framework (Welsh Assembly, 2008) and

Lessons Learnt (GLADA 2005). The guidance addresses both the principles and legislation

upon which SUI is built, alongside methods and tools of engagement and/or involvement.

However, there does not appear to be research on the extent to which these methods or tools

are used across the NHS. Nevertheless, Young (2006) does explore SUI and its influence

across both NHS and Social Care. The study considers government strategy, in terms of SUI,

and its impact on those whose treatment and support crosses the boundaries of the two

distinct organisational cultures (NHS & Social Care).

Previous studies being considered by Young (2006) include those that comment on the

implementation and flexibility of section 31 agreements, partnership working, and the NHS

changing workforce programme. Two key lessons learnt from this study were firstly that

consultation was more likely to take place “ex-post” i.e. service users being asked to confirm

or critique decisions already made, as opposed to “ex-ante” and influencing thinking before

the decision is made (p265). The second lesson was that "real involvement takes a long time

to achieve" (p266). In addition, Speak Up (2007) identified that the most used forms of user

participation methods by Local Authorities were focus groups (84%), postal questionnaires

(79%), whereas the use of service user advocates were used less (35%) followed by peer

reviews (25%).

A review of the literature available found no empirical evidence to suggest that SUI in the

12

planning or delivery of substance misuse services is, to a greater or lesser extent, a mirror of

SUI across the broader spectrum of NHS and Social Care delivery.

2.3 Evaluating User Involvement

The most commonly occurring method of measuring the meaningfulness of user involvement

methods or tools (Robinson, 2004; GLADA, 2005; Welsh Assembly, 2008; DH, 2008b;

Participation Works, 2008; McLaughlin, 2009), as opposed to successful outcomes,

incorporate adaptations to the Ladder of Participation (Arnstein, 1969). A visual

interpretation of citizen participation is provided by Arnstein within eight rungs on the ladder

of citizen participation, ranging from manipulation (rung 1) to citizen power (rung eight). The

rungs in between identify varying layers of participation that move through tokenism to

empowerment. This method of measurement would see the most used participation methods

referred to previously (Speak Up, 2007) on the lower rung of the ladder and regarded as

tokenistic. In general, studies utilising Arnstein’s model note the limitations of its approach

i.e. its view that participation is hierarchical. Nonetheless despite this view the approach is

not often modified to remove this hierarchical structure.

Arnstein’s study was undertaken against the backdrop of political imperatives and debates in

relation to tokenism of participation and the power or influence participation brings to the

governed. Arnstein emphasises the way in which consultation can be frustrating for those

being consulted unless it includes the real ability for the powerless to exert power. Whilst

Arnstein recognises the simplistic interpretation of a complex interaction between those that

govern and those that are governed, the baseline of this approach appears to accept an

underlying principle that those that seek to consult are unwilling or unable to inform

13

decisions based on this interaction i.e. tokenistic engagement. In addition the rungs are

hierarchical and place a distinct value on each rung and therefore are unable to appreciate the

limitations sometimes incumbent in user participation - the focus being the distribution of

power in decision making rather than the process of engagement. Tritter & McCallum, (2006,

p163) make the comment that: "it is the process rather than the outcome that has the greatest

potential for changing organisational culture”. Arnstein’s approach needs to be viewed

within the context of the situation or issues being discussed and there are still similarities

today in terms of the perception of those that see themselves as the powerless. We therefore

need to consider the meaningfulness of participation to the participant (citizen, user or

consumer) and the level to which participation empowers the individual or group, as opposed

to organisational judgements and/or perception of what is meaningful.

In reviewing Arnstein's study in today’s context, consideration should be given to the

evolution of clinical governance in the delivery of health and social care, which has seen

greater accountability for decision making. It could be argued that what may have been

previously seen as tokenism is now an integral part of decision-making and provides a clear

platform for challenge by those that seek to be heard. Available studies on SUI range in their

findings on the emphasis that power in decision-making should be given to service users.

Some of these studies see Arnstein's model still relevant today (Welsh Assembly, 2008),

some seeking to move beyond Arnstein (Tritter & McCallum, 2006) and those that seek to

embrace both sides of the debate (Participation Works, 2008).

What is clear is the need to recognise the availability of choice regarding how the individual

or group wishes to participate. Additionally, there needs to be recognition of who is being

sought to be involved, alongside their desire to be involved. Given the diverse populations (or

14

groups of individuals) being sought for participation by equally diverse agendas, it is

unrealistic to envisage equal levels of involvement or participation across all agendas. In

relation to substance misuse, it is reasonable to argue that due to the nature of drug use and a

desire by the user to remain unnoticed by the larger community, involvement in decision-

making will continue to be problematic and require innovative ways to support and engage

the user's voice in decision-making. As Lessons Learned (GLADA, 2005, pvii) notes:

"Drug users are a diverse group with different needs, expectations and skills. Therefore it’s

unlikely that there is a single approach that would support them to be involved and

enthusiastic about participating more fully in public life."

Studies concerning the usefulness of SUI or participation will continue to provide a positive

(and essential) forum to question and evolve the who, how and why we seek to ensure SUI in

decision-making. However, the reality today is that organisations are required by legislation

(DH, 2006a) to ensure user involvement in decision-making and will continue to be measured

on how effectively this activity is undertaken. SUI and participation in the delivery of

services to vulnerable, excluded or marginalised groups, is often considered to provide

additional benefits beyond that of organisational delivery for example: developing non-

cognitive skills such as perseverance, motivation, self-esteem, verbal and interpersonal skills.

Fischer et al (2007, p36) concluded that:

“...user involvement and retention in treatment can go hand in hand and, by facilitating the

former, agencies have the potential to increase the latter.”

15

Additionally, Listen and Change (Participation Works, 2008, p20), in referring to the added

benefits of user involvement in children’s services, noted that, “if this participation were

tokenistic....... there would be fewer benefits and possibly even some negative

consequences”.

It is important to consider the broad spectrum of studies evaluating user involvement or

participation to understand or gauge the validity of the findings being made. Rowe & Frewer

(2004) consider some of the complexities of defining success, highlighting that participation

is never a clear measurement between A and B (speed or distance). Instead it requires the

understanding of the nature of the participation (e.g. open public consultation, questionnaires,

formal meeting of representatives) and the variables contained within each type of

participation and the outcome produced. An additional point to consider is whether success is

measured purely by the fact that a decision was made, or whether it is measured through

positive outcomes that emerge from the actual decision itself. Also, in considering the variety

of models or techniques that might be utilised in any one area when engaging the public in

consultation or decision-making processes, can the criteria for defining success be applied to

all?

Individual areas, i.e. local drug partnerships, may be seeking different outcomes via the same

or different levels of engagement or consultation. One would also need to consider the

attributes of those they seek to engage and their ability to participate, for example substance

misusers who are positively engaged in treatment versus a transient cohort of substance

misusers not engaged in treatment. The levels of, and category for, successful engagement

within these cohorts would therefore need to be measured differently. Rowe & Frewer (2004)

were unable to establish consistency among previously conducted evaluations on

16

participation. The study went as far as to state that previous research had been disorganised

and sporadic in nature and as such does not provide a platform from which ongoing

evaluation of participation could take place. In addition the lack of detailed evidence of

methodology used meant that few, if any, of these evaluations could be duplicated.

Additionally, Nicholson (2005, p52) noted that: “a greater understanding of what

participation activity works best in what context is curtailed by the lack of a body of robust,

empirical evidence on which to draw conclusions.”

Rifkin et al (1988), in considering measurements of equity and participation in community

development projects, considered whether an analytical framework could be developed to

assess participation. In reviewing existing studies that sought to evaluate either outcomes

derived from participation and/or levels of participation reached, they concluded that: “the

development of process indicators is critical to the understanding of health improvements

and community participation” (Rifkin et al, 1988, p933). Their conclusion being based on

similar arguments already noted previously in this paper such as: limitations of existing

methods in terms of replicating, and/or, the ability to account for nuances in social, cultural or

participant determinants. Rifkin et al propose a methodology to define indicators for

participation in health care programs as to how wide participation is on a continuum

developed for five factors: Needs Assessment, Leadership, Organisation, Resource

Mobilisation, and Management.

The methodology used enabled the evaluator to measure levels of participation within a

program or organisation rather than the effectiveness of participation. The premise here is

that community participation works, rather than establishing whether it works. In addition

there are no judgements as to which of the five factors is more important, but instead

17

recognises that participation may vary at any given time across these factors. The assertion

being made is that this methodology enables a programme or organisation to better

understand where levels of community participation are concentrated and where necessary,

undertake an evaluation at distinct points of engagement.

2.4 Summary

Whilst user involvement in health and social care is often considered as commonplace and

guidance on how and when to involve users is abundant, there is little evidence to support the

effectiveness of involvement methods, or, that user involvement is sufficiently integrated into

policy and planning. In addition the evaluation models currently being used lack the ability,

either due to the limitations of the model or through their implementation, to be replicated

across different programs. Existing evaluations have also been limited to specific intervention

programs or organisations. Consequently, it is not currently possible to develop a consistent

understanding of SUI at a national level. In order to inform national guidance to support the

ongoing development and integration of user participation, we first need to establish a

baseline of activity in relation to process, from which organisations can begin to explore what

works best in terms of engagement in a given locality.

Chapter 3

Methodology

18

Chapter 3. Methodology

An adaptation of the model proposed by Rifkin et al (1988) was developed to map the current

level of user participation within local drug partnerships. This model allows us to understand

where within the organisation, user participation is predominant. This is not to say that any

one function within the organisation is best served by user participation, as this may be

variable at any given time, but to better understand where it occurs.

Rifkin et al (1988, p933) defined factors impacting on participation in Primary Health Care

(PHC) as:

• “Needs Assessment

• Leadership

• Organisation

• Management

• Resource Mobilisation, and

• Focus on the Poor.”

However, Focus on the Poor was not included in the final matrix as it was not able to be

plotted on a continuum. Additionally, its inclusion might have been seen as limiting the

framework’s wider use i.e. programmes targeting specific community groups.

Rifkin (1990, p10) states:

“...participation can best be defined by asking questions about its concrete components.”

19

Therefore, adaptations of the model centred on the factors above and their relevance to local

drug partnerships. In addition, consideration was given to the context within which the

factors were developed i.e. PHC implementation within a community (in its widest sense). In

considering Rifkin’s descriptions of organisation and management, that can be summarised

as:

• Organisation created by the community (either in existence or subsequently created)

designed to improve health services.

• Management refers to programme management and the inclusion of community

leaders within its framework.

We begin to question their relevance to local drug partnerships.

In the delivery of substance misuse provision there has been a significant increase in recent

years of services being either created by, or, delivered in partnership with users. However,

these tend to have been developed to address specific treatment requirements or expression of

views (user groups) within a multidiscipline approach delivered as part of the National Drug

Strategy (Home Office, 2008). Therefore it would be difficult to argue that they represent or

address the needs of the whole user community. In addition, it might be considered that the

ability to create community organisations within the substance misuse field is impacted upon

by the inherent difficulties experienced by such a diverse group with varying needs,

expectations and skills (GLADA, 2005).

20

The ability to categorise community in the context of substance misuse has not featured in

existing research and would be a key addition to the further understanding of SUI in local

drug partnerships. Additionally, those that represent the user view, tend to be individuals

(service users) who have either stabilised, for example through substitute prescribing, and/or

abstinent. This raises additional questions as to when those representatives can still be

considered to represent a community that they themselves are no longer a part of, i.e. a

community that encompasses those that are actively using and/or those actively engaged in

treatment. Whilst some may consider abstinence (recovery) as part of the treatment journey,

the needs, expectations and skills of those individuals shift significantly over time. This shift,

alongside changes in the populations’ drug use and patterns over relatively short periods of

time, would indicate a gap between the community and those that seek to represent it. The

latter becoming a professional advocate over time, as opposed to a member of the community

they seek to serve.

To address the points raised above, the two factors (Organisation and Management) were

replaced by the following:

• Programme/Policy Development: participation of service users in the development

of a programme or policy ensures that the programme or policy has relevance to, and

ownership from, those that it seeks to support – therefore having a greater chance of

success.

• Programme/Policy Implementation: this can often falter because those that it is

intended for or delivered by are not provided with contextual information in a way

that is understandable to them and/or is seen to favour someone else’s agenda –

21

leading to resistance. User participation across all aspects of implementation can

significantly affect the way in which a programme is received by others.

The inclusion of these two factors provides an increased focus on process relevant to local

drug partnership activity (DH, 2006a). And in turn, encompasses components of

‘Organisation’ and ‘Management’, in that SUI in these factors ensures their inclusion within

the organisational decision-making structure and presents a visible presence to the

community.

In further considering the wider impact of removing management as a factor, consideration

was also given to the role of Leadership. Rifkin (1990) viewed Leadership in relation to those

within the community that either through political or natural selection, influenced or

determined the implementation of an initiative. The leader(s) was seen by Rifkin to have a

direct impact on participation residing within the elite of a community or broadening out to

the participation of the wider community. Given the structure of local drug partnerships and

their relatively small staff group (4 – 12), and the points raised above relating to the

substance misuse community, Leadership in this instance was seen to have more significance

by measuring the influence of those who lead within the organisation (local drug partnership),

rather than leaders within the community. This view is in part supported by Patterson et al

(2003), whose findings included leaders (Chief Executive Officers) having significant

influence on the decision-making process of the teams they lead.

22

The factors being measured in this study were subsequently modified to the following:

• Needs Assessment – how the engagement and participation of service users inform

and support the identification and prioritisation of ‘need’ locally.

• Leadership – the influence of Leadership on engagement and implementation of user

participation.

• Programme/Policy Development – the ability of service users to inform both the

context and content of a new initiative or programme.

• Programme/Policy Implementation – the ability of service users to inform and support

programme implementation.

• Resource Mobilisation – the organisations ability to effectively utilise user

participation.

For each factor stated above we can develop a continuum with wide SUI (users plan;

implement and evaluate activity, with professionals used as a resource) at one end and narrow

SUI (professionals plan; implement and evaluate activity, with no SUI) at the other. The

continuum is then divided into a series of points. By placing a mark at the point that most

describes activity in the local drug partnership, we can define process indicators for SUI in

local drug partnerships as the width of SUI on the continuum of each factor. When a mark is

placed on the continuum, those marks can be connected in a spoke arrangement (Figure 1,

below), which provides the base for where SUI is most narrow (outlined in blue). The first

point on the continuum is however not where the spokes connect as it is recognised that

wherever an organisation interacts with a community, there will always be some degree to

which service users will influence delivery. In the example provided (Figure 1, below) we

can see that by placing a mark on the continuum for each factor, which denotes the activity of

23

the organisation being assessed, provides a baseline of activity. This baseline can be used at a

later date, or by other assessors, for a comparative assessment (Rifkin, 1988).

Figure 1: Radar graph

Chapter 4

Data Collection

24

Chapter 4. Data collection

In order to plot the marks on the continuum, it is necessary to establish a consensus as to

where the organisation sees its activities for each factor in relation to SUI. The Delphi

technique (Fiander & Burns, 2000) was used to collect this information as it allowed for

individual views and/or perceptions to be captured through questionnaires, alongside a group

process via a workshop to establish consensus on where SUI should be placed on the

continuum for each factor. An assessment tool questionnaire was developed (Appendix 1 -

Draft Assessment Tool Questionnaire) to capture individual team member perceptions on

where SUI should be placed on the continuum for each factor. The questionnaire provided a

series of questions for each factor, which capture activities mirrored in national guidance and

best practice. These questions were designed to provide an overview to the respondent on

what SUI activity might look like in practice and therefore help them visualise and compare

the activity of the organisation. In addition, this is followed by a brief statement against each

mark (level) on the continuum allowing the respondent to be clear about the statement being

made when selecting where to place the organisational activity.

DAT teams and/or commissioning teams, acting on behalf of the wider partnership, are

relatively small (4-12 staff members) and therefore enabled the questionnaires to be

completed as part of a semi structured interview. The use of semi-structured interviews

allowed for the interviewer to note any difficulties experienced by the respondent in

completing the questionnaire and ask additional questions (Appendix 2 - Assessment Tool

Evaluation Questionnaire) to ascertain the tool’s ease-of-use and ability to support planning

and monitoring of user involvement.

25

The selection of interviewees was made by the participating organisation and the criteria for

selection were that interviewees represented a range of grades with varying responsibility and

experience within the factors being assessed. In addition to staff interviews, a member of the

local user group was also included in the interview process. It is important to note here that

the measurement being undertaken is of where SUI is predominant across the factors

identified and not how well the organisation engages service users. As such, the inclusion of

a member of the service user group was not to illicit their experience of being engaged but to

ascertain when (and if) they are involved in planning, development and decision-making, and

at what stage within the factors being assessed.

Data from the completed assessment tool questionnaires were collated and analysed to

produce an initial outline of perceived activity and feedback to the team in a workshop. The

initial outline, detailed in an interim report, was used as the basis for discussion of current

activity to inform a final decision by the group on where marks should be placed on the

continuum for each factor. The further questioning of scores in the workshop recognised that

at times activity or policies exist but may not have been effectively communicated throughout

the organisation. This enabled staff to evaluate activity based on a collective understanding

rather than individual perceptions. In addition, this process allowed for a wider discussion on

gaps in delivery to inform action planning.

The process described above was undertaken in two organisations, DAT 1 and DAT 2.

However, the process undertaken with DAT 1 did not include the workshop event. DAT 1

was used to further refine the tool, prior to the tool being piloted in DAT 2. The refining of

the tool and questionnaire in DAT 1 was undertaken to allow for the assessment of the

proposed data analysis techniques and the tool’s ease of use to uncover potential problems,

26

thereby providing a cleaner model of enquiry in DAT 2. An outline of the steps undertaken

in this process is provided in Figure 2 below:

Figure 2: Process steps

Chapter 5

Findings & Analysis

27

Chapter 5. Findings & Analysis

The data analysed has been separated into the following sections:

5.1 Drug Action Team (DAT) 1,

Semi-structured Interviews

Provides an overview of levels attributed for each factor

from interviews in DAT 1; level descriptions obtained

for each of the factors, alongside a brief summary of the

perceptions of interviewees.

5.1.1 Efficacy of the Tool to Support,

Plan and Monitor User

Involvement

Provides data from DAT 1 on the number of responses

received, response percentages, variations in responses

and comments from interviewees.

5.1.2 Amendments to Process and/or

Assessment Tool

Provides information on aspects of the process and/or

assessment tool that were amended prior to piloting in

DAT 2.

5.2 Drug Action Team (DAT) 2,

Semi-structured Interviews

Provides an overview of levels attributed for each factor

from interviews in DAT 2; level descriptions obtained

for each of the factors, alongside a brief summary of the

perceptions of interviewees.

5.2.1 Efficacy of the Tool to Support,

Plan and Monitor User

Involvement

Provides data from DAT 2 on the number of responses

received, response percentages, variations in responses

and comments from interviewees.

5.2.2 Workshop Provides an overview of the assessment of

organisational delivery by the DAT 2 commissioning

team in a facilitated workshop. Data generated from the

semi-structured interviews was used as a base line for

discussion.

28

5.1 Drug Action Team (DAT) 1, Semi-structured Interviews

The total number of staff (officers) in DAT 1 is seven and four officers were interviewed. In

addition to the four officers, a member of the local user group was also interviewed and the

total number of interviewees was five. The initial questionnaire presented to DAT 1 officers

contained only the five factors previously identified (Needs Assessment, Leadership,

Programme/Policy Development, Programme/Policy Implementation, Resource

Mobilisation).

However, feedback from the interview process resulted in questions, which combined

‘programme and policy’, being separately asked for each. That amendment increased the

factors being assessed from five to seven. The rationale for this decision was that

interviewees felt that programme development and implementation was a more organic

process than policy development and implementation and offered greater opportunity for

SUI. Whereas, a percentage of policies were nationally driven and presented less opportunity

for SUI, therefore adversely affecting the assessment of programme development and

implementation. Therefore the data presented below reflects interviewees separate

assessments of programme and policy, as distinct factors, and presents data for the following

seven factors:

• Needs Assessment

• Leadership

• Programme development

• Policy development

• Programme implementation

29

• Policy implementation

• Resource Mobilisation

Data from the completed questionnaires was collated and analysed and has been presented

below for each of the seven factors assessed. This section provides: an overview of levels

attributed for each factor (Figure 3, below), level descriptions obtained for each of the

factors, alongside a brief summary of the perceptions of interviewees.

Figure 3: Factor levels attained

30

Needs Assessment:

Factor Level stated Level description

Needs Assessment Level 3

The ‘needs’ are identified by professionals and the

assessment is used to strengthen user involvement,

with a wide range of potential beneficiaries views used

to inform ‘needs’.

The general view of interviewees was that the previous year’s Needs Assessment was very

proactive in seeking user views. However, whilst there was positive engagement with users, it

was generally perceived that their involvement in the Needs Assessment focused more on

testing out ideas or issues rather than informing the direction of travel. In addition, service

users have not previously been involved in shaping the design of the Needs Assessment, and

the experience in meetings with decision-makers has been that the Needs Assessment is more

professional led.

Leadership:


Leadership Level 3

Leaders or programme leads perceive user

participation as a key delivery mechanism and

incorporate user participation as a way to ‘inform’

delivery when seen to be beneficial.

Whilst Leadership within the DAT was perceived positively, with a sense that leaders keep

user involvement at the forefront of delivery, those they report to in the local authority were

seen to be more limited in their engagement of service users. In addition, mechanisms to

31

support service users in decision-making were not felt to be fully implemented and although

service user views were valued and heard, there was a question as to the level of influence

they bring to decision-making at a strategic level i.e. “they have a voice but not a vote”.

Programme Development:


Programme Development Level 2

Users are not involved in setting the aims and

objectives and appraisal of the development of new

programmes, but are involved in other aspects.

User involvement in programme development was perceived to be more of an organic

process, with users asked to comment at first stage drafting rather than at the conceptual

stage. There appeared to be a lack of policy and procedure governing user involvement in this

area, which may result in inconsistency of when and how service users become involved.

Policy Development:


Policy Development Level 2

Users are not involved in setting the aims and

objectives and appraisal of the development of new

policies, but are involved in other aspects.

Perceptions of user involvement in policy development reflected those of programme

development. In addition, involvement was perceived to be further impacted upon by policies

being decided by government departments or locally i.e. local authority and elected members.

32

The view on involvement at a policy level was encapsulated in one interview with the

comment "doing it to people and informing them, as opposed to doing it with people or led by

them".

Programme Implementation:


Programme

Implementation Level 2

Users are consulted in part but not reflective across

all aspects of implementation.

Service users have previously been involved in some aspects of implementation i.e. Training

professionals, and current changes taking place to the user forum are anticipated to strengthen

their role in programme implementation. However, service users were not perceived to be

currently involved in reviewing services or scoping delivery options of new programmes.

Policy Implementation:


Policy Implementation Level 2 Users are consulted in part but not reflective across

all aspects of implementation.

Perceptions of user involvement in policy implementation reflected those of programme

implementation. Procedures do not appear to be in place to support user involvement in the

review or risk analysis of how new policies might impact on the community or services. In

addition, it was felt consideration might be given to SUI in how policies might be more

effectively structured or worded to provide clarity to a wide ranging audience.

33

Resource Mobilisation:


Resource Mobilisation Level 3

User participation is appropriately resourced and

supported; is representative of a cross section of

beneficiaries; but predominantly utilises consultation

as a mechanism for participation.

The resourcing of user involvement encompasses a structured user forum and a supporting

budget, with activities stated in the business plan. The perception of interviewees was that,

whilst there is scope to increase any activity or budget, the current level of resourcing enabled

the partnership to respond appropriately to the current level of need. However, it was

acknowledged that the current structure was better at engaging those that are currently in

treatment and that there was "a need to be more creative".

Level selection and standard deviation:

The levels stated by each interviewee where added together and divided by the number of

interviewees to obtain the mean value. Where the mean value included a decimal place e.g.

2.5, the mean value was either rounded up or down to the nearest whole number. The mean

value and standard deviation are outlined in Table 1 below. The outcome of the rounding

process meant that outliers (standard deviation ≥ 0.6) did not negatively impact on the final

level obtained.

34

However, the outliers identified in Leadership, Programme Implementation and Resource

Mobilisation require further enquiry by the organisation as they represent the experience of

being engaged through existing policies and procedures. Further enquiry might seek to

ascertain how widespread this experience is, if the experience is impacted upon by the level

of feedback to service users on involvement activity, or, the impact of a new SUI structure

and how these changes are being facilitated.

Whilst the inclusion of the local user group was to ascertain when (and if) they are involved

in planning, development and decision-making, the process did highlight questions relating to

their experience of being engaged. Such issues should be raised as part of the workshop

stage, with the expectation that actions were identified to either rectify or further explore their

underlying cause.

In addition, further consideration should be given to the role of local user groups in the

process and whether they should co-facilitate the assessment process. Co-facilitation might

also support wider debate within a workshop and provide validation of the final outcome.

Table 1: Mean value and standard deviation

Factor Responses Mean Standard deviation

Needs Assessment 3,3,3,2,3 2.8 0.44721

Leadership 3,2,4,3,1 2.6 1.14018

Programme Development 2,2,3,3,2 2.4 0.54772

Policy Development 3,2,2,2,2 2.2 0.44721

35

Programme Implementation 3,2,2,2,1 2 0.70711

Policy Implementation 2,2,2,2,1 1.8 0.44721

Resource Mobilisation 4,2,4,3,2 3 1

5.1.1 Efficacy of the Tool to support, Plan and Monitor User Involvement

Each interviewee was presented with nine statements (Assessment Tool Evaluation

Questionnaire, Appendix 2) relating to the assessment tools overall ease-of-use, the factors

being measured and the tool’s ability to support future planning and monitoring of SUI.

However, it should be noted that statements 3, 4, 5, 6, and 7 were replicated for each of the

factors being assessed and therefore the number of statements being responded to in total was

thirty nine. Interviewees were asked to rate their level of agreement with each statement by

stating whether they either: strongly agree; disagree; neither agree or disagree; agree;

strongly agree; not known. This was undertaken as part of the semi structured interviews

conducted to facilitate the completion of the assessment tool.

A total of 195 responses from the five interviewees where recorded. Table 2 (below) shows

that a significant percentage (96.5%) of responses were positive (agree/strongly agree).

Table 2: Response percentages

Strongly disagree 0 .5% (1)

Disagree 2.5% (5)

36

Neither agree or disagree 0 .5% (1)

Agree 55% (107)

Strongly agree 41.5% (81)

Not known 0

Responses that either disagreed or strongly disagreed were in response to the following

statements:

Statement 1 – I found the questionnaire easy to use.

One (1) interviewee strongly disagreed with the statement being made. The

interviewee felt that the questionnaire was not intuitive and required flipping from one

section to another to fully understand and respond to the points being raised.

Statement 2 – The guidance for completing the questionnaire met my needs.

One (1) interviewee disagreed with the statement being made. The interviewee felt

that there was a lot of guidance that at times was “heavy going”. In addition, other

interviewees, whilst understanding the guidance, questioned whether the wording was

at times too academic and would not transfer to a wider audience.

Statement 3 – The questions in Section B, relating to each factor, provide clarity on

the types of activity expected.

37

One (1) interviewee disagreed with the statement being made when applied to

Programme/Policy Development and Programme/Policy Implementation. The

interviewee’s response was based on the view that programme and policy could not

be assessed together due to new policies being, at times, nationally driven. It is worth

noting that other interviewees had provided positive responses to this statement for

each of the factors stated but with the caveat that the two items (programme and

policy) were separated as factors.

All interviewees responded positively (agree/strongly agree) to all other statements as

outlined below:

Statement 4 – The questions in Section B, relating to each factor, helped me decide

where to place my organisation on the continuum. (Agree 57.2%, strongly agree

42.8%)

Statement 5 – The levels stated for each factor were fair. (Agree 77.1%, strongly

agree 22.9%)

Statement 6 – The factors used represent key factors for SUI in organisational

planning, development and decision-making processes. (Agree 40%, strongly agree

60%)

Statement 7 – The assessment tool supported me in understanding the types of

activity I should consider for effective SUI. (Agree 40%, strongly agree 60%)

38

Statement 8 – The assessment tool would help me plan SUI more effectively. (Agree

40%, strongly agree 60%)

Statement 9 – The assessment tool would help me monitor how SUI is being

delivered. (Agree 20%, strongly agree 80%)

Comments made by interviewees during the completion of the questionnaire included:

• "A lot of guidance! Methodology was interesting....... personally I could have lived

without it."

• "Could too easily be used as a tick box exercise if not conducted as part of an

interview."

• "You got the answers out because you've asked me, if you are just reading from this

you wouldn't understand where I'm coming from."

• "The questions (Section B) were more helpful than the levels."

• "Thought-provoking (Section B)"

• "It does make you question the kinds of support you give to people to actually take

part."

• "It raised a couple of issues for me (previous planning)."

Data from semi structured interviews and the questionnaire on the tool’s ease-of-use show

that interviewees perceived that: a) the content and structure of the assessment tool supported

them in its completion, and, b) that the assessment tool itself was effective in supporting

planning and monitoring of SUI. However, the volume of guidance needed to be reduced; the

wording should be more reader friendly and that programme and policy should be separated

39

into individual factors. In addition, all interviewees reported that the iterative process of a

semi-structured interview enabled them to better understand their rationale for the levels

chosen and where improvements in the system could be considered. The interviewer found

that the interview process provided context for the levels attributed, which might contribute

to providing a richer picture to present in the workshop stage and form the basis for a wider

debate in planning future actions.

5.1.2 Amendments to Process and/or Assessment Tool

In response to data collected from the Draft Assessment Tool Questionnaire (Appendix 1),

the Assessment Tool Evaluation Questionnaire (Appendix 2) and general comments from

DAT 1, elements of the process and the questionnaires were amended. Amendments to the

process and questionnaires were as follows:

1. Semi-structured interviews were incorporated as part of the data collection process.

The use of semi-structured interviews was initially undertaken to a) enable the

interviewer to note any difficulties experienced by the respondent in completing the

assessment tool questionnaire, and, b) ask additional questions to ascertain the tool’s

ease-of-use and ability to support planning and monitoring of user involvement.

However, interviewees reported that the iterative process of a semi-structured

interview supported their decision-making and the interviewer identified that the

process supported the collection of additional data to support the next stage

40

(workshop). In addition, using interviews as the vehicle for data collection allowed for

a reduction in the level of guidance required for completing the questionnaire.

2. Factors that combined ‘programme and policy’ were separated and the factors being

assessed were increased from five to seven.

Interviewees felt that programme development and implementation was a more

organic process than policy development and implementation and offered greater

opportunity for SUI. Whereas, a percentage of policies were nationally driven and

presented less opportunity for SUI, therefore adversely affecting the assessment of

programme development and implementation.

3. The Assessment tool questionnaire was re-configured and the methodology section

was removed. (NB The questionnaire was subsequently renamed as ‘User

Involvement Questionnaire’)

Interviewees felt that there was a significant amount of guidance in the questionnaire.

It was noted by the interviewer, alongside one comment, that interviewees needed to

constantly refer to different sections of the questionnaire to fully consider their

responses. Changes to the questionnaire were undertaken to reduce the necessity to

refer to Section B to enable completion. In addition, incorporating semi-structured

interviews as the vehicle for data collection also removed the necessity for the

methodology section as this could be conveyed by the interviewer during the

interview.

41

The amendment to the number of factors being assessed (item 2, above) also required these

changes to be incorporated into the Assessment Tool Evaluation Questionnaire.

It should be noted that data from DAT 1 was utilised to refine the assessment process and

questionnaires. Therefore, conclusions drawn from findings and analysis will refer only to

DAT 2 as the pilot site. The revised questionnaires (User Involvement Questionnaire,

Appendix 3 and Assessment Tool Evaluation Questionnaire v2, Appendix 4) were

subsequently used to run the pilot in DAT 2.

5.2 Drug Action Team (DAT) 2, Semi-structured Interviews

The total number of staff (officers) in DAT 2 is ten and seven officers were interviewed. In

addition to the seven officers, a member of the local user group was also interviewed and the

total number of interviewees was eight.

Data from the completed questionnaires was collated and analysed and has been presented

below for each of the seven factors assessed. This section provides: an overview of levels

attributed for each factor (Figure 4, below), level descriptions obtained for each of the

factors, alongside a brief summary of the perceptions of interviewees.

42

Figure 4: Factor levels attained

Needs Assessment:


Needs Assessment Level 4

Users inform the ‘needs’ identified and are part of the

decision making process on how these ‘needs can be

best met.

The view of all interviewees was that the ‘Needs Assessment’ was an inclusive process,

which is ongoing and incorporates audits undertaken throughout the year that were either

service user led, or validated by service users. Whilst it was recognised that professionals

ultimately make the decisions, it was service users’ ability to challenge those decisions, that

43

holds those responsible accountable. However, the DAAT appeared reliant on the service

user council to provide users’ views from a breadth of possible beneficiaries.

Leadership:


Leadership Level 4

Leaders and programme leads perceive user

participation as a key delivery mechanism and actively

seek it to inform delivery wherever possible.

Leadership within the DAAT was perceived positively, with a sense that service user

involvement is so “embedded” in individual culture and practice that “they wouldn’t think of

doing anything without user involvement”. However, there was recognition from some

interviewees that whilst this is the case, this position is reliant on current culture and working

practice. As such, further exploration of this might consider how existing governance

structures would sustain user involvement should the culture and practice of individuals

change i.e. changes in senior management.

Programme Development:


Programme Development Level 4

User participation is incorporated in all aspects of

programme development and is encompassed in

organisational policy and procedures to guide

professionals.

44

User involvement in programme development was perceived to be embedded at a policy

level, which was evident throughout the recent service redesign with service users

participating in planning, tendering and interviewing. This view was encapsulated in one

interview with the comment “we make sure service users have their fingers in all the pies”.

However, some interviewees questioned if involvement consistently began at the first stage

of development and whether there was sufficient understanding of the objectives for

involvement and clarity of roles and responsibilities.

Policy Development:


Policy Development Level 3

User participation is incorporated in aspects of policy

development, including either setting aims and

objectives or appraisal but is not encompassed in

organisational policy or procedures to guide

professionals.

User involvement in policy development was perceived as positive with anecdotal evidence

of the service user council being pro-active in identifying changes and recommendations for

policy reviews. However, where policies are developed locally there was a perception by

some that whilst there were checks and balances in place to ensure user involvement, with

national policies a gap was perceived in guidance to ensure policies were referred back into

local processes.

45

Programme Implementation:


Programme

Implementation Level 3

There is more active user participation, rather than

consultation, and in the majority of activity.

Active participation of service users was perceived at a strategic level within the DAAT.

However, this was seen by some as dependent on the size of the programme and also the

service users council capacity to respond e.g. where membership changes. In addition, user

involvement in programme implementation was impacted by the externalising of programmes

and communication with providers requires “tweaking”/”ironing out” to improve the level of

participation. The transition to a new treatment system was acknowledged as a key factor in

interviewees’ perceptions of current activity.

Policy Implementation:


Policy Implementation Level 4

Active participation in all aspects of implementing a

new policy, except the review of local drug partnership

delivery plans.

Perceptions of user involvement in policy implementation were positive and included an

acknowledgement of service users’ ongoing involvement in reviewing policies post

implementation and participation in governance arrangements. However, some interviewees

questioned whether the level of influence across all aspects of policy implementation was

consistent.

46

Resource Mobilisation:


Resource Mobilisation Level 5

The organisation allocates resources to support and

deliver user participation and ‘users’ have a clear role

in decision making; including the allocation and/or

setting of resources.

The resourcing of user involvement was perceived to be robust, and whilst set in the context

of normal budgetary constraints, it was felt the current level of resourcing was appropriate to

the perceived need. In addition, service users are involved in agreeing overall treatment

delivery budgets and the prioritising of resources.

Level selection and standard deviation:

The levels stated by each interviewee were added together and divided by the number of

interviewees (8) to obtain the mean value. Where the mean value included a decimal place

e.g. 2.5, the mean value was either rounded up or down to the nearest whole number. The

mean value and standard deviation are outlined in Table 3 below. The outcome of the

rounding process meant that outliers (standard deviation ≥ 0.6) did not negatively/positively

impact on the final level obtained, with the exception of Policy Implementation, which was

positively impacted upon.

47

The outliers identified in Leadership, Programme Development and Policy Implementation

require further enquiry as they represent the allocation of a level 5 by some interviewees. A

level 5 represents an organisational position where service users plan; implement and

evaluate activity, with professionals used as a resource. Whilst the attainment of a level 5 is

not impossible, it is unlikely given the current organisational structure of DAT’s nationally,

and therefore requires clarification in the workshop stage.

Table 3: Mean value and standard deviation

Factor Responses Mean Standard deviation

Needs Assessment 4,4,4,4,4,4,4,4 4 0

Leadership 5,5,4,3,3,4,4,5 4.1 0.83452

Programme Development 3,4,4,3,4,4,4,5 3.8 0.64087

Policy Development 3,3,4,4,3,4,3,3 3.3 0.51755

Programme Implementation 3,3,3,4,3,3,4,3 3.2 0.46291

Policy Implementation 3,3,3,5,4,5,3,4 3.7 0.88641

Resource Mobilisation 5,5,4,5,4,5,4,5 4.6 0.51755

48

5.2.1 Efficacy of the Tool to Support, Plan and Monitor User Involvement

A total of 312 responses from eight interviewees where recorded from the Assessment Tool

Evaluation Questionnaire v2 (Appendix 4). Table 4 (below) shows that a significant

percentage (92%) of responses were positive (agree/strongly agree).

Table 4: Response percentages

Strongly disagree 0

Disagree 0.3% (1)

Neither agree or disagree 3.2% (10)

Agree 41% (128)

Strongly agree 51% (159)

Not known 4.5% (14)

Responses that either disagreed, neither agreed nor disagreed or stated not known were in

response to the following statements:

Statement 2 – The guidance for completing the questionnaire met my needs.

One (1) interviewee neither agreed nor disagreed with the statement being made. The

interviewee questioned whether a service user new to treatment would understand the

guidance for completing the questionnaire.

Statement 3 – The questions in Section B, relating to each factor, provide clarity on

the types of activity expected.

49

One (1) interviewee felt they could not answer the question (Not known) as they had

not used the questions in Section B and therefore could not comment.

Statement 4 – The questions in Section B, relating to each factor, helped me decide

where to place my organisation on the continuum.

One (1) interviewee felt they could not answer the question (Not known) as they had

not used the questions in Section B and therefore could not comment.

One (1) interviewee disagreed with the statement being made for ‘Leadership’ only as

they felt the questions were not as expansive as those for other factors.

Statement 5 – The levels stated for each factor were fair.

One (1) interviewee neither agreed nor disagreed with the statement being made. The

interviewee felt that the statements against each level were at times too prescriptive

and did not lend themselves to the complexity of the situation “not enough wriggle

room”.

One (1) interviewee neither agreed nor disagreed with the statement being made for

Programme and Policy Development respectively. The interviewee felt that the level

statements for these two factors did not fully reflect their view.

50

All interviewees responded positively (agree/strongly agree) to all other statements as

outlined below:

Statement 1 – I found the questionnaire easy to use. (Agree 62.5%, strongly agree

37.5%)

Statement 6 – The factors used represent key factors for SUI in organisational

planning, development and decision-making processes. (Agree 62.5%, strongly agree

37.5%)

Statement 7 – The assessment tool supported me in understanding the types of

activity I should consider for effective SUI. (Strongly agree 100%)

Statement 8 – The assessment tool would help me plan SUI more effectively. (Agree

37.5%, strongly agree 62.5%)

Statement 9 – The assessment tool would help me monitor how SUI is being

delivered. (Agree 62.5%, strongly agree 37.5%)

Comments made by interviewees during the completion of the questionnaire included:

• “Reading this made me very aware that I don’t always reflect on where I could ask

more of SUI to support various work streams.”

• "Required dialogue to fully understand all the statements."

51

• "Near enough, language appropriate for professionals but would question

understanding for individuals with less experience.”

• "I felt at times it was too prescriptive.”

• "Would look at different process for engaging young people.”

Data from semi structured interviews and the questionnaire on the tool’s ease-of-use show

that interviewees perceived that: a) the content and structure of the assessment tool supported

them in its completion, and, b) that the assessment tool itself was effective in supporting

planning and monitoring of SUI. However, the statements for each of the levels were at times

too prescriptive and/or required dialogue with the interviewer to understand their relation to

organisational activity.

5.2.2 Workshop

A workshop (2 hours) with the DAT 2 commissioning team, and a member of the local user

group, was held to facilitate a final decision on where marks should be placed on the

continuum for each factor. Data from the semi-structured interviews (4.2, above) was used as

the basis for discussion. The process used to reach agreement was as follows:

1. An overview of the data from semi-structured interviews was presented to the group,

followed by the outcome for each factor and general perceptions of current activity.

2. For each factor, the group were asked to:

52

• Consider whether the level attained from the interview process was a fair

representation of organisational delivery

• Discuss why, or why it is not a fair representation

• Vote on which level they believe is a fair representation of organisational

delivery

• Identify actions that would either improve, or, sustain current levels of

delivery

At the beginning of the process (above) the group where asked to agree how levels would be

attributed where there was not unanimous agreement in the group. The group decided a

majority vote would identify each level. Where there was no majority, the mean would be

established, as per the interview process (see Table 3, above). However, where the mean

contained a fraction, the number would be rounded down at 0.5, as opposed to ‘up’ which

was undertaken in the interview process. The rationale for this decision was that they felt

there needed to be proportionally more positive than negative actions to warrant rounding up.

The levels attributed at the workshop (final outcome) and comparisons to those attributed

from interviews are contained below in Table 5.

53

Table 5: Levels attributed from workshop

Factors being assessed Levels attributed

from interviews

Levels attributed

from workshop

(final outcome)

Variations in levels

attributed

Needs Assessment 4 4 No change

Leadership 4 4 No change

Programme Development 4 4 No change

Policy Development 3 4 Level increase from 3 - 4

Programme Implementation 3 3 No change

Policy Implementation 4 4 No change

Resource Mobilisation 5 4 Level decrease from 5 - 4

Two factor levels where changed as a result of debate by the team in the workshop. The

factors changed were a) Policy Development, and, b) Resource Mobilisation. The rationale

for these changes is as follows:

a) Policy Develop was identified through interviews as a level 3. This was due to some

interviewees perceiving a gap in ensuring national policies were referred back into

local processes. However, through wider discussion and mapping previous actions

and process, the group felt this was not a true reflection of delivery. Whilst responses

to national and/or local policies (that are not substance misuse specific) are at times

impacted by time constraints, the team felt that internal processes and actions reflect a

level 4.

54

b) Resource Mobilisation was identified through interviews as a level 5. This represents

an organisational position where service users plan, implement and evaluate activity,

with professionals used as a resource. However, during discussions it was

acknowledged that ‘professional’s’ ultimately have responsibility for decision-

making. The selection of level 5 by some interviewees’ had been governed by the

level description used in the assessment tool itself. The wording of a level 4

description was subsequently replaced by the original level 5 description and the team

allocated a level 4 for this factor. This was undertaken to reflect organisational

activity but acknowledged that professional’s ultimately have responsibility for

decision-making.

Recommendations to support and maintain future user involvement of the DAT included:

• Support the local user group to link with other user groups within the borough. This

action is reliant on the ongoing development of the user group and its resourcing,

alongside a borough wide identification of need and structure to support such an

initiative.

• Recommend the consideration of borough wide service user involvement in

forthcoming Partnership Board reviews.

• Monitor the impact of NHS and public sector reform on service user involvement

within the DAT.

• Ensure the completion of appropriate levels of documentation to support and monitor

service user involvement e.g. project initiation documents. This action is reliant on

both team and departmental/partnership compliance. Departmental/partnership

55

compliance is often more difficult to influence and could be supported through a

structured feedback process.

• Continue to monitor and support the levels of, and requirements for, skills within the

local user group. Membership of user groups naturally changes over time as new

service users are engaged, and as such, skills and knowledge will vary. Organisations

must therefore maintain a balance between supporting individuals to participate in

user groups, in a way that does not negatively impact on their treatment journey,

whilst ensuring the group has an appropriate skill set to effectively represent the

views of the community it seeks to serve.

• Maintain the inclusion of service users in the risk analysis of programme and policy

implementation. This action can be impacted upon jointly by timescales and the

capacity within the local user group. Timescale and capacity relates to a) where

implementation is for services that are not substance misuse specific and are

implemented by other departments or partners, and, b) implementation by

organisation’s commissioned by the DAT. Consideration should be given to the level

of user involvement required on a case-by-case basis. In situation ‘a’, where full user

involvement is not undertaken, the reasons for the level of involvement undertaken

and any subsequent risks should be noted. In situation ‘b’, levels of user involvement

should be supported, monitored and/or challenged through contracting processes.

The data from the semi-structured interviews provided a rich picture from which to generate

debate and challenge perceptions of organisational delivery during the workshop. In addition,

whilst there were minimal changes to the levels attained, those that did change have a clear

rationale for doing so.

56

However, the ability of the team to openly debate organisational delivery was seen as a

reflection on the level of ease that was apparent between the team and senior management

present. In addition, the teams’ willingness and ability to question individual practice and

organisational deliver supported the piloting of the assessment tool. As such, future

assessment of the tools’ use might consider the impact on the final outcome where senior

management (Leadership) appears less supportive of user involvement.

Whilst there is no expectation that outcomes from semi-structured interviews should mirror

those of the workshop, any significant and/or multiple shifts in levels should be fully

explored by the facilitator. In addition, the independence of the facilitator (interviewer)

should be explored as a factor in establishing a fair representation of organisational delivery.

In the piloting of the assessment tool, a member of the Alliance monitored both the semi-

structured interviews and the workshop. Their prior knowledge and perception of DAT 2’s

organisational delivery was comparable to the outcome of the assessment.

Chapter 6

Conclusions

57

Chapter 6. Conclusions

Whilst current policy and guidance seeks to support and guide user involvement, far less

attention has been given to where and how involvement takes place in organisations. As such,

we find ourselves in the position whereby we are unable to:

• routinely measure levels of activity,

• benchmark activity to support future development and research, or

• provide accountability.

The focus of this study was to establish whether an assessment tool could a) measure user

involvement in the planning, development and decision-making activity of a DAT, and, b) be

used to support future planning and monitoring of user involvement.

The piloting of the assessment tool has shown that by focusing on organisational processes

and where users’ are involved in its planning, development and decision-making processes,

we are able to:

d) recognise gaps in delivery and identify actions to improve and/or sustain user

involvement

e) support individuals and teams to understand and consider activity to support effective

SUI

f) effectively plan and monitor SUI

58

The use of individual interviews provided an effective vehicle for staff members to reflect on

their own practice and wider organisational delivery. This reflection provided the interviewer

with a rich picture to inform wider debate in the team workshop and challenge perceptions of

practice and process. The open debate and challenge of the workshop enabled the team to not

only identify areas for improvement but also recognise good practice that takes place and

what the strengths and weaknesses of the team were.

In addition, individual and team reflection of practice during the assessment may yield future

outcomes through greater understanding and focus. Those that participated in the pilot

strongly agreed with the statement that:

• The assessment tool supported me in understanding the types of activity I should

consider for effective SUI (strongly agree 100%).

“Understanding user involvement as a small part of a larger system helps bridge the divide

between micro level changes and system-wide reforms.”

(Tritter & McCallum, 2006, p166)

Whilst some comments on the assessment tools ease-of-use require further consideration i.e.

the wording for ‘level descriptions’, all participants agreed that:

• The assessment tool would help plan SUI more effectively.

• The assessment tool would help monitor how SUI is being delivered.

59

Levels of service user engagement will always be variable from organisation to organisation,

and impacted upon by a multitude of factors such as: rural versus urban; those in treatment

versus treatment naive; methods of engagement; resources; skills and experience. However,

effective user engagement is underpinned by an organisation with:

• a clear understanding of why, where and when users’ should be involved

• the ability to continually and consistently review policy and practice, and

• accountability to those they seek to serve.

Whilst we must continue to evolve the methods and tools of engagement, we first need to

assure ourselves that we, and the organisations we work in, fully understand the why, where

and when of user involvement. Only then can we appreciate and comprehend the HOW.

Chapter 7

Recommendations

60

Chapter 7. Recommendations

The piloting of the assessment tool was undertaken in a DAT that has been recognised within

their region as having progressive user involvement activity. This was reflected in their

willingness and ability to question individual practice and organisational delivery and was

perceived as a contributing factor to the outcome of the pilot. However, to fully understand

the limitations and/or challenges of undertaking such an assessment, further testing of the

assessment tool should take place in organisations with a range of user involvement

experience and organisational support for user involvement.

Wider testing of the assessment tool would also need to consider the need for the

independence of the facilitator (interviewer). Whilst the independence of the facilitator might

be seen to bring a level of validation to the outcome of the process, facilitation by a member

of the organisation may provide oversight of proposed actions and integration of

organisational learning.

Co-facilitation by a representative of the local user group would be seen as a key addition to

any future testing or implementation of the assessment tool. The inclusion of a representative

of the local user group not only supports wider debate throughout the process but would also

support the consideration of next steps and how future engagement takes place. In addition,

where involvement is less evolved, co-facilitation may also support the longer-term

relationship between staff and users.

Key to any further evaluation of the assessment tool’s efficacy is the support and/or

involvement of those with responsibility for the oversight and governance of the

61

organisations (local drug partnerships, or equivalent i.e. DAT) being assessed. Whilst there

are organisations that recognise and value user involvement, the inclusion of those with less

organisational support for it, presents a greater challenge.

62

References:

• Arnstein, S. R; 1969, A Ladder Of Citizen Participation, Journal of the American

Planning Association,35:4,216 – 224

• Carr, Sarah; 2004, Has service user participation made a difference to social care

services?, Social Care Institute for Excellence, British Library Cataloguing in

Publication Data, ISBN 1 904812 10 4

• Commission for Healthcare Audit and Inspection, January 2009, Improving services

for substance misuse, Diversity, and inpatient and residential rehabilitation services.

ISBN 978-1-84562-212-1

• Department of Health, 2006a, The National Health Services Act (modified 13 April

2007), viewed online 02/06/2010

http://www.dh.gov.uk/en/Publicationsandstatistics/Legislation/Actsandbills/DH_0641

03

• Department of Health, 2006b, Our health, our care our say – a new direction for

community services, The Stationary Office, ISBN 0-10-167372-8

• Department of Health, 30th June 2008a, High quality care for all: NHS Next Stage

Review final report, ISBN 978-0-10-174322-8

• Department of Health, 2008b, Real Involvement, Gateway reference 10546,

www.dh.gov.uk/publications

• Department of Health, 2010, NHS Constitution, viewed online 02/06/2010

http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/@ps/docu

ments/digitalasset/dh_113645.pdf

• Fiander, M; Burns, T; 2000, Psychiatric Services Vol. 51 No. 5 A Delphi Approach to

Describing Service Models of Community Mental Health Practice, Psychiatric

63

Services Vol. 51 No. 5 viewed on line 04/10/09

http://psychservices.psychiatryonline.org/cgi/reprint/51/5/656

• Fischer, J; Jenkins, N; Bloor, M; Neale, J; Berney, L; 17 March 2007, Drug user

involvement in treatment decisions, Joseph Rowntree Foundation, ISBN 978 1 85935

565 7 (pdf: available at www.jrf.org.uk)

• GLADA (Greater London alcohol and drugs alliance), 2005, Lessons Learned,

Greater London Authority, ISBN 1 85261 707 1

• Home Office, 1 January 1998, Tackling Drugs To Build a Better Britain, Product

number: 0101394527

• Home Office, February 2008, Drugs: protecting families and communities – the 2008

drug strategy, ISBN 978-1-84726-615-6

• Jürgens, R; 2008, “Nothing about us without us” — Greater, meaningful involvement

of people who use illegal drugs, Canadian HIV/AIDS Legal Network, International

HIV/AIDS Alliance, Open Society Institute ISBN 978-1-896735-92-4

• McLaughlin, H; 2009, ‘Keeping Service User Involvement in Research Honest’,

Oxford University Press.

• National Treatment Agency, June 2006, NTA Guidance for local partnerships on user

and carer involvement. viewed online 12/06/2010

http://www.nta.nhs.uk/areas/users_and_carers/publications/nta_guide_to_u&c_involv

ement6.6.pdf

• National Treatment Agency, July 2009, Undertaking needs assessment, viewed online

12/06/2010

http://www.nta.nhs.uk/areas/treatment_planning/treatment_plans_2010_11/needs_ass

essment_2010_11.pdf

64

• National Treatment Agency for Substance Misuse, 2010, Commissioning for recovery

Drug treatment, reintegration and recovery in the community and prisons: a guide for

drug partnerships. London: NTA Gateway number: 12944. Available as download:

http://www.nta.nhs.uk/uploads/summary_c4r_2010.pdf

• Nicholson, L; 2005, Civic Participation in Public Policy-Making: a Literature Review,

The Research Shop, ISBN 0 7559 2700 1 (Web only publication)

http://www.scotland.gov.uk/Publications/2005/09/16120247/02537

• Participation Works, 2008, Listen and Change, Children’s Rights Alliance for

England, ISBN: 978-1-898961-19-2

• Patterson, S; Crawford, M.J; Weaver T; Rutter, D; Agath, K; Albert, E; Hunt, A;

Jones, V; 2007, User involvement in efforts to improve the quality of drug misuse

services: factors that promote and hinder successful working, London: Department of

Health; 2007.

• Patterson, R, S., Smith, D, B., Martorana, P, V., Owens, P, D., 2003. The Impact of

Chief Executive Officer Personality on Top Management Team Dynamics: One

Mechanism by Which Leadership Affects Organizational Performance, Journal of

Applied Psychology 2003, Vol. 88, No. 5, 795–808

• Rifkin, S, B; Muller, F; Bichmann, B; 1988, Primary health care: on measuring

participation, Social Science and Medicine, vol 26 (9) pp.931-940.

• Rifkin, S, B., 1990. Community Participation in maternal and child health/family

planning programmes: An analysis based on case study material, WHO/UNICEF,

Geneva, In press.

• Robinson, S.K; 2004, Consumer and Community Participation Self-Assessment Tool,

Version 2.0, National Resource Centre for Consumer Participation in Health, ISBN 0-

9750924-8-0

65

• Rowe, G; Frewer, L.J.; 2004, Evaluating Public-Participation Exercises, Science,

Technology, & Human Values, Vol. 29, No. 4 (Autumn, 2004), pp. 512-557

Published by: Sage Publications

• Speak Up, 2007, Bolton Council, national service user involvement survey, viewed

online, 26/03/2010 http://www.serviceuserinvolvement.co.uk/

• Tritter, J.Q.; McCallum, A; 2006, The snakes and ladders of user involvement:

Moving beyond Arnstein, Science Direct Health Policy 76 (2006) 156–168

• Welsh Assembly, 2008, User Involvement Framework,

http://www.adjudicationpanelwales.org.uk/dsjlg/publications/commmunitysafety/sub

misusetreatframework/usere.pdf;jsessionid=4rwyKPhK9DnC8T2SG0d1szL0Hp2zzR

wC4Df2NvjMpfFBVRtv7sYV!-692465818?cr=4&lang=en&ts=1

• Young, S; 2006, ‘Introducing Role and Service Changes in Health and Social Care:

The Impact and Influence of User Involvement in England and Wales’ Social Policy

& Society 5:2, 249–268

Service User Involvement: How do we know it's a reality?

Documents

Transcript of Service User Involvement: How do we know it's a reality?