Improving Machine Diagnostics and Customer Self-Service – Respecifying the design space

Jack Whalen

A fieldwork case study

Project rationale n  The company’s service costs can be significantly

reduced if more problems can be solved remotely rather than by a CSE (Customer Service Engineer) on site

n  Customer self-service is naturally the most cost-effective solution

Project rationale n  The company’s service costs can be significantly

reduced if more problems can be solved remotely rather than by a CSE (Customer Service Engineer) on site

n  Customer self-service is naturally the most cost-effective solution

n  And so the company has a ‘move to the left’ strategy

Customer self-service

Support Centre solve

Second Level Customer Support Solve

CSE Remote Call Solve

CSE On-Site

The more we move calls to the left - the lower the cost to the company - the faster ‘time to solution’ is for customer

Project rationale n  The company’s service costs can be significantly

reduced if more problems can be solved remotely rather than by a CSE (Customer Service Engineer) on site

n  Customer self-service is naturally the most cost-effective solution

n  And so the company has a ‘move to the left’ strategy n  ’Voice of the customer’ data suggests customers

might be motivated to fix things themselves n  …but the company’s own data from the field shows

many lost opportunities n  Consequently, this ‘move to the left’ strategy faces

some daunting challenges

Challenges facing ‘move to the left’ n  Technical

–  Insufficient diagnostic information –  Lack of customer self-help tools –  Lack of coverage and specificity in the existing fault code

structure

n  Organisational –  Lack of integration (including information sharing) between

service and support organisations Field service/CSEs + field analysts, 2nd level customer support, Customer Support Centres, web self-service, training for customers

n  Socio-cultural –  Customer practices –  CSE practices

Challenges facing ‘move to the left’ n  Technical

–  Insufficient diagnostic information –  Lack of customer self-help tools –  Lack of coverage and specificity in the existing fault code

structure

n  Organisational –  Lack of integration (including information sharing) between

service and support organizations Field service/CSEs + field analysts, 2nd level customer support, Customer Support Centres, web self-service, training for customers

n  Socio-cultural –  Customer practices –  CSE practices

Can we make our machines smarter? More and better sensors, better inference engine = better diagnostics (and prognostics)

Challenges facing ‘move to the left’ n  Technical

–  Insufficient diagnostic information –  Lack of customer self-help tools –  Lack of coverage and specificity in the existing fault code

structure

n  Organisational –  Lack of integration (including information sharing) between

service and support organizations Field service/CSEs + field analysts, 2nd level customer support, Customer Support Centres, web self-service, training for customers

n  Socio-cultural –  Customer practices –  CSE practices

Can we make our machines smarter? More and better sensors, better inference engine = better diagnostics (and prognostics)

2009 objective: Develop an embedded diagnostics prototype for increasing customer self-help or remote solves Focus: IQ problems, ‘streaks’ in particular

Challenges facing ‘move to the left’ n  Technical

–  Insufficient diagnostic information –  Lack of customer self-help tools –  Lack of coverage and specificity in the existing fault code

structure

n  Organisational –  Lack of integration (including information sharing) between

service and support organizations Field service/CSEs, 2nd level customer support, Customer Support Centres, web self-service, training for customers

n  Socio-cultural –  Customer practices –  CSE (and Customer Support Centre) practices

But will machine users be able to take advantage of better diagnostic information and customer self-help tools?

Fieldwork is playing a key role n  Exposing the research engineers to the reality of

life in the field and the real-world contexts in which ’diagnosis’ actually takes place

n  Building realistic models of different kinds of users –  User ability to distinguish certain failure modes

(e.g., types of ‘streaks’) –  User ability to perform certain tests

n  Helping to lead the iterative development of our diagnostic tools (including the diagnostic/inference model) through studies/interactions with users

Field studies n  Customer sites

–  Cupertino, CA –  Pawtucket, RI –  East Greenwich, RI –  Providence, RI –  St Charles, IL –  Buffalo Grove, IL

n  CSEs –  Peninsula/South Bay –  Chicago area

Site profiles n  Detailed profiles have been assembled for each site, including things like business

model, types of equipment, impressions per month, UM rate, operator characteristics, and the like

n  Profiles show that these sites fall into several different categories, covering much of the printing industry and the company’s customer base…

–  Small print shops struggling to find their niche and survive in a consolidating industry, with only one or two office-level machines

–  Mid-size print shops facing similar economic pressures but with a broader customer base and a significant commitment to the company’ equipment and high-speed digital

–  Mid-size internal printing operations (and so largely immune from industry consolidation pressures) that are committed to the company’s equipment and high-speed digital

–  Large printing operations that rely exclusively (or extensively) on high-speed digital equipment and appear committed to the company

»  But note that one large operation does mostly black-and-white or jobs that use only a small-to-medium amount of color, with more transaction rather than publishing jobs, while another produces color-intensive material for its customers and does more publishing jobs

–  Large printing operations that are almost exclusively

Site profiles n  Detailed profiles have been assembled for each site, including things like business

model, types of equipment, impressions per month, UM rate, operator characteristics, and the like

n  Profiles show that these sites fall into several different categories, covering much of the printing industry and the company’s customer base…

–  Small print shops struggling to find their niche and survive in a consolidating industry, with only one or two office-level machines

–  Mid-size print shops facing similar economic pressures but with a broader customer base and a significant commitment to the company’ equipment and high-speed digital

–  Mid-size internal printing operations (and so largely immune from industry consolidation pressures) that are committed to the company’s equipment and high-speed digital

–  Large printing operations that rely exclusively (or extensively) on high-speed digital equipment and appear committed to the company

»  But note that one large operation does mostly black-and-white or jobs that use only a small-to-medium amount of color, with more transaction rather than publishing jobs, while another produces color-intensive material for its customers and does more publishing jobs

–  Large printing operations that are almost exclusively

n  I did not observe business office environments, where there may be several printers but where the work at hand is not printing

–  My travels with CSEs sometimes brought me to these offices, however, and I then had the opportunity to observe the service challenges such environments present

Knowledge sources & users map

CSE Community

Documentation Community

Engineering Community

Users/ Operators

Machines

Eureka

Kana

User Forums

User Knowledge

eDocs RAPS

Design docs FMEAs

2nd level CS

Centre

The company

Push Machine DB

1st level CS

Centre

Customer Problems

CSE Knowledge

Local Log?

Customer Access

CSEs

Areas of investigation and preliminary findings n  Distinguishing between operators, users and customers n  ‘Diagnosis’ as in situ work n  Differences in the level of operator training and skill (as

well as self-service faculty) –  How do operators learn what they know?

n  Technician-operator relationships n  Operator self-service heuristics

–  With detailed examples of ‘problem’ episodes n  Seeing color n  How CSEs try to manage life in the field

–  With detailed examples of ‘problem’ episodes n  Community knowledge-sharing opportunities

Operators, users, and customers n  We should make a distinction between those whose job it is to

operate printing equipment — ’operator’ as an occupation — and those who simply use that equipment in the course of doing their work

n  People who occasionally use the machines are perhaps best referred to as ‘users’ –  The lack of training and experience amongst users is plainly a

significant problem when it comes to unnecessary service calls, inability or reluctance to replace CRUs, and the like (this is very evident when accompanying CSEs on UM calls to office environments)

n  ’Customer’ does not seem to be a particularly useful term when it comes to describing people who actually operate or use the machine; however, it is the right term for those whose main responsibility or concern is the business relationship with the company

Machines and their troubles n  Many reported service problems do not involve

machine faults (as represented by fault codes) but rather human judgments about machine actions, especially print results (basically, image quality perceptions) — not what the machine says but rather what the machine is seen to have done

n  The problem will likely involve a breakdown in the user’s relationship with a machine (Orr, ‘Talking about machines’)

n  Some common problems, jams in particular, often involve hardware failures or defects that can only be detected through expert human observation (i.e., cannot be ‘seen’ by machine sensors)

Technician-operator relationships n  Every customer of high-speed equipment that

had experience with other companies stated that the company’s service was far superior “They’re the only ones who know what they’re doing”

n  The relationship between their operators and our technicians was a key factor in this regard

n  Operators valued a close relationship with the technician above everything else

n  This relationship was often one of apprenticeship — of teaching and learning

–  Sometimes the learning occurs through observation rather than explicit teaching

–  Technicians commonly give out their mobile numbers, so they can be reached directly

–  This reduces UM calls, as problems can often be diagnosed and instructions given for resolving them over the phone

n  But little or none of this learning is shared beyond these local relationships

–  Experienced operators sometimes consult with colleagues they know from previous jobs in the industry, but this is naturally hit-and-miss with no systematic organisation

n  The result is that we have only local pockets of knowledge and practices that are potentially valuable for many customers

I've been doing this about sixteen years. I’ve worked on just about every machine that ever came out. I'm very very close with the techs, to the point where I'm almost as knowledgeable in a lot of things, because they teach me so much, to eliminate me calling them in crucial situations. They taught me to change belts, fusers, certain rollers, drums — I'd say pretty much about 60% of the maintenance I do myself. With the help of them through phone conversations — I have a direct line [mobile phone number] to each one of my techs. I talk to them first before I even place a [service] call. Unless it's really intricate, they pretty much talk me through everything.

Operator self-service (and other practices)

’I don’t like the machine to beat me’ Solving a power fault problem

Fuser roll swap-out

Operator self-service (and other practices)

Operator heuristics on high-speed printer “What can it hurt to clean off the dust, right?”

A series of codes (jams) —

�  09-9002 Feeder 1: remove sheets… �  12-9066 Finisher A: Jam in area 1a �  12-9078 Finisher A: Jam in area 3 � 12-9066 (again)

Making your own trouble?

•  Manager  replaces  corotron  assembly  …  ’Background  problem’  disappears,  as  does  ’streaking’  problem  

•  ’Problem’ discovered –  Discovered while reviewing prints –  Described as ’slight streaking’

•  What to do? –  Operator decides to clean the charge

corotrons •  What happens?

–  Problem gets worse; now described as ’background problem’

•  What to do now? –  Cleans corotrons again

•  Result: problem gets even worse! •  Operator’s next actions

–  Checks calibration (???), runs test prints –  But problem remains

•  Manager (responsible for scheduling) happens upon the scene, operator explains situation, manager pulls out corotron cleaning wand associated with cyan toner … and finds a small plastic piece/part hanging off end…

Making your own trouble?

•  Manager  replaces  corotron  assembly  …  ’Background  problem’  disappears,  as  does  ’streaking’  problem  

•  ‘Problem’ discovered –  Discovered while reviewing prints –  Described as “slight streaking”

•  What to do? –  Operator decides to clean the charge

corotrons •  What happens?

–  Problem gets worse; now described as “background problem”

•  What to do now? –  Cleans corotrons again

•  Result: problem gets even worse! •  Operator’s next actions

–  Checks calibration (???), runs test prints –  But problem remains

•  Manager (responsible for scheduling) happens upon the scene, operator explains situation, manager pulls out corotron cleaning wand associated with cyan toner … and finds a small plastic piece/part hanging off end…

Tip: To ensure proper cleaning of the pad, slowly pull out the wand until it reaches the end of the assembly; it will stop when it reaches the end.... CAUTION: To avoid damage to the corortron assembly, carefully and slowly pull out the Cleaning Wand....

Managing life in the field

Assessing machine/customer problems and deciding what to do is shaped by this socio-organisational context

Decisions about where to go, what to do and when to do it (including then actions related to diagnosis and repair) are commonly shaped more by the CSE’s local situation than by technical or operational protocols

n  The call queue: How many calls are in the queue? How many assigned to me? How much time can I actually spare here to do this versus that, given the calls that are holding?

n  The team’s situation: Who in my team (or another team) has asked for help? Should I help the other technician out or leave that to someone else? Is the technician asking for help someone who warrants it (rather than someone who asks too often)?

n  Travel: How far to go to this place versus that? In what order can I do these calls, given where they are?

n  Customer situation and characteristics: Is their machine truly down or can the problem be worked around for now? What kinds of jobs do they need to run? Is this customer going to make life difficult for me if I leave here without fixing everything, or if I don’t go there soon?

n  ……

Does anyone here know anything about what’s wrong with this machine? n  In office environments, the CSE often arrives to find that there is no one at

the location who knows anything about the problem –  Information sent to the CSE from the Customer Support Centre via their

Blackberry is necessarily cryptic and lacks context »  A ’reasoning about the evidence’ the CSR and caller developed together to achieve

some minimally shared understanding of (agreement about) the problem »  …and which the CSR then had to translate into that cryptic account, using information

and reasoning that is of course unavailable to the CSE

n  Detective work ensues, with the CSE typically looking over the fault and failure histories but often finding that this information also is lacking context, which is usually needed to truly understand the reported problem

n  The CSE might then resort to rummaging through discarded prints (seeking any evidence he can find) to see what might be learned from them

n  This can also happen at print shops like Kinko’s, where there are three shifts and a CSE arriving in the morning cannot find anyone on duty was has knowledge about a machine that is identified as ‘down’

A case in point… n  Message on CSE Blackberry said that a real estate office was

reporting a ’registration fault’ on their machine; the brief message from the Customer Support Centre also referred to copy quality (CQ)

n  But when the CSE arrived there was no one at the office to speak with about the problem (both employees who operate the machine were away), so the CSE looked through some discarded prints

–  Evidence of CQ or registration problems not apparent, however

n  The CSE then reviewed the machine’s recent fault history –  There are no registration fault codes –  The most recent code was a 024-747, which the CSE did not recognize

»  He looked it up on his PWS; it indicated a ’print instruction failure’ »  But that occurred six days earlier — hardly recent enough to have

caused the current problem, whatever that might be n  The CSE then checked the Fiery for a list of print jobs, to see what

they were running recently (’context’ again…) — but that too did not help resolve the ’What’s wrong here?’ puzzle

n  Then he checked the machine’s shutdown history –  There were ten 042-603 failures listed, occurring over the last three

days –  He looks up that code on his PWS — ’Suction/Ozone Filter Life End

Failure’ — and decides to follow the associated RAP…

*with the repair procedure being ’Execute job after re-setting paper size, feed tray, Duplex designation, output tray etc.’

A case in point… (continued) n  Following the RAP, the CSE removes a back panel to inspect the

ozone filter –  ’It’s not that dirty. I’ve seen worse.’ –  He decides to change it anyway — “If someone checks, they can’t say I

didn’t do anything.” –  He also resets the HFSI value for Ozone Filter to zero (still following the

RAP)

n  The CSE runs some test copies — and the first sheet jams in area 2; more test copies are run … it seems the machine is reading paper size incorrectly

–  ’Is the stack a little off? Or the guide not sensing properly?’ –  He inspects the stack tray, adjusts the paper with the guides, then runs some

more test copies from all trays without any problem n  The test copies are inspected for color registration – no registration

problems are evident … Paper stacking is checked again n  Then the machine operator arrives … and explains the problem he has

been experiencing –  There has been some skewing with jobs run from Tray 2 – there were no

registration fault codes, however –  When the operator phoned the Customer Support Centre, he stated he was

having skewing problems and mentioned registration as well, he thinks … the CSR pressed him for fault codes and he tried to explain he had none

n  The CSE explained the probable cause of the skewing problem – it had to do with paper size changes and the ’soft guides’ in Tray 2, which likely threw the alignment of the paper off a bit and caused some skewing … and he also explains he can come back and run a three-hour alignment program for the machine later if the problem persists

What exactly is the problem here? n  Digital side of a mid-sized printing operation in Chicago that mainly

serves the local Polish community n  CSE thought he was taking me there just to meet the owner and talk

about their experience with the machine … only after speaking on the phone with the shop did he learn that they had a problem and believed he was coming to fix it

n  When we arrived the secretary tried to describe the problem —something about the quality of the prints — and the CSE was a bit mystified … he and a product specialist had been there just the prior week to reset the registration

n  Looking at the prints lying on and near the machine, it appeared registration was okay but there was some slight (<1mm) skewing of the image, resulting in two-sided copies (including sheets of business cards) where the two sets of images were not aligned properly

n  The owner then arrived and launched into a very animated account of the problems he was having — not just skewing but some gloss differential (and some other things) … he showed us many flawed prints

n  The CSE called a product specialist and discussed the situation … the slight skewing problem was deemed ’normal’ for the machine… but the owner was having none of it — for him, this machine was not doing what he had been ’promised’ (by sales) it could do, and he could not meet his customers’ requirements

n  The CSE told the owner that a product specialist would meet with him the following week to try and resolve the situation — he also told the customer that ’grain long’ paper (grain is parallel to the paper's long side) was needed for the jobs they were running, not ’grain short’ (parallel to short side)

No CSE with whom I spoke was even aware of the company’s online support site, and I could not find an operator/user who knew anything about that resource.