Paper  presented  to  the  American  Evaluation  Association  Conference    27  October  2012,  Minneapolis,  USA  

DOI: 10.13140/2.1.3273.5043

Understanding  the  subjective:  eliciting  hidden  meaning  David  Roberts  

Abstract  Understanding  the  subjective  is  the  key  to  understanding  people’s  interactions  within  a  program.  However,  the  subjective  tends  to  be  ‘hidden’  especially  in  the  ‘objective’  environment  of  evaluation  interviews.  David  Roberts  has  been  using  projective  techniques  to  reveal  the  ‘hidden’  subjective  meanings  for  more  than  20  years.  This  paper  presents  research  that  explores  how  an  understanding  of  participants’  cognitive  processes  might  assist  us  to  assess  the  validity  of  interview  techniques.  Extensive  debriefing  of  people  who  had  participated  in  a  photo-­‐elicitation  exercise  showed  that  their  responses  and  answers  were  derived  from  abstract  generalizations  (or  ‘typifications’)  and  developed  afterwards  by  rationalization.  The  participants  used  such  typifications  to  help  them  to  think  about  the  subject  of  the  activity.  Identifying  the  participants’  typifications  revealed  information  about  their  attitudes  towards  the  topic.  Such  attitudes  are  not  a  predictor  of  behaviour  but  may  influence,  and  hence  help  to  explain,  their  interactions.  

Introduction      Many  evaluators  and  social  researchers  treat  participants’  reports  of  behaviour  and  attitudes  as  relatively  unproblematic.  There  is  an  implicit  and  sometimes  explicit  assumption  (Krosnik  personal  communication;  Tourangeau  1987)  that  what  a  person  ‘Says  About’  a  phenomenon  in  a  research  context  is  generally  a  reasonably  accurate  reflection  of  that  person’s  attitude  and  behaviour  outside  the  research  context.  

However,  evaluations  and  social  research  throw  up  many  examples  of  differences  between  what  people  ‘Say  About’  (themselves  and  their  experiences)  and  what  they  ‘Do’  (Bell  and  Buchanan  1966;  Brown  and  Gilmartin  1969;  Bryan  et  al  2009;  Dillman  1978;  Phillips  1972;  Strack  &  Martin  1987;  Schwarz  1987;  Tourangeau  1987).    An  individual  will  also  express  different  attitudes  and  descriptions  in  different  interviews  (Holloway  &  Jefferson  2003;  Jakobsen  2012;  Mannay  2010)  and  in  different  surveys  (Bryan  et  al  2009;  Dillman  1978;  Strack  &  Martin  1987;  Tourangeau  1987).  

Moreover,  we  can  all  think  of  many  occasions  when  what  individuals  ‘Say  About’  phenomena,  particularly  when  talking  about  themselves,  is  different  from  what  they  do.  

“I’m  going  to  tell  that  bully  what  I  think  of  her!”    or  even    “I  told  him!”    

“I  only  buy  healthy  food  for  my  kids.”      from  a  woman  with  a  large  case  of  diet  cola  in  her  shopping  trolley.  

Reports  about  what  people  saw  or  heard  are  also  problematic.    Any  policeman,  court,  or  lawyer  will  also  tell  you  that  people  are  very  unreliable  witnesses.    Eyewitness  identification  of  suspects  is  at  best  only  50%  accurate  (Brewer  2012).  We  generally  do  NOT  report  events  accurately,  even  less  so  when  it  comes  to  our  own  behaviour.      

  2  

Projective  Techniques  In  order  to  address  the  difference  between  what  we  SAY  and  what  we  DID  or  SAW,  Haire  (1950)  borrowed  projective  techniques  from  psychology  and  applied  them  in  market  research.    

Since  then  a  wide  range  of  projective  techniques  have  been  used  in  evaluation  and  research  (Boddy  2008;  Catterall  &  Ibbotson  2000;  Donoghue  2000).    Such  techniques  appear  to  provide  information  about  the  subjective  factors  that  influence  behaviour:  and  hence  provide  useful  insights  for  evaluators  and  researchers  (Catterall  &  Ibbotson  2000;  Durgee  1988;  Jenkins  et  al  2010;  Mannay  2010).  

The  most  common  techniques  are:  ‘picture  drawing’  (Mannay  2010,  Roberts  2005);  Sentence  Completion  (Jacques  2005);  vignettes  –  story  completion  (Jenkins  et  al  2010);  photo-­‐elicitation  (Hurworth  );  Thematic  Apperception  Testing  (Donoghue  2000;  Jacques  2005);  Role  Playing  (Jacobsen  2012;  Jacques  2005);  and  various  forms  of  association,  seeking  an  immediate  response  to  a  stimulus  (e.g.  words  or  pictures)  (Donoghue,  2000;  Jacques  2005).      

How  do  projective  techniques  work?  There  are  differences  between  the  techniques  but  they  all  present  participants  with  a  task:  e.g.  respond  to  a  stimulus,  finish  a  story,  act  a  role.  The  tasks  all  have  significant  elements  that  are  not  defined  or  are  ambiguous  (Boddy  2008;  Donoghue  2000).    The  absence  of  concrete  information  presented  to  participants  appears  to  require  the  participant  to  make  sense  of  the  situation  and  complete  the  task  using  their  personal  knowledge  (Catterall  &  Ibbotson  2000;  Kay  et  al  2008).    

Common  sense  supports  the  idea  that  participants  use  their  personal  knowledge  to  complete  the  task  but  there  is  little  research  in  this  area.    Indeed,  there  is  very  little  research  into  any  aspect  of  how  projective  techniques  work  (Boddy  2008;  Catterall  &  Ibbotson  2000;  Donghue  2000).  Unanswered  questions  include:  How  do  projective  techniques  work?  Why  do  they  sometimes  provide  different  answers  to  direct  questions?  Are  people  lying  when  there  is  a  difference  between  what  they  SAY  and  what  they  DO?    How  can  we  assess  the  value  of  data  provided  by  different  evaluation  interviews?  

Some  writers  have  speculated  that  classic  psychological  projection  provides  the  mechanism  for  projective  techniques  (Boddy  2008;  Donoghue  2000).    In  this  model  participants  ‘project’  aspects  of  their  personality,  attitudes  and  feelings  onto  the  task;  and  projection  is  a  defence  mechanism  that  reduces  the  stress  of  discussing  difficult  personal  issues.  However,  there  is  very  little  empirical  support  for  the  classical  theory  of  projection  (Lillienfield  et  al,  2007).  Such  a  theory  is  also  very  difficult  to  prove  or  disprove:  and  offers  very  few  answers  to  the  questions  posed  in  the  paragraph  above.        

Automatic  Behaviour  and  Schema  Fortunately,  cognitive  science,  education  theory  and  phenomenology  independently  provide  some  clues  as  to  what  might  be  happening  in  evaluation  interviews  and  surveys  (Bodenhausen  &  Wyer  1987;  Hastie  1987;  Schutz  1967;  Tourangeau  1987).    There  is  very  good  evidence  that  our  behaviour  and  thinking  is  predominantly  automatic  and  below  conscious  awareness  (Kahneman  2011).    Such  automatic  behaviour  is  rooted  in  

  3  

our  emotions  not  rationality  (Damasio  2008).1    More  importantly,  this  automatic  thinking  cannot  be  turned  off  and  exists  even  when  we  are  consciously  and  explicitly  thinking  (Damasio  2008;  Kahneman  2011).      

In  fact  very  little  of  our  thinking  is  explicit,  conscious  thought.2    Instead,  for  most  of  our  behaviour  we  respond  without  conscious  thought  using  mental  shortcuts  (described  variously  by  Bartlett  1932;  Bodenhausen  &  Wyer  1987;  Hastie  1987;  Kahneman  2011;  Schutz  1967;  Tourangeau  1987).3    There  are  a  wide  variety  of  names  for  such  mental  shortcuts:  ‘heuristics’  (Bodenhauser  &  Wyer  1987:9-­‐10,  21,  28;  ),  ‘scripts’,  ‘frames’  and  ‘concepts’  (Tourangeau  1987:154).    I  will  use  the  term  ‘schema’  (plural  ‘schemata’)  because  it  has  the  longest  history  and  broadest  application  (Bartlett  1932;  Conway  &  Holmes  2005;  Hastie  1987;  Hitch  2005;  Schutz  1967).  

Schemata  help  us  manage  our  world  without  having  to  consciously  think  about  such  things  as  how  far  and  how  fast  to  turn  the  steering  wheel  so  our  car  goes  round  the  curve  (Gladwell  2005);  or  to  speak  without  considering  what  we  want  to  say.    In  any  particular  situation,  our  brains  provide  us  with  an  automatic  behaviour,  a  response,  or  a  ‘recipe  for  action’  (Schutz  1967)  for  that  situation.    Our  responses  are  subjective  in  that  they  come  from  within  and  are  not  based  on  rational  thought.      

However,  schemata  are  more  than  simple  behavioural  responses,  they  also  contain  implicit  knowledge  (assumptions)  that  allows  us  to  make  sense  of  our  situation.    Such  implicit  knowledge  is  generalised  and  abstract  (Brewer  1987;  Rutherford  2005)  and  it  informs  and  underlies  our  thinking  and  speech  in  that  situation  (Quenza  2005;  Schutz  1967).    A  schema  constrains  what  details  are  recalled  from  memory  (McVee  et  al  2005;  Hick  2005;  Schutz  1944).    Schemata  mediate  our  interactions  with  the  world;  what  we  perceive  and  how  we  respond.  4  

There  is  of  course  a  role  for  conscious  reflection.    We  may  choose  to  test  our  assumptions  and  consciously  consider  alternatives,  solve  problems  and  so  on.    However,  our  ability  to  do  so  is  limited  (Fischer  et  al  2008)  and  our  conscious  thinking  is  constrained  by  the  assumptions  we  are  not  testing  and  the  memories  that  can  be  recalled  within  the  active  schemata  (Damasio  2008;  Kahneman  2011).5  

If  schemata  are  present  in  all  our  thinking  and  action,  are  they  present  in  evaluation  interviews?  If  so,  the  information  presented  in  evaluation  interviews  needs  to  be  analysed  to  identify  the  schema  that  is  operating,  at  what  level  of  consciousness  and                                                                                                                  1 Hence the title of Damasio’s book ‘Descartes’ Error’. ‘Cogito ergo sum’ (“I think therefore I am’) is a fallacy. Fundamentally we are emotional animals not rational beings. 2 It takes far too much effort to question our assumptions and think about what we are doing (Fischer et al 3 The evidence for the use of such mental shortcuts is among the strongest in cognitive science. 4 There is a range of different conceptions of schema (Quenza 2005). At one end schemata are seen as relatively mechanistic, static cognitive structures and at the other end they are seen as fluid, dynamic structures that develop and respond to experiences and interactions underlie all thinking and influence all information processing. The evidence supports the more dynamic formulation (Bartlett 1932; McVee et al 2005; Quenza 2005; Rutherford 2005).

Schema can also be embedded within each other to form multi-layered schemata (McVee et al 2005). 5 One implication of the psychological and cognitive research is that even ‘thoughtful’ ‘considered’ answers may be predicated on the active schemata and the knowledge available within that schema. However, that is a matter for further research.

  4  

how  and  when  that  schema  might  be  activated  in  other  situations.  The  answers  to  such  questions  require  an  extended  program  of  research.  

This  study  is  preliminary  research  only  to  test  whether  schemata  operate  in  a  projective  technique  and  to  explore  some  ideas  about  how  we  might  analyse  such  data.      

Methods  The  research  consisted  of  eight  cognitive  interviews  with  experienced  evaluators.      Each  interview  had  two  parts:  an  ‘originating  interview’  and  a  ‘debrief.’    In  the  originating  interview  I  used  a  technique  known  as  Thematic  Apperception  Testing  (or  TAT)  (Donoghue  2000;    Jacques  2000)6  ,  though  I  prefer  to  see  it  as  a  variation  of  photo-­‐elicitation  (Hurworth  2005).  The  technique  was  chosen  to  provide  a  null  hypothesis  because  it  encouraged  participants  to  use  a  schema.  I  argue  if  could  not  see  evidence  of  schema  using  this  technique  then  the  theory  of  schema  probably  does  not  apply.  

The  reason  I  chose  experienced  evaluators  was  I  believed  their  experience  of  interviews  would  make  them  less  susceptible  to  using  schema  and  they  would  provide  valuable  insights  in  the  cognitive  debrief  process.  

Each  participant  was  shown  a  range  of  16  portrait  photographs  (See  Figure  1).    Each  participant  was  asked  to  select  one  photograph  that  represented  the  person  who  had  commissioned  an  evaluation  and  was  most  likely  to  use  the  results  of  the  evaluation  well:  and  an  evaluation  commissioner  who  was  least  likely  to  make  good  use  of  the  evaluation.    Each  participant  was  then  asked  to  describe  the  two  persons  represented  in  the  photographs  and  how  those  persons  would  behave  when  they  received  an  evaluation  report.      

Figure  1.  Example  of  photograph    

                                                                                                               6 The term ‘Thematic Apperception Testing’ is misleading in this context (cf. psychology) because its use in evaluations, market research and social research is not ‘testing’ in any meaningful sense of the term.

  5  

 

Each  interview  was  videoed.      After  a  short  break  at  the  conclusion  of  each  interview7  the  video  was  used  to  conduct  a  cognitive  debrief  with  the  participant  (Beatty  &  Willis  2007).  8  The  video  was  replayed  on  a  laptop  computer.    I  moved  so  I  was  sitting  next  to  the  participant  and  we  could  watch  the  video  together.    I  stopped  the  video  at  different  points  to  ask  the  participant  what  s/he  had  been  thinking  or  how  s/he  had  made  the  judgements  expressed  in  the  interview.  I  also  instructed  the  participant  to  stop  the  video  whenever  they  wanted  to  discuss  what  was  happening  in  the  interview.    The  cognitive  interview  was  also  videoed.  

Results    Some  participants  selected  the  two  photographs  in  less  than  20  seconds.    Other  participants  sometimes  appeared  to  take  some  time  to  settle  on  the  two  portrait  photographs.    However,  in  the  debrief  six  participants  reported  either  that  they  actually  made  the  decision  very  quickly  (in  seconds)  and  then  spent  some  time  ‘testing’  the  judgement,  or  that  they  had  very  quickly  narrowed  the  decision  down  to  one  or  two  photographs.      

                                                                                                               7 The break was necessary to load the video onto the computer so it could be replayed: and I usually offered the participant a drink in the break. The combination of the break, the change in my position were all designed to create a different context and minimize the influence of the previous interview context on the cognitive interview. 8 The use of video offered me the opportunity to conduct the originating interview without disruption and to recreate graphically for the participant what actually happened in the originating interview, rather than relying on the participants’ memory of the interview (cf. Beatty & Willis 2007).

  6  

Some  participants  selected  the  same  portraits  as  each  other  but  generally  they  chose  different  portraits.    In  two  instances,  a  portrait  selected  as  ‘best’  commissioner  of  an  evaluation  by  one  participant  was  selected  as  the  ‘worst’  by  another  participant.      

The  participants’  rationales  for  selecting  the  photographs  were  often  phrased  as  intuitive  judgements  (“I  felt  …”):  and  explained  in  terms  of  specific,  sometimes  very  subtle,  attributes  of  the  portrait.    For  example,  it  was  suggested  that  the  man  portrayed  in  Figure  1  had  limited  ability  or  desire  to  pay  attention  to  the  report  or  to  what  the  evaluator  had  to  say.    Phrases  used  to  describe  him  included:  ‘limited  bandwidth’;  ‘busy’  and  ‘the  computer  is  a  barrier.’  Comments  about  the  ‘best  commissioner’  similarly  referred  to  subtle  cues  in  the  portrait.    For  example,  a  photograph  of  a  woman  on  a  park  bench  was  reportedly  selected  because  ‘she  is  relaxed’;  ‘wearing  a  casual  jacket’;  ‘I  felt  a  connection’;  and  the  setting  was  characterised  as  ‘away  from  distractions’.  One  participant  also  referred  to  the  physical  attractiveness  of  the  person  portrayed  in  the  photograph.      

One  participant  reported  that  she  had  a  specific  set  of  characteristics  in  mind  when  choosing  the  photograph  and  saw  them  in  the  photographs.    However  the  characteristics  were  quite  subtle.    One  portrait  was  selected  because  it  reminded  her  of  a  person  of  a  different  gender.    In  fact  most  participants  reported  that  the  selection  (and  later  the  description)  was  based  on  details  that  reminded  them  of  one  or  more  of  their  own  personal  experiences.  In  some  cases,  one  experience  with  a  strong  emotional  impact  drove  the  selection.    In  other  cases  participants  reported  that  it  was  an  amalgam  of  repeated  experiences:  and  in  other  cases  the  participants’  referred  to  recent  experiences.    

When  asked  to  describe  the  behaviour  and  motives  of  the  person  depicted  in  the  photograph,  some  participants  expressed  discomfort  at  ‘stereotyping’.    Nevertheless  they  all  did  as  they  were  asked  and  described  the  character  and  behaviour  of  the  person  portrayed  in  the  photograph.    The  descriptions  were  quite  specific  and  included  comments  on  attitudes  and  management  style.  Most  participants  picked  up  the  photograph  of  the  person  they  were  describing  and  pointed  to  details  in  the  photograph  that  supported  the  participant’s  interpretation  of  the  portrayed  person’s  behaviour.  

Similarly,  in  the  cognitive  debrief  all  participants  referred  to  specific  details  in  the  photograph  to  support  their  descriptions  of  behaviour.      

Discussion  It  is  important  to  note  that  there  was  reasonable  variation  between  participants  in  the  selection  of  photographs.  Even  when  participants  described  similar  characteristics  they  usually  chose  different  photographs  to  represent  the  persons  described.    The  sample  is  too  small  to  draw  any  definite  conclusions  but  there  is  no  evidence  that  there  might  be  a  shared  understanding  of  what  the  two  ‘types’  of  people  might  look  like:  or  indeed,  that  some  quality  intrinsic  to  a  photograph  stimulated  the  participants’  selection  of  that  photograph.  Instead  each  participant  appears  to  have  had  their  own  individual  interpretation  of  what  characteristics  most  closely  represented  the  ‘types’  they  were  asked  to  select.    

  7  

Selection  from  implicit  knowledge  Participants  were  asked  to  select  two  portraits  from  an  array  of  16  photographs.    Each  photograph  contained  an  enormous  amount  of  detail.  Yet  the  initial  judgements,  on  which  the  selections  were  based,  were  made  very  quickly.    When  asked  to  explain  their  judgements,  the  participants  referred  to  subtle  similarities  between  the  portrait  and  participants’  and  experiences  of  ‘people  like  that’.  Importantly,  the  experiences  cited  were  all  salient  experiences  though  the  reason  for  such  saliency  varied.    

The  participants  could  not  possibly  have  been  consciously  aware  of  all  the  details  in  each  photograph  in  the  time  they  took  to  make  those  judgements.    Nor  did  they  need  to  do  so.    Instead  the  participants  used  implicit  knowledge  to  make  those  judgements  (see  Damasio  2005;  Kahneman  2011;  Rutherford  2005;  Schutz  1967).      

Such  judgements  were  an  expression  of  their  subjective  understandings  about  ‘people  like  that:’  about  types  of  people.  In  this  case,  the  type  of  person  is  a  group  of  people  who  is  not  like  the  participant.    Schutz  (1967)  talks  about  such  types  as  the  ‘generalised  other’  or  ‘They’:  people  who  are  known  to  exist  but  who  are  believed  to  have  a  set  of  understandings  different  from  oneself.  9      The  participants  in  this  research  used  such  typifications  because  it  allowed  them  to  respond  to  the  task  they  had  been  given.  

The  participants’  responses  to  their  task  occurred  before  explicit  thinking.    They  are  part  of  a  class  of  thinking  (and  behaviour)  known  variously  as  ‘pre-­‐suppositional”  (Mead  1912);  ‘pre-­‐phenomenal’  and  ‘pre-­‐predicative’  (Schutz  1967)  and  ‘System  1’  thinking  (Kahneman  2011).    Such  responses  are  ‘Doing’  in  the  present  moment  (now)  and  in  the  current  context  (here).10    

In  other  words  the  selection  process  had  all  the  characteristics  of  schema  thinking.    Participants  identified  a  ‘type’  very  quickly  based  on  very  little  information  and  sometimes  very  subtle  cues  (posture,  situation,  etc.).    The  participants  could  only  have  made  the  judgements  using  implicit  knowledge  (subjective  perceptions),  they  simply  did  not  have  time  to  consciously  reflect  on  all  the  details  in  the  all  the  portraits.    The  knowledge  they  used  to  make  the  judgements  was  what  Schutz  (1967)  called  a  ‘typification,’  generalised  knowledge  (in  this  case  about  people)  of  a  type.  

Description  developed  in  interview  The  descriptions  people  gave  of  the  person  represented  in  the  photograph  had  a  different  quality.  In  the  descriptive  stage  of  the  interview,  participants  looked  at  the  

                                                                                                               9 Other techniques and interviews trigger may trigger other orientations. Jenkins et al (2010) suggest that vignettes may lead to a participant placing himself or herself in the position of the protagonist (a ‘person like me’) and then use that knowledge to complete the story. However, if the participant does not identify with the circumstances of the protagonist, they may not see themselves as similar to the protagonist.

Jenkins et al (2010) uses Schutz’s (1967) notion of ‘Thou’ to explain how participants relate to the protagonist in a vignette. For Schutz (1967) a ‘Thou’ orientation occurs when a person is aware of another but the other is unaware of them. Jenkins et al (2010) argues both that such a situation pertains in a vignette: and that the participant places himself or herself in the position of the protagonist. I think the two orientations are incompatible. In any case I think it is more informative to recognize that the participants may see the protagonist as a ‘person like me’ and at other times might see a protagonist as a ‘person I know’, or even as a ‘They’ a ‘person like that.’ 10 ‘Doing’ can be contrasted with Talcott Parsons’ notion of ‘Action’ as intentional and explicit behavior.

  8  

interviews  to  find  and  refer  to  details  that  supported  their  initial  judgements.    Some  of  the  participants  indicated  that  they  had  used  their  existing  judgements  as  the  basis  for  developing  their  descriptions  and  that  they  developed  their  descriptions  in  the  course  of  the  interview.  I  suspect  that  all  participants  did  so.      

The  idea  that  the  descriptions  are  socially  constructed  in  interviews  is  a  fundamental  tenet  of  post-­‐modernist  thinking  (Kvale  1996;  Schostak  2006).    There  is  strong  evidence  from  other  fields  to  support  such  a  notion.  Brockmeier  (2010)  shows  that  memory  is  not  like  a  data  store  where  specific  information  is  waiting  to  be  retrieved:  instead  memory  is  constructed  when  it  is  needed.    Specific  memories  are  constructed  from  generalised  memories  and  the  details  filled  in  as  needed  for  the  specific  context  (Rutherford  2005).11    Analysis  of  video  ethnography  and  interviews  also  supported  the  notion  that  explicit  description  is  constructed  at  the  time  it  is  required  (Knoblauch  &  Schnettler  2012;  Mondada  2012).    Schwarz  and  Hippler  (1987)  show  that  respondents  glean  information  about  social  norms  from  the  questions  asked  and  then  use  that  information  to  construct  their  responses.    In  other  words  conscious,  or  explicit,  thinking  and  memory  are  constructed  in  the  moment;  and  follow  on  and  may  be  based  on  the  initial  schema  (Hitch  2005;  Kahneman  2011;  Schutz  1967).12      

Different  Thinking  Systems  It  seems  clear  that  there  were  two  distinct  ways  of  thinking  reflected  in  the  interview.  In  the  selection  of  photographs,  the  participants  were  responding  before  explicit  thinking.    They  were  expressing  subjective  perceptions  and  using  implicit  knowledge  to  make  the  judgements  about  which  photographs  to  choose.      

In  the  descriptive  part  of  the  originating  interview,  participants  ‘talked  about’  two  different  ‘people  like  that’.    They  were  consciously  reflecting  on  the  judgements  they  had  already  made  and  ‘rationalising’  their  judgements.    In  doing  so  they  were  treating  the  phenomena  they  were  describing  as  cognitive  objects  and  manipulating  those  objects  in  their  conscious  thinking  (Kahneman  2011).    The  participants  assumed  a  particular  orientation  towards  those  others  as  ‘They’  (Jenkins  et  al  2010;  Schutz  1944;  1967).    The  explicit  conversations  about  ‘They’  occurred  separately  from,  but  used  as  a  base,  the  subjective  responses  elicited  in  the  selection  phase  of  the  interview.    

Nature  of  data  What  then  is  the  nature  and  value  of  the  data  obtained  in  this  research.    The  data  tells  us  nothing  about  the  people  represented  as  ‘They’.    It  only  tells  us:  

• what  decisions  the  participants  made  about  ‘They’  and  • what  the  participants  ‘Say’  they  perceived    about  ‘They’  at  that  moment  in  the  

context  of  the  interview.    

                                                                                                               11 One can of course probe and foster more detailed recall by using associations (Geiselman et al 2003). It is not at all clear how the detailed memories are stored or retrieved but it is clear that such recall is secondary to the immediate generalized memory. 12 This is not to suggest that individuals cannot go beyond the initiating schema when they think explicitly. In fact we clearly can and do (Kahneman 2011). However our ability to question assumptions and test the underlying schema is severely constrained and we tire quickly when we do such mental work (Fischer et al 2005; Kahneman 2011)

  9  

The  first  is  about  what  participants  ‘DID’  and  the  second  is  what  they  ‘SAID.’  There  may  in  fact  be  layers  of  both  implicit  and  explicit  thinking  between  what  the  participants  DID  and  what  they  SAID.    There  can  be  a  distinction  between  what  people  think  explicitly  and  what  they  express  to  a  particular  audience  (e.g.  Briggs  1986;  Jakobsen  2102;  Kvale  1996;  Schostak  2006).13      Even  in  surveys,  people  will  exaggerate  or  enhance  the  image  they  present  (Fisher  1993;  Jakobsen  2012;  Tourangeau  &  Yan  2007).    What  people  “Say”  is  mediated  by  the  presentation  of  self  and  the  social  context  (Jakobsen  21012;  Mead  1912;  Kvale  1996;  Schostak  2006).    Though  interestingly  Fisher  (1993)  shows  that  projective  techniques  reduce  social  desirability  bias  on  issues  that  relate  to  social  outcomes.  

Multiple  stocks  of  knowledge  The  expressed  presentations  are  not  necessarily  deliberate  misrepresentations  but  a  consequence  of  our  limited  capacity  for  conscious  thought  (Hicks  2005).    

“We  are  aware  of  only  a  few  ideas  at  once”    (Hastie  1987:53)  

As  a  result  we  are  often  unaware  of  the  differences  and  contradictions  between  behaviours  and  memories  (Bodenhausen  &  Wyer  1987;  Rutherford  2005).    We  can  and  do  hold  and  express  a  number  of  quite  different  and  contradictory  attitudes  and  opinions  (Tourangeau  1987).    Our  ‘knowledge’  of  ourselves  and  the  world  around  us  is,    

“…  not  homogenous;  it  is  (1)  incoherent,  (2)  only  partially  clear,  and  (3)  not  at  all  free  from  contradictions.”  (Schutz,  1944:500)  

In  other  words  one  cannot  assume  that  what  an  individual  ‘Says’  in  one  context  is  consistent  with  what  that  person  might  ‘Say’  in  another  context,  or  that  it  is  likely  to  be  consistent  with  what  they  might  ‘Do.’    Indeed,  if  one  accepts  that  ‘Doing’  is  situational  (Mead  1912)  one  cannot  even  assume  that  behaviour  will  be  consistent  across  contexts.  

Behavioural  Preferences  However,  it  is  a  truism  of  psychology  that  the  best  indicator  of  a  person’s  behaviour  is  that  person’s  past  behaviour,  especially  when  that  person  is  under  pressure  and  has  little  time  to  consciously  choose  or  modify  their  behaviour.    People  tend  to  repeat  behaviours  across  different  contexts.    If  so,  one  might  ask  if  those  behavioural  preferences  are  the  result  of  certain  schemata  being  preferred.  

For  evaluators,  the  question  might  become  ‘How  can  we  identify  preferred  schema?’    ‘What  techniques  are  most  likely  to  reveal  such  preferred  schema?’      

Conclusion  This  preliminary  research  indicates  that  participants  exhibited  two  different  thinking  systems  in  the  interview.      The  choice  of  photograph  portraits  has  the  characteristics  of  schema  thinking.    Participants  used  implicit  knowledge  or  intuition  to  make  the  selections.    In  the  second  phase  of  the  interview  participants  developed  an  explicit  explanation  of  their  selections.    The  explicit  knowledge  was  created  in  the  interview  and  developed  from  the  implicit  knowledge  reflected  in  the  original  judgement.  

                                                                                                               13 Jakobsen (2012) cogently argues that people make different representations in different types of social context, especially when being asked to make judgements

  10  

Both  systems  of  thinking  related  to  a  ‘generalised  other’  a  ‘They’.    The  participants’  had  no  information  about  the  person  depicted  in  the  photograph.  Nevertheless  the  participants’  ascribed  characteristics  and  behaviour  to  that  ‘person’  based  on  the  participants’  experience  of  other  people.  Such  data  tells  us  very  little  about  how  those  others’  might  behave  but  it  does  provide  information  about  how  the  participants’  perceive  such  ‘others.’  

The  study  suggests  that  understanding  how  participants  generate  answers  to  questions  is  an  important  step  in  understanding  and  analysing  the  information  provided  in  evaluation  interviews.      

References  Bartlett,  F.C.  (1932).  Remembering:  A  Study  in  Experimental  and  Social  Psychology.  Cambridge,  England:  Cambridge  University  Press.  

Beatty,  P.,  &  Willis,  G.  (2007).  Research  Synthesis:  The  Practice  of  Cognitive  interviewing.  Public  Opinion  Quarterly,  71:  2,  287-­‐311.  

Bodenhauser,  G.  &  Wyer,  R.  (1987).  Social  Cognition  and  Social  reality:  Information  Acquisitions  and  use  in  the  laboratory  and  the  real  world.  in  Hippler,  H-­‐J.,  Schwarz,  N.,  &  Sudman,  S.  (eds),  Social  Information  Processing  and  Survey  Methodology,  New  York,  Springer-­‐Verlag.  

Brewer,  W.    (1987).  Schemas  versus  mental  models  in  human  memory.  In  P.  Morris  (Ed.),  Modelling  cognition  187-­‐197.  Chichester:  Wiley  

Briggs,  C.  (1986).  Learning  how  to  ask:  A  sociolinguistic  appraisal  of  the  role  of  the  interview  in  social  science  research.  Cambridge,  Cambridge  University  Press  

Brockmeier,  J.  (2010).  After  the  Archive:  Remapping  Memory.  Culture  Psychology,  16,  5,  5-­‐35.  

Damasio,  A.  (2005)  Descartes'  Error:  Emotion,  Reason,  and  the  Human  Brain,  Penguin  

Fischer,  P.;  Greitemeyer,  T.;  Frey,  D.  (2008)  “Self-­‐regulation  and  selective  exposure:  The  impact  of  depleted  self-­‐regulation  resources  on  confirmatory  information  processing.”  Journal  of  Personality  and  Social  Psychology.  Vol.  94(3),  March  2008,  pp.  382-­‐395        

Fisher,  R.  (1993)  Social  Desirability  Bias  and  the  Validity  of  Indirect  Questioning.    Journal  of  Consumer  Research,  20,  303-­‐315.  

Geiselman,  R.,  Fisher,  P.,  MacKinnon,  D.,  &  Holland,  H.  (2003)  Enhancement  of  Eyewitness  memory  with  Cognitive  Interview.  in  Fielding,  N.  (ed.)  Interviewing,  Vol.  II,  London,  Sage  Publications  Ltd.  pp.  3-­‐18.  (Originally  published  in  American  Journal  of  Psychology,  vol.  99,  no.  3,  1986,  385-­‐401)  

Haire,  M.  (1950).  Projective  Techniques  in  Marketing  Research.  The  Journal  of  Marketing,  XIV:5,  649-­‐656.  

  11  

Hastie,  R  (1987)  “Information  Processing  theory  for  the  Social  Researcher”  in  Hippler,  H.;  Schwarz,  N.  and  Sudman,  S.  (eds)  Social  Information  Processing  and  Survey  Methodology,  NY,  Springer-­‐Verlag  

Hitch,  G.  (2005)  “Working  Memory”  in  Braisby  N  &  Gellatly  A  (eds.)  Cognitive  Psychology,  Oxford,  OUP  Hurworth,  R.,  Clark,  E.,  Martin,  J.  &  Thomsen,  S  (2005).  The  use  of  photo-­‐interviewing:  Three  examples  from  health  evaluation  research.  Evaluation  Journal  of  Australasia,  4(1&2),  52-­‐62  

Jakobsen,  H  (2012)  “Focus  groups  and  methodological  rigour  outside  the  minority  world:  making  the  method  work  to  its  strengths  in  Tanzania,”  Qualitative  Research,  Vol  12,  No.2,  2012,  pp.111-­‐130  

Kahneman,  D.  (2011)  Thinking  Fast:  Thinking  Slow,  New  York,  Farrar  Straus  &  Giroux  

Knoblauch,  H.  &  Schnettler  (2012)  “Videography:  analyzing  video  data  as  a  ‘focsed’  ethnographic  and  hermeneutical  exercise”,  Qualitative  Research,  Vol  12,  No.3,  pp.334-­‐356.  

Kvale,  S.  (1996).  Interviews:  An  Introduction  to  Qualitative  Research  Interviewing,  Thousand  Oaks,  Sage  

Lilienfeld,  S.,  Wood,  J.,  Garb,  H.  (2000).  The  Scientific  Status  of  Projective  Techniques.  Psychological  Science  in  the  Public  Interest,  1,  2,  27-­‐66.  

Mannay,  D.    (2010).  Masking  the  familiar  strange:  can  visual  research  methods  render  the  familiar  setting  more  perceptible?  Qualitative  Research,  10,  1,  91-­‐111  

McVee, M. Dunsmore K,  Gavelek,  J.  “Schema  Theory  Revisited,”  Review  of  Educational  Research  Winter  2005,  Vol.  75,  No.  4,  pp.  531–566  

Mead,  G.  (1912)    “The  Mechanism  of  Social  Consciousness”    The  Journal  of  Philosophy,  Psychology  and  Scientific  Methods,  Vol.  9,  No.  15  (Jul.  18),  pp.  401-­‐406  

Mondada,  L.  (2012)  “Video  analysis  and  the  temporality  of  inscriptions  within  social  interaction:  the  case  of  architects  at  work,”    Qualitative  Research,  Vol  12,  No.3,  pp.304-­‐333.  

Roberts,  D.  2005,  “Recreating  Experience:  Improving  the  validity  of  data,”  Evaluation  Journal  of  Australasia,  Vol.  4,  1  &  2:  44-­‐51,  March/April    

Rutherford,  A.  (2005)  “Long-­‐term  Memory:  encoding  to  retrieval”  in  Braisby  N  &  Gellatly  A  (eds.)  Cognitive  Psychology,  Oxford,  OUP  

Schostak,  J.  (2006)  Interviewing  and  Representation  in  Qualitative  Research,  Maidenhead,  Open  University  Press.  

Schutz,  A.  (1944)  “The  Stranger:  An  Essay  in  Social  Psychology”,  American  Journal  of  Sociology,  Vol.  49,  No.6  (May),  pp.  499-­‐507  

  12  

Schutz,  A.  (1967),  The  Phenomenology  of  the  Social  World.  Evanston,  IL:  Northwestern  University  Press.  

Schwarz,  N.  &  Hippler,  J-­‐J.  (1987).  What  response  scales  may  tell  your  respondents:  informative  functions  of  response  alternatives.  Hippler,  H-­‐J.,  Schwarz,  N.  &  Sudman,  S.  (eds).  Social  Information  Processing  and  Survey  Methodology,  New  York,  Springer-­‐Verlag  

Tourangeau,  R  .,  (1987).  “Attitude  Measurement:  A  Cognitive  Perspective”  in  Hippler,  H-­‐J;  Schwarz,  N  &  Sudman  S  (eds),  Social  Information  Processing  and  Survey  Methodology,  New  York,  Springer-­‐Verlag.  

Tourangeau,  R.  &  Yan,  T.  (2009)  Sensitive  Questions  in  Surveys.  Psycholgical  Bulletin,  133,  5,  859-­‐883