Post on 23-Jan-2023
What is “Benchmarking”• Depends on context
– Carpentry, computer speed, statistics
• Essentially “comparing to a standard”– Self in past– External best practice/professional standard
– Other empirical data
Benchmarking Surveys• Emphasis on empirical comparisons
– Our survey estimates behavior A to be XX%. Their survey estimates behavior A to be XY%.
• Differences v. evaluation– There’s no “answer key”
• Difficulty of gold standards– Bigger not necessarily better– Method more important than size– The more topics you cover the coarser your measure on ea.
How do Surveys Differ?” Interactive Part”
• Question wording• Mode• Sampling method and sampling frame
• Weighting/analysis/estimation• Others?
How to Consider These All at Once
• Total Survey Error (TSE) framework– Heuristic framework – Developed by Groves and colleagues
•Groves, 1989; Groves et al 2004, 2007• May not lead to obvious estimation techniques for all error sources– See Little and Rubin missing data approaches
• Not our concern today
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
Measurement Representation
Respondents
Dual Inference Process Groves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Survey Statistic
Edited Data
Statistic-Specific
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
CoverageError
Measurement Representation
Respondents
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
CoverageError
Sampling Error
Measurement Representation
Respondents
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
CoverageError
Sampling Error
Measurement Representation
Respondents
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Nonresponse Error
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
CoverageError
Sampling Error
Measurement Representation
RespondentsAdjustment Error
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Nonresponse Error
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
Validity
CoverageError
Sampling Error
Measurement Representation
RespondentsAdjustment Error
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Nonresponse Error
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
Validity
Measurement
Error
CoverageError
Sampling Error
Measurement Representation
RespondentsAdjustment Error
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Nonresponse Error
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Construct
Measurement
Response
Population Mean
Sampling Frame
Sample
Validity
Measurement
Error
CoverageError
Sampling Error
Measurement Representation
RespondentsAdjustment Error
ProcessingError
Dual Inference ProcessGroves, et al., (2009), Survey Methodology
Post-surveyAdjusted Data
Nonresponse Error
Survey Statistic
Edited Data
i
iY
iy
Y
CY
sy
ry
rwy
rwy
ipy
Two CHIS Benchmarks (maybe)
• National Health Interview Survey (NHIS)– Conducted by US Census Bureau for Nat’l Center for Health Statistics (CDC)
– “Granddaddy” to CHIS– Ongoing since 1956
• LA County Health Survey (LA CHS)– Conducted by Abt/SRBI for LA County Department of Health
– Data collection years 1999-2011
Errors of NonobservationTSE Feature CHIS NHIS LA CHSCoverage Error
Dual frame LL/Cell RDD frame
Area probability frame (listed HUs)
Dual frame LL/Cell RDD frame
Sampling Error
Stratified by County, Oversamples race, age, etc.
Oversamples race, other?
REVIEW
Nonresponse Error
Issue11s with contacting people by phone
Issues with contacting people at home
Issues with contacting people by phone
Adjustment Error
Dual frame compositing and other adjustments
Imputation
Weighting and adjustments
Imputation
Dual frame compositing and other adjustments
Imputation
Errors of ObservationTSE Feature CHIS NHIS LA CHSValidity (of the construct type)
How do we develop and analyze our constructs from our survey items
Ditto-Questionnaire development with experts-Cognitive testing, etc.
Ditto-Questionnaire development with experts-Cognitive testing, etc.
Measurement Error
Mode issuesInterviewers more distantRecency effects on phone(note: mode crosses error source)
Interviewers present for sensitive questions
Recency effects on phone
Processing Error
CATI must work right“Upcoding” rulesData editing rules
CAPI must work right“Upcoding” rulesData editing rules
CATI must work right“Upcoding” rulesData editing rules
Other Reading on Total Survey Error
• Deming (1944) On Errors in Surveys• Hansen, Hurwitz, & Madow (1966) • Kish (1965)• Lessler & Kalsbeek (1992) Nonsampling Error in
Surveys• Little & Groves (forthcoming…eventually) Total Survey Error: Missing Conceptual Components and Design-Based/Model-Based Viewpoints Journal of Official Statistics, From 2011 Morris Hansen Lecture.
• Weisberg, Herbert F. The Total Survey Error Approach: A Guide to the New Science of Survey Research