Cognitive Interviewing

A Tool for Improving Questionnaire Design
Gordon B. Willis
Cognitive Interviewing
September 2004 | 352 pages | Sage US
Create Flyer

If you’re in North America, please visit our Sage College Publishing website to purchase or sample this book:

Go to College Publishing Website

Description

"As both an academic instructor in questionnaire design and a research design methodologist for the federal government, I feel this book is very timely, useful for students and practitioners, and unique in its use of real world practical examples that most everyone can relate."
—Terry Richardson, General Accounting Office

"The combination of theory and practical application will make this a useful book for students as well as professionals who want to learn how to incorporate cognitive interviewing into the questionnaire design process."
—Rachel Caspar, RTI International

The design and evaluation of questionnaires—and of other written and oral materials—is a challenging endeavor, fraught with potential pitfalls. Cognitive Interviewing: A Tool for Improving Questionnaire Design describes a means of systematically developing survey questions through investigations that intensively probe the thought processes of individuals who are presented with those inquiries. The work provides general guidance about questionnaire design, development, and pre-testing sequence, with an emphasis on the cognitive interview. In particular, the book gives detailed instructions about the use of verbal probing techniques, and how one can elicit additional information from subjects about their thinking and about the manner in which they react to tested questions. These tools help researchers discover how well their questions are working, where they are failing, and determine what they can do to rectify the wide variety of problems that may surface while working with questionnaires.

Cognitive Interviewing is ideally suited as a course text for advanced undergraduate and graduate research courses across the social sciences. Professional researchers and faculty in the social sciences, as well as practice fields such as medicine, business, and education, will also find this an invaluable reference for survey research. There is no other book on the market that covers cognitive interviewing as applied to questionnaire design.



Contents

ORIENTATION AND BACKGROUND

  • Introduction to Cognitive Interviewing
  • What is (and isn't) cognitive interviewing?
  • A broader perspective: The cognitive testing process
  • Setting the Stage: First Principles of Questionnaire Design
  • Questionnaire-related (and other) sources of error in self-report surveys
  • Why do survey questions produce response error? A sociolinguistic perspective
  • How can we avoid problems in survey questions?
  • If we are proficient designers, why do we need to test questions?
  • Chapter summary
  • Exercise: Question evaluation
  • The CASM Approach to Questionnaire Design
  • Origins of CASM
  • Applications of Basic CASM Research
  • Cognitive interviewing as Applied CASM Research
  • Chapter summary
  • Cognitive Interviewing in Practice: Think-Aloud, Verbal Probing, and Other Techniques
  • Think-Aloud interviewing
  • Verbal Probing Techniques
  • Concurrent versus retrospective interviewing
  • Choosing between think-aloud and probing techniques
  • Vignettes, card sorts, and field-based probes
  • Chapter summary

THE INTRICACIES OF VERBAL PROBING

  • Developing Standard Cognitive Probes
  • Examples of cognitive probing
  • Logical and structural problems
  • A systematic approach to probe development: The Question Pitfalls Model
  • Chapter summary
  • Exercise: Using the QAS to develop probe questions
  • Beyond the Standard Model of Verbal Probing
  • A classification of probe types
  • Proactive versus Reactive probing
  • Standardized versus free-form probes
  • Chapter summary
  • Exercise: Emergent probing
  • A Further Perspective: Probing as Expansive Interviewing
  • A broad view of cognitive interviewing
  • The Ethnographic Interview as an alternative perspective
  • Merging Cognitive and Expansive interviewing
  • Chapter summary
  • Avoiding Probing Pitfalls
  • Are we in danger of finding problems that don't exist?
  • Practices that avoid artificial problems
  • Chapter summary

THE COGNITIVE TESTING PROCESS

  • Training of Cognitive Interviewers
  • Who makes a good cognitive interviewer?
  • Technical background of the cognitive interviewer
  • How should cognitive interviewer training be accomplished?
  • Continuing education in cognitive interviewing
  • Chapter summary
  • Planning and Conducting Cognitive Interviews
  • Fitting cognitive testing into the overall design sequence
  • Preparing for the interview: Subject recruitment
  • The interviewing process
  • Logistic issues in the cognitive interview
  • Chapter summary
  • Analyzing and Documenting Cognitive Interview Results
  • Characterizing cognitive interview outcomes
  • The analysis of think-aloud interview results
  • Analysis of the probed interview
  • Persistent analysis issues
  • Chapter summary
  • Exercise: Analyzing cognitive interviews

OTHER ISSUES AND TOPICS

  • Special Applications of Cognitive Interviewing
  • Adjusting to survey administration mode
  • Cognitive testing of sensitive questions
  • Cognitive interviewing and establishment surveys
  • Testing questions on attitudes and opinions
  • Interviewing across the age range
  • Testing non-questionnaire materials
  • Chapter summary
  • Evaluation of Cognitive Interviewing Techniques
  • Theoretical perspectives
  • Empirical evaluation of cognitive interviewing
  • Are we evaluating the right outcome?
  • Limitations to cognitive interviews
  • Chapter summary
  • Beyond Cognitive Testing: Affiliated Pretesting Methods
  • Expert Review
  • Focus Groups
  • Behavior Coding
  • Reinterview surveys
  • How do pretesting methods compare?
  • Chapter summary
  • Recommendations and Future Directions
  • Twelve recommendations for cognitive interviewing practice
  • Future directions for cognitive interviewing
  • A final case study
  • In conclusion

References

  • Appendix 1. Sample cognitive testing protocol
  • Appendix 2. Example

Description

"As both an academic instructor in questionnaire design and a research design methodologist for the federal government, I feel this book is very timely, useful for students and practitioners, and unique in its use of real world practical examples that most everyone can relate."
—Terry Richardson, General Accounting Office

"The combination of theory and practical application will make this a useful book for students as well as professionals who want to learn how to incorporate cognitive interviewing into the questionnaire design process."
—Rachel Caspar, RTI International

The design and evaluation of questionnaires—and of other written and oral materials—is a challenging endeavor, fraught with potential pitfalls. Cognitive Interviewing: A Tool for Improving Questionnaire Design describes a means of systematically developing survey questions through investigations that intensively probe the thought processes of individuals who are presented with those inquiries. The work provides general guidance about questionnaire design, development, and pre-testing sequence, with an emphasis on the cognitive interview. In particular, the book gives detailed instructions about the use of verbal probing techniques, and how one can elicit additional information from subjects about their thinking and about the manner in which they react to tested questions. These tools help researchers discover how well their questions are working, where they are failing, and determine what they can do to rectify the wide variety of problems that may surface while working with questionnaires.

Cognitive Interviewing is ideally suited as a course text for advanced undergraduate and graduate research courses across the social sciences. Professional researchers and faculty in the social sciences, as well as practice fields such as medicine, business, and education, will also find this an invaluable reference for survey research. There is no other book on the market that covers cognitive interviewing as applied to questionnaire design.



Contents

ORIENTATION AND BACKGROUND

  • Introduction to Cognitive Interviewing
  • What is (and isn't) cognitive interviewing?
  • A broader perspective: The cognitive testing process
  • Setting the Stage: First Principles of Questionnaire Design
  • Questionnaire-related (and other) sources of error in self-report surveys
  • Why do survey questions produce response error? A sociolinguistic perspective
  • How can we avoid problems in survey questions?
  • If we are proficient designers, why do we need to test questions?
  • Chapter summary
  • Exercise: Question evaluation
  • The CASM Approach to Questionnaire Design
  • Origins of CASM
  • Applications of Basic CASM Research
  • Cognitive interviewing as Applied CASM Research
  • Chapter summary
  • Cognitive Interviewing in Practice: Think-Aloud, Verbal Probing, and Other Techniques
  • Think-Aloud interviewing
  • Verbal Probing Techniques
  • Concurrent versus retrospective interviewing
  • Choosing between think-aloud and probing techniques
  • Vignettes, card sorts, and field-based probes
  • Chapter summary

THE INTRICACIES OF VERBAL PROBING

  • Developing Standard Cognitive Probes
  • Examples of cognitive probing
  • Logical and structural problems
  • A systematic approach to probe development: The Question Pitfalls Model
  • Chapter summary
  • Exercise: Using the QAS to develop probe questions
  • Beyond the Standard Model of Verbal Probing
  • A classification of probe types
  • Proactive versus Reactive probing
  • Standardized versus free-form probes
  • Chapter summary
  • Exercise: Emergent probing
  • A Further Perspective: Probing as Expansive Interviewing
  • A broad view of cognitive interviewing
  • The Ethnographic Interview as an alternative perspective
  • Merging Cognitive and Expansive interviewing
  • Chapter summary
  • Avoiding Probing Pitfalls
  • Are we in danger of finding problems that don't exist?
  • Practices that avoid artificial problems
  • Chapter summary

THE COGNITIVE TESTING PROCESS

  • Training of Cognitive Interviewers
  • Who makes a good cognitive interviewer?
  • Technical background of the cognitive interviewer
  • How should cognitive interviewer training be accomplished?
  • Continuing education in cognitive interviewing
  • Chapter summary
  • Planning and Conducting Cognitive Interviews
  • Fitting cognitive testing into the overall design sequence
  • Preparing for the interview: Subject recruitment
  • The interviewing process
  • Logistic issues in the cognitive interview
  • Chapter summary
  • Analyzing and Documenting Cognitive Interview Results
  • Characterizing cognitive interview outcomes
  • The analysis of think-aloud interview results
  • Analysis of the probed interview
  • Persistent analysis issues
  • Chapter summary
  • Exercise: Analyzing cognitive interviews

OTHER ISSUES AND TOPICS

  • Special Applications of Cognitive Interviewing
  • Adjusting to survey administration mode
  • Cognitive testing of sensitive questions
  • Cognitive interviewing and establishment surveys
  • Testing questions on attitudes and opinions
  • Interviewing across the age range
  • Testing non-questionnaire materials
  • Chapter summary
  • Evaluation of Cognitive Interviewing Techniques
  • Theoretical perspectives
  • Empirical evaluation of cognitive interviewing
  • Are we evaluating the right outcome?
  • Limitations to cognitive interviews
  • Chapter summary
  • Beyond Cognitive Testing: Affiliated Pretesting Methods
  • Expert Review
  • Focus Groups
  • Behavior Coding
  • Reinterview surveys
  • How do pretesting methods compare?
  • Chapter summary
  • Recommendations and Future Directions
  • Twelve recommendations for cognitive interviewing practice
  • Future directions for cognitive interviewing
  • A final case study
  • In conclusion

References

  • Appendix 1. Sample cognitive testing protocol
  • Appendix 2. Example
SAGE Publishing Logo

Cognitive Interviewing

A Tool for Improving Questionnaire Design

Gordon B. Willis
Cancel Print

September 2004 | 352 pages | Sage US

Format Published Date ISBN Price

"As both an academic instructor in questionnaire design and a research design methodologist for the federal government, I feel this book is very timely, useful for students and practitioners, and unique in its use of real world practical examples that most everyone can relate."
—Terry Richardson, General Accounting Office

"The combination of theory and practical application will make this a useful book for students as well as professionals who want to learn how to incorporate cognitive interviewing into the questionnaire design process."
—Rachel Caspar, RTI International

The design and evaluation of questionnaires—and of other written and oral materials—is a challenging endeavor, fraught with potential pitfalls. Cognitive Interviewing: A Tool for Improving Questionnaire Design describes a means of systematically developing survey questions through investigations that intensively probe the thought processes of individuals who are presented with those inquiries. The work provides general guidance about questionnaire design, development, and pre-testing sequence, with an emphasis on the cognitive interview. In particular, the book gives detailed instructions about the use of verbal probing techniques, and how one can elicit additional information from subjects about their thinking and about the manner in which they react to tested questions. These tools help researchers discover how well their questions are working, where they are failing, and determine what they can do to rectify the wide variety of problems that may surface while working with questionnaires.

Cognitive Interviewing is ideally suited as a course text for advanced undergraduate and graduate research courses across the social sciences. Professional researchers and faculty in the social sciences, as well as practice fields such as medicine, business, and education, will also find this an invaluable reference for survey research. There is no other book on the market that covers cognitive interviewing as applied to questionnaire design.




Table Of Contents:

  • ORIENTATION AND BACKGROUND
  • Introduction to Cognitive Interviewing
  • What is (and isn't) cognitive interviewing?
  • A broader perspective: The cognitive testing process
  • Setting the Stage: First Principles of Questionnaire Design
  • Questionnaire-related (and other) sources of error in self-report surveys
  • Why do survey questions produce response error? A sociolinguistic perspective
  • How can we avoid problems in survey questions?
  • If we are proficient designers, why do we need to test questions?
  • Chapter summary
  • Exercise: Question evaluation
  • The CASM Approach to Questionnaire Design
  • Origins of CASM
  • Applications of Basic CASM Research
  • Cognitive interviewing as Applied CASM Research
  • Chapter summary
  • Cognitive Interviewing in Practice: Think-Aloud, Verbal Probing, and Other Techniques
  • Think-Aloud interviewing
  • Verbal Probing Techniques
  • Concurrent versus retrospective interviewing
  • Choosing between think-aloud and probing techniques
  • Vignettes, card sorts, and field-based probes
  • Chapter summary
  • THE INTRICACIES OF VERBAL PROBING
  • Developing Standard Cognitive Probes
  • Examples of cognitive probing
  • Logical and structural problems
  • A systematic approach to probe development: The Question Pitfalls Model
  • Chapter summary
  • Exercise: Using the QAS to develop probe questions
  • Beyond the Standard Model of Verbal Probing
  • A classification of probe types
  • Proactive versus Reactive probing
  • Standardized versus free-form probes
  • Chapter summary
  • Exercise: Emergent probing
  • A Further Perspective: Probing as Expansive Interviewing
  • A broad view of cognitive interviewing
  • The Ethnographic Interview as an alternative perspective
  • Merging Cognitive and Expansive interviewing
  • Chapter summary
  • Avoiding Probing Pitfalls
  • Are we in danger of finding problems that don't exist?
  • Practices that avoid artificial problems
  • Chapter summary
  • THE COGNITIVE TESTING PROCESS
  • Training of Cognitive Interviewers
  • Who makes a good cognitive interviewer?
  • Technical background of the cognitive interviewer
  • How should cognitive interviewer training be accomplished?
  • Continuing education in cognitive interviewing
  • Chapter summary
  • Planning and Conducting Cognitive Interviews
  • Fitting cognitive testing into the overall design sequence
  • Preparing for the interview: Subject recruitment
  • The interviewing process
  • Logistic issues in the cognitive interview
  • Chapter summary
  • Analyzing and Documenting Cognitive Interview Results
  • Characterizing cognitive interview outcomes
  • The analysis of think-aloud interview results
  • Analysis of the probed interview
  • Persistent analysis issues
  • Chapter summary
  • Exercise: Analyzing cognitive interviews
  • OTHER ISSUES AND TOPICS
  • Special Applications of Cognitive Interviewing
  • Adjusting to survey administration mode
  • Cognitive testing of sensitive questions
  • Cognitive interviewing and establishment surveys
  • Testing questions on attitudes and opinions
  • Interviewing across the age range
  • Testing non-questionnaire materials
  • Chapter summary
  • Evaluation of Cognitive Interviewing Techniques
  • Theoretical perspectives
  • Empirical evaluation of cognitive interviewing
  • Are we evaluating the right outcome?
  • Limitations to cognitive interviews
  • Chapter summary
  • Beyond Cognitive Testing: Affiliated Pretesting Methods
  • Expert Review
  • Focus Groups
  • Behavior Coding
  • Reinterview surveys
  • How do pretesting methods compare?
  • Chapter summary
  • Recommendations and Future Directions
  • Twelve recommendations for cognitive interviewing practice
  • Future directions for cognitive interviewing
  • A final case study
  • In conclusion
  • References
  • Appendix 1. Sample cognitive testing protocol
  • Appendix 2. Example

Recent Product Reviews:

"The book has definite strong points. After reading Willis's book, I feel more comfortable about developing questions for customer surveys. It's a great reminder that people being surveyed may need more specific questions to provide an accurate response in the cognitive interview."
Rhonda Lunemann
“As both an academic instructor in questionnaire design and a research design methodologist for the federal government, I feel this book is very timely, useful for students and practitioners, and unique in its use of real world practical examples that most everyone can relate.”
Terry Richardson, General Accounting Office
“The combination of theory and practical application will make this a useful book for students as well as professionals who want to learn how to incorporate cognitive interviewing into the questionnaire design process.”
Rachel Caspar, RTI International

Recommendations