Designing Small Evaluation Studies

Larry V. Hedges - Northwestern University, USA

Elizabeth Tipton - Northwestern University, USA

Program Evaluation | Policy Evaluations | Evaluation

April 2025 | 352 pages | Sage US

Create Flyer

If you’re in the Americas, please visit our Sage College Publishing website to purchase or sample this book:

Go to College Publishing Website

Description

"The book will be an important addition to instruction in designs for causal inference in the field of education. It is long overdue." - Thomas J. Lipscomb, The University of Southern Mississippi

This text describes how to design and analyze small efficacy or evaluation studies, typically carried out as part of the development of programs or interventions in areas such as education. The problem facing many researchers is how to design a study that is as small as possible, yet big enough to yield relatively unambiguous evidence about an intervention’s average effect. This text begins with an overview of validity, causal inference, statistics, effect sizes, and measurement. The authors then focus on designs for small, randomized trials, followed by a section on non-randomized causal designs: here they focus on three designs most useful for small studies including the non-equivalent control group, difference-in-difference, and interrupted time series designs. The final section summarizes the book, compares designs, discusses approaches to choosing a design, and provides guidance on reporting. Five case examples are used throughout the book to illustrate the material and there is a glossary of terms and concepts.

Preface

About the Author

Chapter 1: Introduction
A. What Is an Intervention?
B. Examples

Section I: Background Concepts

Chapter 2: Introduction to Section I: Is a Small Efficacy Study Right for You?
A. What Is the Logic of An Efficacy Study?
B. How Can This Test Be Operationalized?
C. What About Small Efficacy Studies?
D. What Questions Can’t a Small Efficacy Study Answer Well?
E. What Are Other Options?
F. Moving Forward
Chapter 3: Research Design
A. Logic of Inquiry
B. Principles of Research Design
C. Types of Quantitative Research Designs
Questions to Test Your Knowledge
Chapter 4: Validity of Research Designs
A. Causal Inference
B. Internal validity
C. External Validity
D. Statistical Conclusion Validity
E. Construct Validity of Explanation
F. Conclusion
Questions to Test Your Knowledge
Chapter 5: A Brief Review of Statistics
A. Populations and Samples
B. Random Sampling
C. Models and Notations
D. Estimators and Sampling Distributions
E. Statistical Inference
F. Design Sensitivity
G. Design Complexities and Statistical Models
H. Multiple Regression Analysis
I. Multilevel Statistical Models
J. Research Designs
Questions to Test Your Knowledge
Chapter 6: A Brief Review of Measurement
A. What is Measurement?
B. Measurement Theory
C. Outcome Concept Domains
D. Types of Measures
E. Scoring of Measures
F. Choosing an Outcome Measure
Questions to Test Your Knowledge
Chapter 7: Choosing an Appropriate Effect Size
A. What is an Effect Size?
B. Choosing an Effect Size Value
C. Other Measurement and Statistical Considerations
Questions to Test Your Knowledge

Section II: Randomized Designs

Chapter 8: Introduction to Section II: What is Randomization?
A. Randomization
B. Theoretical Objections to Random Assignment
C. Small Efficacy Studies
D. Overview of The Next Four Chapters
Chapter 9: Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. The Statistical Analysis of the Design
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 10: Multisite Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multisite Individually Randomized Designs With Fixed Site Effects
D. Increasing Design Sensitivity
E. Multisite Individually Randomized Designs with Random Site Effects
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 11: Multisite Cluster Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multi-Site Cluster Randomized Design
D. Increasing Design Sensitivity
E. Examples
F. Conclusions
Appendix
Chapter 12: Cluster Randomized Designs
A. General Approach
B. Statistical Model and Analysis
C. Increasing Design Sensitivity
D. Examples
E. Conclusions
Questions to Test Your Learning
Appendix

Section III: Quasi-Experimental Designs

Chapter 13: Intro to Section III: What Is A ‘Quasi’ Experiment?
A. Threats to Internal Validity in QEDs
B. Why Pre-Post Designs are Not Adequate
C. Statistical Analyses of QEDs
D. QEDs for Small Efficacy Studies
Chapter 14: Nonequivalent Control Group Designs
A. General Approach
B. Statistical Models and Analysis
C. Selection Bias
D. Confounders and Covariate Selection
E. Matching Models
F. Statistical Adjustment Methods
G. Designing a NECD Study
H. Examples
I. Conclusions
Questions to Test Your Learning
Chapter 15: The Difference in Differences Design
A. General Approach
B. Comparisons of Individuals (No Nesting)
C. Multiple Subgroups Within Each Treatment Group
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 16: Interrupted Time Series Designs
A. General Approach
B. Model and Notation
C. Estimation and Analysis of the Design
D. Design Sensitivity
E. Variants of This Design
F. Conclusions
Questions to Test Your Learning
Appendix

Section IV: Tools and Reporting

Chapter 17: Introduction to Section IV: Tools and Considerations for Practice
A. Principles of Small Efficacy Studies
B. Randomized Designs
C. Quasi-Experimental Designs
D. Tools for Use in the Field
Chapter 18: Choosing a Research Design
A. Know Your Setting
B. Is a Comparison Group Possible?
C. Is Randomization Possible?
D. Can an Adequately Matched Comparison Group Be Formed?
E. Which Designs are Possible?
F. What If None of These Designs Can Be Implemented?
Questions to Test Your Learning
Chapter 19: Worksheets for Comparing Designs
A. Overview of Worksheet 1
B. Overview of Worksheet 2
Chapter 20: Best Practices for Reporting Small Efficacy Study Results
A. Reporting Standards
B. Special Considerations for Small Efficacy Studies

List of Tables and Figures

Glossary

References

Index

Additional materials

Ch1-2

Description

Preface

About the Author

Chapter 1: Introduction
A. What Is an Intervention?
B. Examples

Section I: Background Concepts

Chapter 2: Introduction to Section I: Is a Small Efficacy Study Right for You?
A. What Is the Logic of An Efficacy Study?
B. How Can This Test Be Operationalized?
C. What About Small Efficacy Studies?
D. What Questions Can’t a Small Efficacy Study Answer Well?
E. What Are Other Options?
F. Moving Forward
Chapter 3: Research Design
A. Logic of Inquiry
B. Principles of Research Design
C. Types of Quantitative Research Designs
Questions to Test Your Knowledge
Chapter 4: Validity of Research Designs
A. Causal Inference
B. Internal validity
C. External Validity
D. Statistical Conclusion Validity
E. Construct Validity of Explanation
F. Conclusion
Questions to Test Your Knowledge
Chapter 5: A Brief Review of Statistics
A. Populations and Samples
B. Random Sampling
C. Models and Notations
D. Estimators and Sampling Distributions
E. Statistical Inference
F. Design Sensitivity
G. Design Complexities and Statistical Models
H. Multiple Regression Analysis
I. Multilevel Statistical Models
J. Research Designs
Questions to Test Your Knowledge
Chapter 6: A Brief Review of Measurement
A. What is Measurement?
B. Measurement Theory
C. Outcome Concept Domains
D. Types of Measures
E. Scoring of Measures
F. Choosing an Outcome Measure
Questions to Test Your Knowledge
Chapter 7: Choosing an Appropriate Effect Size
A. What is an Effect Size?
B. Choosing an Effect Size Value
C. Other Measurement and Statistical Considerations
Questions to Test Your Knowledge

Section II: Randomized Designs

Chapter 8: Introduction to Section II: What is Randomization?
A. Randomization
B. Theoretical Objections to Random Assignment
C. Small Efficacy Studies
D. Overview of The Next Four Chapters
Chapter 9: Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. The Statistical Analysis of the Design
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 10: Multisite Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multisite Individually Randomized Designs With Fixed Site Effects
D. Increasing Design Sensitivity
E. Multisite Individually Randomized Designs with Random Site Effects
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 11: Multisite Cluster Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multi-Site Cluster Randomized Design
D. Increasing Design Sensitivity
E. Examples
F. Conclusions
Appendix
Chapter 12: Cluster Randomized Designs
A. General Approach
B. Statistical Model and Analysis
C. Increasing Design Sensitivity
D. Examples
E. Conclusions
Questions to Test Your Learning
Appendix

Section III: Quasi-Experimental Designs

Chapter 13: Intro to Section III: What Is A ‘Quasi’ Experiment?
A. Threats to Internal Validity in QEDs
B. Why Pre-Post Designs are Not Adequate
C. Statistical Analyses of QEDs
D. QEDs for Small Efficacy Studies
Chapter 14: Nonequivalent Control Group Designs
A. General Approach
B. Statistical Models and Analysis
C. Selection Bias
D. Confounders and Covariate Selection
E. Matching Models
F. Statistical Adjustment Methods
G. Designing a NECD Study
H. Examples
I. Conclusions
Questions to Test Your Learning
Chapter 15: The Difference in Differences Design
A. General Approach
B. Comparisons of Individuals (No Nesting)
C. Multiple Subgroups Within Each Treatment Group
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 16: Interrupted Time Series Designs
A. General Approach
B. Model and Notation
C. Estimation and Analysis of the Design
D. Design Sensitivity
E. Variants of This Design
F. Conclusions
Questions to Test Your Learning
Appendix

Section IV: Tools and Reporting

Chapter 17: Introduction to Section IV: Tools and Considerations for Practice
A. Principles of Small Efficacy Studies
B. Randomized Designs
C. Quasi-Experimental Designs
D. Tools for Use in the Field
Chapter 18: Choosing a Research Design
A. Know Your Setting
B. Is a Comparison Group Possible?
C. Is Randomization Possible?
D. Can an Adequately Matched Comparison Group Be Formed?
E. Which Designs are Possible?
F. What If None of These Designs Can Be Implemented?
Questions to Test Your Learning
Chapter 19: Worksheets for Comparing Designs
A. Overview of Worksheet 1
B. Overview of Worksheet 2
Chapter 20: Best Practices for Reporting Small Efficacy Study Results
A. Reporting Standards
B. Special Considerations for Small Efficacy Studies

List of Tables and Figures

Glossary

References

Index

Additional materials

Ch1-2

Reviews

The book will be an important addition to instruction in designs for causal inference in the field of education. It is long overdue. Press enter to read the full quote (will open a modal popup)
Thomas J. Lipscomb, The University of Southern Mississippi

Designing Small Evaluation Studies

Larry V. Hedges , Elizabeth Tipton

Cancel Print

April 2025 | 352 pages | Sage US

Format	Published Date	ISBN	Price

Preface
About the Author
Chapter 1: Introduction
A. What Is an Intervention?
B. Examples
Section I: Background Concepts
Chapter 2: Introduction to Section I: Is a Small Efficacy Study Right for You?
A. What Is the Logic of An Efficacy Study?
B. How Can This Test Be Operationalized?
C. What About Small Efficacy Studies?
D. What Questions Can’t a Small Efficacy Study Answer Well?
E. What Are Other Options?
F. Moving Forward
Chapter 3: Research Design
A. Logic of Inquiry
B. Principles of Research Design
C. Types of Quantitative Research Designs
Questions to Test Your Knowledge
Chapter 4: Validity of Research Designs
A. Causal Inference
B. Internal validity
C. External Validity
D. Statistical Conclusion Validity
E. Construct Validity of Explanation
F. Conclusion
Questions to Test Your Knowledge
Chapter 5: A Brief Review of Statistics
A. Populations and Samples
B. Random Sampling
C. Models and Notations
D. Estimators and Sampling Distributions
E. Statistical Inference
F. Design Sensitivity
G. Design Complexities and Statistical Models
H. Multiple Regression Analysis
I. Multilevel Statistical Models
J. Research Designs
Questions to Test Your Knowledge
Chapter 6: A Brief Review of Measurement
A. What is Measurement?
B. Measurement Theory
C. Outcome Concept Domains
D. Types of Measures
E. Scoring of Measures
F. Choosing an Outcome Measure
Questions to Test Your Knowledge
Chapter 7: Choosing an Appropriate Effect Size
A. What is an Effect Size?
B. Choosing an Effect Size Value
C. Other Measurement and Statistical Considerations
Questions to Test Your Knowledge
Section II: Randomized Designs
Chapter 8: Introduction to Section II: What is Randomization?
A. Randomization
B. Theoretical Objections to Random Assignment
C. Small Efficacy Studies
D. Overview of The Next Four Chapters
Chapter 9: Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. The Statistical Analysis of the Design
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 10: Multisite Individually Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multisite Individually Randomized Designs With Fixed Site Effects
D. Increasing Design Sensitivity
E. Multisite Individually Randomized Designs with Random Site Effects
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 11: Multisite Cluster Randomized Designs
A. General Approach
B. Statistical Model and Notation
C. Multi-Site Cluster Randomized Design
D. Increasing Design Sensitivity
E. Examples
F. Conclusions
Appendix
Chapter 12: Cluster Randomized Designs
A. General Approach
B. Statistical Model and Analysis
C. Increasing Design Sensitivity
D. Examples
E. Conclusions
Questions to Test Your Learning
Appendix
Section III: Quasi-Experimental Designs
Chapter 13: Intro to Section III: What Is A ‘Quasi’ Experiment?
A. Threats to Internal Validity in QEDs
B. Why Pre-Post Designs are Not Adequate
C. Statistical Analyses of QEDs
D. QEDs for Small Efficacy Studies
Chapter 14: Nonequivalent Control Group Designs
A. General Approach
B. Statistical Models and Analysis
C. Selection Bias
D. Confounders and Covariate Selection
E. Matching Models
F. Statistical Adjustment Methods
G. Designing a NECD Study
H. Examples
I. Conclusions
Questions to Test Your Learning
Chapter 15: The Difference in Differences Design
A. General Approach
B. Comparisons of Individuals (No Nesting)
C. Multiple Subgroups Within Each Treatment Group
D. Design Sensitivity
E. Strategies to Increase Design Sensitivity
F. Examples
G. Conclusions
Questions to Test Your Learning
Appendix
Chapter 16: Interrupted Time Series Designs
A. General Approach
B. Model and Notation
C. Estimation and Analysis of the Design
D. Design Sensitivity
E. Variants of This Design
F. Conclusions
Questions to Test Your Learning
Appendix
Section IV: Tools and Reporting
Chapter 17: Introduction to Section IV: Tools and Considerations for Practice
A. Principles of Small Efficacy Studies
B. Randomized Designs
C. Quasi-Experimental Designs
D. Tools for Use in the Field
Chapter 18: Choosing a Research Design
A. Know Your Setting
B. Is a Comparison Group Possible?
C. Is Randomization Possible?
D. Can an Adequately Matched Comparison Group Be Formed?
E. Which Designs are Possible?
F. What If None of These Designs Can Be Implemented?
Questions to Test Your Learning
Chapter 19: Worksheets for Comparing Designs
A. Overview of Worksheet 1
B. Overview of Worksheet 2
Chapter 20: Best Practices for Reporting Small Efficacy Study Results
A. Reporting Standards
B. Special Considerations for Small Efficacy Studies
List of Tables and Figures
Glossary
References
Index

Designing Small Evaluation Studies

Description

Contents

Additional materials

Description

Contents

Preface

Preface

About the Author

About the Author

Section I: Background Concepts

Section I: Background Concepts

Section II: Randomized Designs

Section II: Randomized Designs

Section III: Quasi-Experimental Designs

Section III: Quasi-Experimental Designs

Section IV: Tools and Reporting

Section IV: Tools and Reporting

List of Tables and Figures

List of Tables and Figures

Glossary

Glossary

References

References

Index

Index

Additional materials

Description

Description

Contents

Contents

Preface

Preface

About the Author

About the Author

Section I: Background Concepts

Section I: Background Concepts

Section II: Randomized Designs

Section II: Randomized Designs

Section III: Quasi-Experimental Designs

Section III: Quasi-Experimental Designs

Section IV: Tools and Reporting

Section IV: Tools and Reporting

List of Tables and Figures

List of Tables and Figures

Glossary

Glossary

References

References

Index

Index

Additional materials

Additional materials

Reviews

Designing Small Evaluation Studies

Table Of Contents:

Recent Product Reviews:

Recommendations