Table Of Contents:

• A Step-by-Step Guide to the CAATS Model
• Foreword by Richard Kunkel
• Preface
• Acknowledgments
• About the Authors
• 1. Expectations and Options for Accountability and Teacher Assessment
• What This Chapter Is About
• The Challenge from the National Commission on Teaching and America’s Future
• Title II of the Higher Education Act Amendments of 1998
• No Child Left Behind Legislation
• National Research Council (2001) -- the Committee on Assessment and Teacher Quality
• What A Few Others Have Said: A Brief Review of the Literature on Testing and Licensure
• Standards: The Roadmap to Accountability and Scientifically-Based Performance Assessment
• The Principal Sets of Standards Governing Our Work
• National and State Pedagogical and Content Standards
• Unit Accreditation and Operational Standards
• Technical Standards for Measurement of Teacher Competency
• Some Major Threats to Validity in Most Current Assessment Systems
• Conceptual Frameworks: Pulling It All Together
• NCATE Standards
• AERA, APA, & NCME
• INTASC Principles: Where NCATE and AERA, APA, & NCME Standards Converge
• Making Sense of Conceptual Frameworks
• Our Conceptual Framework: What We Value
• Assessment Options
• Records of Training Completed
• Tests and Exam Scores
• Observations of Performance
• Portfolios of Assessable Artifacts
• Job-Related Tasks and Work Sample Products
• K-12 Student Work Samples
• Wrap-Up
• Activity #1: What’s Happening in Your State and School?
• Activity #2: Questionnaire for Faculty Views on Competency Assessment
• Activity #3: Assessment Belief Scale
• Activity #4: Assessment Options
• 2. Portfolios – To Be or Not To Be?? That IS the Question!
• What This Chapter Is About
• The Portfolio: Panacea or Pandora’s Box
• Portfolios as Certification “Tests”: Lessons from Standards and History
• Assessment Illiteracy, Paradigms Shifts, and Conflicting Purposes
• The Conflict of Formative vs. Summative Evaluation
• The Conflict of Program Approval vs. Accreditation
• The Conflict of Business or Legislative Personal/Professional Perspectives vs. Accountability Through Title II Requirements
• The Conflict of Academic Freedom vs. Accountability
• The Conflict of Constructivism vs. Positivism
• Recommendations for Use of Portfolios in Accountability Contexts
• Ten Recommendations for Assessment System Design
• A Recommended, Standards-Based Model
• Overview of “Competency Assessment Aligned with Teacher Standards” (CAATS) Model
• CAATS Step 1: Define content, purpose, use, and other contextual
• CAATS Step 2: Develop a valid sampling plan
• CAATS Step 3: Create or update tasks aligned with standards and
• CAATS Step 4: Design and implement data aggregation tracking and management systems
• CAATS Step 5: Ensure credibility and utility of data
• Wrap-Up
• A Note about Chapters 8 and 9
• Activities
• Activity #1: Review Your Feelings
• Activity #2: Thinking about Conflicting Paradigms
• Activity #3: Getting Your Action Plan Started
• 3. CAATS Step 1: Assessment Design Inputs
• Where We Have Been So Far
• What This Chapter Is About
• CAATS Step 1A: Define the Purposes and Uses of the System
• The Importance of Purpose and Use
• Different Strokes for Different Folks for Different Purposes
• Purpose and Use in the Accountability Context
• More on Accountability Based Systems: State Program Approval
• CAATS Step 1B: Define the Propositions or Principles that Guide the System
• CAATS Step 1C: Define the Conceptual Framework or Contents of the System
• So What Is Assessment Content?
• Standards as the Link between Purpose, Use, and Content
• CAATS Step 1D: Review Local Factors That Impact the System
• Wrap Up
• Worksheets
• Worksheet #1: Purpose, Use, Propositions, Content, & Context Checksheet
• Worksheet #2: Purpose, Use, Content, Draft
• Worksheet #3: Propositions
• Worksheet #4: Contextual Analysis
• 4. CAATS Step 2: Planning with a Continuing Eye on Valid Assessment Decisions
• Where We Have Been So Far
• What This Chapter Is About
• CAATS Step 2A: Organize Standards into Content Domains
• All Those Standards Sets
• Organizing for Alignment
• “A Rose Is a Rose” Or More of the Same
• Why Bother?
• Crosswalks and Standards
• CAATS Step 2B: Visualize the Competent Teacher Based on the Standards
• CAATS Step 2C: Brainstorm a Set of Summative Tasks (Sampling Plan)
• CAATS Step 2D: Sort Tasks into Formative and Summative Assessments
• CAATS Step 2E: Build Assessment Frameworks
• Framework Options
• A Special Case: Aligning Tasks with NCATE Requirements
• Wrap-Up
• Worksheets
• Worksheet #1: Organizing for Alignment (Version 1)
• Worksheet #1: Organizing for Alignment (Version 2)
• Worksheet #2: Our Critical Skills
• Worksheet #3: Visualizing the Competent Teacher
• Worksheet #4: Critical Task List
• Worksheet #5: Sorting Formative and Summative Tasks
• Worksheet #6: List of Summative Assessments by Competency Type
• Worksheet #7: List of Summative Assessments by Levels of Inference
• Worksheet #8: List of Summative Assessments by Points in Time
• Worksheet #9: Matrix of Standard by Competency Type
• Worksheet #10: Matrix of Critical Tasks by Competency Type and Benchmark
• Worksheet #11: Aligning Tasks with NCATE Thematic Portfolios
• 5. CAATS Step 3 -- Writing Tasks Designed to Maximize Validity and Reliability
• Where We Have Been So Far
• What This Chapter Is About
• CAATS Step 3A: Determine the Task Format for Data Aggregation
• What Happens When There Is No Format?
• Can I Use Percents and Total Points? No, They Don’t Cut It!
• Elements of a Common Task Format
• CAATS Step 3B: Create New Tasks or Modify Existing Tasks
• Basic Concepts about Tasks
• Hints and Advice about Writing Tasks
• Rubric Examples: A Rose is a Rose
• CAATS Step 3C: Conduct First Validity Study
• CAATS Step 3D: Align Tasks with Instruction
• Wrap-Up
• Worksheets
• Worksheet #1: Proficiency Level Descriptions
• Worksheet #2: Task Design
• Worksheet #3: Standards and Indicators Coverage Report
• Worksheet #4: Individual Task Review for Job-Relatedness
• Worksheet #5: Checklist for Reviewing Individual Tasks
• Worksheet #6: Instructional Alignment
• 6. CAATS Step 4: Decision-Making and Data Management
• Where We Have Been So Far
• What This Chapter Is About
• CAATS Step 4A: Determine How Data Will Be Aggregated
• So What Is Data Aggregation?
• The Relationship of Data Aggregation (Step 4A), Cut Scores, (Step 4B), and
• An Approach to Decision-Making without Using Points and Percents
• CAATS Step 4B: Set Standards for Minimal Competency
• Different Strokes for Different Folks – Déjà Vu or Purpose Revisited
• Is 100% Really Reasonable?
• Criticality Yardstick Approach (CYA) to Cut Score Setting in Complex
• Performance Assessments
• First Cut
• Second Cut
• Working with Judges
• An Example of What to Say to the Judges
• CAATS Step 4C: Select and Develop a Tracking System
• Sharing Information for Decision-Making: The Big Challenge
• Data Storage Option #1: Course Grades or Records of Participation/Attendance
• Data Storage Option #2: Teacher Folders
• Data Storage Option #3: Portfolios
• Data Storage Option #4: Electronic Data Management System
• Reporting Aggregated Data
• CAATS Step 4D: Develop Management System
• Advising and Due Process
• Scoring Procedures
• Implementation
• Wrap-Up
• Worksheets
• Worksheet #1: Cut Score Decisions
• Worksheet #2: Sample Format for Candidate/Teacher Tracking Form
• Worksheet #3: Format for Data Aggregation
• Worksheet #4: Management Plan
• Worksheet #5: Rater Monitoring Record
• 7. CAATS Step 5: Credible Data
• Where We Have Been So Far
• What This Chapter Is About
• What Is Psychometric Integrity and Why Do We Have to Worry About It?
• CAATS Step 5A: Create a Plan to Provide Evidence of Validity, Reliability, Fairness, and Utility
• Elements of a Plan
• Element #1: Purpose and Use
• Element #2: Construct Measured
• Element #3: Interpretation and Reporting of Scores
• Element #4: Assessment Specifications and Content Map
• Element #5: Assessor/Rater Selection and Training Procedures
• Element #6: Analysis Methodology
• Element #7: External Review Personnel and Methodology
• Element #8: Evidence of Validity, Reliability, and Fairness (VRF)
• CAATS Step 5B: Implement the Plan Conscientiously
• Wrap Up
• Worksheets and Examples
• Worksheet #1: Assessment Specifications
• Worksheet #2: Analysis of Appropriateness of Decisions for Teacher Failures
• Worksheet #3: Program Improvement Record
• Worksheet #4: Expert Rescoring
• Worksheet #5: Fairness Review
• Worksheet #6: Analysis of Remediation Efforts and EO Impact
• Worksheet #7: Psychometric Plan Format
• Example #1 (Empirical Data): Logistic Ruler for Content Validity
• Example #2 (Empirical Data): Computation of the Lawshe (1975) Content
• Example #3 (Empirical Data): Disparate Impact Analysis
• Example #4 (Empirical Data): Computation of Cohen’s Kappa (1960) for
• Example #5 (Empirical Data): Spearman Correlation Coefficient and Scatterplot:
• Example #6 (Empirical Data): Pearson Correlation Coefficient and Scatterplot:
• Example #7 (Empirical Data): Correlation Matrix and Scatterplots Knowledge,
• Example #8 (Empirical Data): T-Test Comparing Mathematics and Science
• Example #9 (Empirical Data): Differential Item and Person Functioning
• 8. The Trouble with Tribbles: Standard Setting for Professional Certification
• What This Chapter Is About
• An Overview of Cut Score Setting
• From Norm-Referenced Objective Tests to Criterion-Referenced Subjective Tasks
• Controlling Human Judgment
• Difficulty vs. Importance
• Social Consequences
• Tidbits to Remember
• Cut Scores Setting in Traditional, Objective Tests
• Holistic Impressions: Method One
• Item Content: Method Two
• Performance of Examinees: Method Three
• Combination Approach: Method Four
• Standard or Cut Score Setting in Performance Tests for Professionals: Is it the Same as Multiple Choice?
• Quotes to Remember
• The Extended Angoff Procedure for Performance Assessment
• The Judgmental Policy Capturing Approach for Performance Assessment
• The Dominant Profile Judgment for Performance Assessment
• Standard Setting Using Item Response Theory
• What’s Really Wrong with Current Approaches to Cut Score Setting
• Berk’s Suggestions and Commentaries Over the Years
• The Criticality Yardstick Approach
• Wrap Up
• Activities
• Activity #1: Change the Difficulty of an Item
• Activity #2: Change the Criticality of a Task
• Activity #3: Replicate a Cut Score
• Activity #4: Take Your Choice
• 9. Using Teacher Scores for Continuous Improvement
• What This Chapter is About
• Reasons Why We Use the Rasch Model
• The Classical Approach
• A Quick Overview of Where Rasch Fits Into the Grand Scheme of IRT Models
• Rasch: The Basics
• Getting Started
• Differences that Item Writers Make
• Guttman Scaling
• A Sample Rasch Ruler
• From Pictures to Numbers
• The Fit Statistic
• Gain Scores – Real or Imagined?
• Ratings and Raters
• Learning More about Rasch
• Wrap Up
• Activities
• Activity #1: Decision Making Tool for Measurement
• 10. Legal Integrity
• What IF?? A Legal Scenario
• Psychometric Issues and Legal Challenges
• Legal Issues and Precedents
• End Note
• Appendix: Tasks Developed for Florida Alternative Certification Program
• Glossary
• References
• Index