Evidence Library

Psychometric Analysis of Child Outcome Measures with American Indian and Alaska Native Preschoolers: Initial Evidence from AIAN FACES 2015
Evidence | Publications

Psychometric Analysis of Child Outcome Measures with American Indian and Alaska Native Preschoolers: Initial Evidence from AIAN FACES 2015

OPRE Technical Report 2018-21
Published: Feb 28, 2018
Publisher: Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services
Human Services Early Childhood Child Development

Associated Project

American Indian and Alaska Native Head Start Family and Child Experiences Survey (AIAN FACES)

Time frame: 2014-2022

Prepared for:

U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation

Clients

OPRE Logo

Authors

Lizabeth Malone

Sara Bernstein

Sally Atkins-Burnett

Yange Xue

Key Findings

  • For most of the measures, findings from these analyses suggest that it is appropriate to report the AI/AN FACES 2015 preschool child outcomes scores, the exception being one of the two measures of executive function.
  • All measures demonstrated acceptable reliability with alphas of 0.70 or above.
  • The strength of correlations between measures is in an expected pattern. Correlations are stronger between measures of similar constructs (for example, receptive and expressive language) than between different constructs (for example, social behavior and language).
  • Among six cognitive measures flagged across reviews, none warrant additional follow-up based on the DIF analyses. Most cognitive measures did not show evidence of performing differently across groups based on DIF analysis. Two cognitive measures (Peabody Picture Vocabulary Test-Fourth Edition and Expressive One-Word Picture Vocabulary Test-Fourth Edition) had items demonstrating DIF; however, the number of items with DIF was close to or less than the number we would expect by chance and item difficulties were balanced overall, with some easier for AI/AN children and others easier for White children.
  • None of the teacher- and assessor-reported social-emotional measures exhibited performance concerns based on the current review.
  • Examination of the executive function measures indicated that the pencil tapping task is an appropriate measure for this sample. However, a floor effect was found with the HTKS (the majority of children scored near the bottom). Therefore, the measure provided limited information to distinguish the children in this sample.

AIAN FACES 2015 is the first national study of Region XI AI/AN Head Start children and their families, classrooms, and programs. To date, the Head Start Family and Child Experiences Survey (FACES) has been a major source of descriptive information on Head Start and preschool children ages 3 to 5 years old who attend the program. FACES gathers data from Regions I-X, the 10 geographically based Head Start regions, with the most recent round conducted in 2014.The AIAN FACES 2015 study presents a new opportunity to explore the psychometric performance of commonly used measures of preschoolers’ cognitive and social-emotional development. The reliability and validity of a measure are not inherent but depend on its use. Norming samples for most child assessment measures do not include large numbers of AIAN children and as a result little is known about measure performance when administered to AI/AN children. Concerns exist about whether scores from these measures accurately reflect the children’s abilities, skills, and knowledge. Previous smaller studies have used these measures with AIAN children, but none were large enough to test the measures’ psychometric performance. Child outcomes measures in AIAN FACES 2015 were aligned with those in FACES 2014. Therefore, this alignment allows us to learn how standardized child development measures performed when administered to a large sample of AIAN children. This technical report describes the performance of cognitive and social-emotional measures of preschoolers’ development for AIAN children, using recent data from AIAN FACES 2015 and FACES 2014.

Efficiency Meets Impact.
That's Progress Together.

To solve their most pressing challenges, organizations turn to Mathematica for deeply integrated expertise. We bring together subject matter and policy experts, data scientists, methodologists, and technologists who work across topics and sectors to help our partners design, improve, and scale evidence-based solutions.

Work With Us