Validation of our Measures

Validation Studies for our Referral and Evaluation Process

We have conducted several studies to explore the feasibility of implementing the CSBS-DP Checklist, CQ, and BS and to examine the reliability and validity of these measures (Wetherby & Goldstein, 1999; Wetherby, Cleary, Allen & Goldstein, 2000; Wetherby & Prizant, 2001). The validation groups ranged in age from 6 to 24 months and include a culturally diverse population with over one-fourth African American, the largest minority residing in Leon County. Following is a summary of our findings.

Study 1. The first study was designed to explore the relationship among the three CSBS-DP measures- the Checklist, CQ, and BS, on a sample of young children including those who were typically developing and those at risk for having developmental delays or disabilities. This study assessed the concurrent validity of the parent report measures by comparing them with measures derived from behavior samples of the children. Participants in this study were the first 1,000 families contacted in Tallahassee, Florida who completed a Checklist between June, 1997 and July, 1999. Interrater reliability was established for the scoring of the BS by having all three trained examiners independently score 15% of the videotaped samples that were randomly selected. Two of the three examiners scored an additional 15% of randomly selected samples. Interrater reliability coefficients for the three examiners ranged from .70 to .95 for the cluster scores and .95 to .99 for the total scores. These results indicate good interrater reliability for scoring the BS.

The relationship among the Checklist, CQ and BS measures was examined by calculating correlation coefficients on pairs of measures obtained from the same children. The correlation coefficients for the total scores on the Checklist, CQ, and BS are presented below:

 

 

Strong, significant correlations were observed among all of the total scores and almost all of the clusters. The strongest correlations were observed for the sounds, use of words and understanding of words clusters. These findings support the validity of parent report as a measure of these early prelinguistic communication abilities. The brief parent report tool (i.e., the Checklist) had a very strong correlation with the more in-depth parent report tool. Furthermore, most children who performed poorly on the parent report tools (i.e., the Checklist and CQ) also performed poorly on the direct evaluation of the child (i.e., BS).

Study 2. The second study was designed to assess the test-retest reliability of the Checklist, CQ, and BS over a 3- to 4-month retest interval. The relationship between the test and retest for the Checklist, CQ and BS measures was examined by calculating correlation coefficients on pairs of measures obtained from the same children. The correlation coefficients for the total scores on the Checklist, CQ, and BS are presented below:

 

 

Significant correlations were observed among all of the total scores and most of the clusters. The correlations were moderately high for all of the clusters and very high for the total scores. These findings support the test-retest reliability of these three measures over a 3- to 4-month retest interval and indicate that these are stable measures of prelinguistic communication.

Study 3. The third study was designed to explore the relationship of the Checklist, CQ, and BS and later vocabulary production measured at 24 months with a parent report tool for about 100 children. About half of these children were tested with the CSBS-DP at an average age of 21 months and the other half at an average age of 15 months. First, the relationship between the three CSBS-DP measures and vocabulary production was examined by calculating correlation coefficients on pairs of measures obtained from the same children. The correlation coefficients for the use of words cluster and the total scores on the Checklist, CQ, and BS are presented below:

 

 

We found moderate, significant correlations with all three of our measures and the size of a child's vocabulary at age 24 months for both groups of children. The strongest correlations were observed for the sounds, use of words and understanding of words clusters. These findings support the predictive validity of the three CSBS measures of early prelinguistic communication abilities. In other words these three measures, which were gathered up to an average of 9 months earlier were effective at predicting a child's relative performance on vocabulary production at 24 months of age.

 

Study 4. The fourth study was designed to validate the Checklist, BS and 24-month parent report measure of vocabulary production with standardized testing at 25 months of age. The Mullen Scales of Early Learning was used for the standardized testing at 2 years, which measures gross motor, fine motor, visual recognition, receptive language, and expressive language. The sensitivity (true positives), specificity (true negatives), overreferral (false positives) and underreferral (false negatives) rates for each measure were calculated with the standardized testing outcome as the gold standard as follows:

 

 

These findings suggest that both the CSBS-DP and vocabulary production are more accurate with young children in early identification than commonly used tests, such as the Denver Developmental Screening Test. Furthermore, both the Checklist and BS, which were collected under 24 months of age and up to a year before the standardized testing, had more precision than the vocabulary production measure, which was collected within a month of the standardized testing. These findings are illustrated in the figure below; they strongly support the validity of the Checklist and BS and the use of a collection of prelinguistic measures, rather than the use of words alone, to improve the sensitivity of early identification efforts.

In conclusion, these validation studies demonstrate that the FIRST WORDS Project screening and evaluation model is effective at measuring early prelinguistic communication skills and predicting a child's relative performance on measures up to a year later. We are now following these children until they turn at least 3 years of age to have a more definitive measure of cognitive and language outcome, since some of these children may catch up on their own. Furthermore, we are planning to follow them until they are at least 5 years of age to have a more definitive measure of emergent literacy outcome to explore the relationship between these prelinguistic measures and risk for reading difficulties.

References

Squires, J., Potter, L., & Bricker, D. (1999). The ASQ User's Guide for the Ages & Stages Questionnaires: A Parent-Completed, Child-Monitoring System (Second edition). Baltimore, MD: Paul H. Brookes Publishing Co.

Wetherby, A., Cleary, J., Allen, L., & Goldstein, H. (2000). FIRST WORDS Project: Improving early identification of communication disorders. Conference session presented at the Annual International Division of Early Childhood Conference, Albuquerque, New Mexico, December, 2000.

Wetherby, A. & Goldstein, H. (1999). FIRST WORDS Project: How early can we identify young children with communication disorders? Seminar presented at the Annual Convention of the American Speech-Language-Hearing Association, San Francisco, California, November, 1999.

Wetherby, A. and Prizant, B. (1998). Communication and Symbolic Behavior Scales- Developmental Profile (Research Edition). Chicago, IL: Applied Symbolix.

Wetherby, A. and Prizant, B. (in press). Communication and Symbolic Behavior Scales- Developmental Profile (Preliminary Normed Edition). Chicago, IL: Applied Symbolix.

The FIRST WORDS® Project has received support for research from the U.S. Department of Education, Office of Special Education Programs and Institute of Education Sciences; the National Institutes of Health, Institute on Deafness and other Communication Disorders; the Centers for Disease Control and Prevention; the Jessie Ball duPont Fund, Jacksonville, Florida; the Phillips Endowment; Todd and Annie Trimmer; and Delta Airlines.