The Hidden Influence of Social Desirability Bias: Assessing the Validity of a Parenting and Family Skills Assessment to Measure Change over Time in Child Welfare

 

Purpose:  The child welfare system continues in its pursuit of delivering evidence-based interventions (EBI) for improving parent and family skills. Despite the frequent reporting of findings regarding program effectiveness, the psychometric properties of instruments commonly used to test program effects remains under-studied.   Understanding the validity of parent and family skills instruments with this population is needed in order to advance research into EBI effectiveness in child welfare.  The purpose of this study is to evaluate the validity of the five parenting scales and five family scales contained in the Strengthening Families Program (SFP) instrument for measuring change over time. 

Method:  Data were gathered in conjunction with a federal evaluation of SFP in a Midwestern state between 2008 and 2012.  The analytic sample contained 411 cases who completed at least one pair of pre and post-tests.  Psychometric testing involved 20 confirmatory factor analyses (CFA) on prospective pre-tests, retrospective pre-tests, and post-tests for five parent and five family scales.  After establishing the factor structure, three levels of invariance testing were conducted for each scale for prospective/post-tests and retrospective/post-tests to establish measurement invariance at the level of factor structure, factor loadings, and mean structure.  Change in CFI/TLI of <.01 was used to establish invariance.  If scales passed weak invariance testing, longitudinal structural equation models (LSEM) were estimated to compare pre-tests to post-tests. 

Results: Four of the five parent scales were reduced in number of indicators to achieve configural invariance.  Weak or partial weak invariance for all parent scales and partial strong invariance was achieved for four parent scales using the prospective pre-test.  Weak invariance was achieved for two of the five scales using the retrospective pre-test.  Two scales were combined in order to estimate the half panel structural model using the prospective pre-test and final 4-factor LSEM fit was acceptable [RMSEA=.053, CFI/TLI=.878/.863].  Only one of five family scales was reduced in number of indicators to achieve configural invariance, with one scale dropped because of item duplication and one created.  Weak or partial weak invariance and partial strong invariance was achieved for family scales using the prospective pre-test.  Partial weak invariance and partial strong invariance was achieved for four of five scales using the retrospective pre-test.  Using the prospective pre-test, final 4-factor LSEM fit was acceptable [RMSEA=.054, CFI/TLI=.838/.827].  Prospective pre-tests significantly predicted post-test scores for parent and family scales. 

Implications:  Improving parenting and family functioning are important outcomes for EBI research in child welfare, yet the psychometric properties of evaluation instruments are rarely reported.  Results of this study suggest that the validity of retrospective pre-tests may be reduced compared to prospective pre-tests for this population.  Moreover, our findings suggest that parents may struggle to validly assess their own parenting skills compared to their family’s functioning.  These findings suggest that pressures on child welfare-involved parents to exhibit strong parenting skills despite deficits in that domain—consistent with the phenomenon of social desirability bias—may threaten accurate measurement.