Digital Systematic Social Observations: Promises and Pitfalls of a Neighborhood Measurement Approach

Background: Depending on the presence of risk- and protective factors, neighborhoods can compromise or strengthen the health and wellbeing of residents. Generally, research on these so called “neighborhood effects” has relied on secondary data sources, ethnographic accounts, resident surveys or interviews, or-- in a minority of cases-- systematic social observations (SSO’s). While systematic social observations offer several advantages to other neighborhood measurement strategies, they are notoriously resource intensive and raise concern regarding the intrusion of researchers into what are often marginalized communities. One way in which researchers might avoid these problems, while capitalizing on the benefits of SSO’s, is by conducting observations digitally (i.e., via virtual walks in Google Earth). This paper describes the process of conducting digital SSO’s. It then discusses the psycho- and ecometric properties of a neighborhood disorder measure collected via digital SSO.

Methods: The SSO was conducted as part of a larger study aimed at understanding the relationship between residential contexts and probationer recidivism. For this study, I identified a systematic sample of 504 street block faces in San Francisco, California, for observation.  Drawing on an approach previously used in child development research, research assistants collected data on six signs of neighborhood physical and social disorder as they virtually walked street blocks in Google Earth.  The six items were double coded in a subsample of blocks (n = 43) to assess inter-rater reliability.  I used one-way inter class correlation (ICC) and Cronbach’s alpha to test inter-rater reliability and internal consistency, respectively. I examined items and extracted standardized disorder scores for Census tracts with a two-level Item Response Theory (IRT) model.   I assessed convergent validity correlating the standardized scores with related Census and crime measures, and I assessed criterion validity by regressing criminal recidivism events on disorder scores.

Results: SSO’s yielded good interrater reliability for the disorder measure (ICC = .73), and acceptable internal consistency (alpha = .72). The IRT model indicated that items may best capture higher disorder contexts, and social disorder items may contribute instability to the overall measure.  The standardized, conditional (on item and block face random effects) disorder scores manifested a theoretically coherent pattern of relationships, correlating inversely with owner occupied households (r = -.22), positively with poverty (r = .32), and positively and more strongly with drug crime (r = .39). Disorder also predicted recidivism among nearby probationers (B = .38, p < .001).

Conclusion and Implications:  Results lend support to an emerging body of research on the utility of digital SSO’s.  We find that, when used to measure disorder, digital SSO’s can yield results with adequate psycho- and ecometric properties, on par with other neighborhood observational approaches. Challenges to using the digital SSO approach, limitations of disorder measures, and opportunities for future use will be discussed.