Concept mapping - structured conceptualization – a sequence of concrete operationally defined steps yielding a conceptual representation was used with agency staff to develop clusters of outcome statements developing a conceptual framework for the evaluation of a comprehensive homeless services agency. In meetings held with staff, each identified outcomes for the homeless services agency clients. Staff addressed two questions: What are the outcomes that you wish for your clients? What interventions or services do you offer to help clients achieve these outcomes?

Eighty non-redundant statements of client outcomes resulted from the work of staff members from the three agency program components. The second step in developing client outcomes involved engaging a smaller group of staff to sort the outcome statements into groups of like statements. The sorting resulted in seven distinct clusters: (1) Health & Mental Health Care; Self-Esteem & Self Awareness; (2) Relationships & Social Supports; (3) Abstinence & Recovery; (4) Life & Independent Living; (5) Employment & Self-Sufficiency; (6) Parenting & Child Development, and (7) Permanent Housing.

The next step involved creating rating scales used by case management staff to assess client change in each of the seven areas. The result of that work is the ESI Connections Client Assessment Scales (CCAS), and the Assessment of Life and Independent Living Skills (ALILS), a client self-report instrument. The CCAS measures multiple domains of client change using case managers' assessments of clients at intake and 3-, 6- and 12-month intervals. The case manager assigned to work with each client is in the best position to observe directly the client's progress for each dimension measured by the CCAS. The seven dimensions of the CCAS rating scales include multiple items with 5-point anchoring statements for each. Agency staff reviewed and helped revise the rating items and anchoring statements over the 18-month development period.

Each client voluntarily choosing to participate in the evaluation completes a battery of self-report assessment instruments in addition to demographic information (e.g., gender, age, race/ethnicity, relationship status, educational attainment, employment status, and work history) and the ALILS (a 20-item self-report assessment of life and independent living skills items parallel to the CCAS). The client self-report measures used to validate the CCAS and ALILS include: CES-D (Center for Epidemiological Studies Depression Scale) (Radloff, L. S. (1977); Rosenberg Self-Esteem Scale (Rosenberg, 1965); the Hope Scale (Snyder, et al., 1991); Alcohol Use Disorders Identification Test (AUDIT) (Babor, et al., 1992); Quality of Life Rating Scale (Lehman, 1988); self-report measure of social network; self-report measure of contact with members of the client's social network.

This paper reports the item analysis of the ALILS, the criterion validity of the CCAS, and the inter- and intra-rater reliability of case managers' judgments of client change using the CCAS. The presentation emphasizes the processes of developing evaluation instruments tailored to meet the needs of human services agencies with recommendations for participatory evaluation and mixed method approaches. Additionally, the paper describes the challenges of instrument development and evaluation.