Supported with funding from the Annie E. Casey Foundation, the Evidence Generation team is working with two innovative youth services agencies to improve their capacity for sound evaluation and to evaluate their effectiveness. The two agencies, Youth Advocate Programs, Inc. (YAP) based in Harrisburg, PA and exalt of New York City, are dynamic providers of community-based services and supports for disconnected, at-risk, and justice-involved youth.
The Evidence Generation initiative had already been working with both YAP and exalt for more than two years at the launch of the Casey project. During this time, staff worked with the agencies to document their routine practices, to create rigorous theories of change and logic models, and to explore the measures necessary to organize mission-relevant data. The enhanced funding from the Casey Foundation allows us to work with each agency to resolve any strategic issues remaining from our previous collaborative work. Next, we will help the agencies to create effective comparison samples that may be utilized for quasi-experimental evaluation studies.
Jeffrey A. Butts (PI), Doug Evans (Project Director), two Graduate Research Fellows (TBA), Rhoda Ramdeen
$150,000 in Year 1, with additional years possible. Annie E. Casey Foundation, Baltimore, MD.
In collaboration with the Research & Evaluation Center at John Jay College of Criminal Justice, the New York State Division of Criminal Justice Services(DCJS) developed a protocol for conducting follow-up studies of client recidivism for agencies affiliated with the R&E Center’s Evidence Generation initiative. Each study adheres to the data security safeguards required by DCJS. The resulting reports provide de-identified recidivism data that affiliated agencies may use to assess their effectiveness. Each report documents the proportion of former clients that had new justice contacts (i.e. arrests and convictions) after leaving an agency’s program(s), but the reports do not reveal the identity of individual clients.
We do not publish full reports, but brief examples are available here:
The criminal history reporting process begins when an agency submits a list of former clients with identifying information (name, sex, race, date of birth, etc.) to the research team at Evidence Generation. The clients are usually those served by the affiliated agency three to five years in the past (in order to allow sufficient follow-up time). Researchers from the Evidence Generation initiative conduct a criminal history search for each name on the list using the statewide records maintained by DCJS.
All researchers having any contact with DCJS records are first trained by DCJS in all relevant data security and privacy procedures. Before conducting any data searches, every analyst is also fingerprinted to ensure the security and confidentiality of the process.
The results of the records search are compiled into a de-identified database. Researchers first attach the criminal history records from DCJS to the records of client information supplied by affiliated agencies. All identifying information is then stripped from the database before the newly combined data file is returned to John Jay College. Researchers analyze the data to examine recidivism across different client characteristics and levels of program participation without including any specific information about the clients themselves.
The submitting agencies receive written reports about these analyses, but the reports do not name the agencies. Pseudonyms are used instead, such as Agency A, Agency B, etc. Findings from the analysis are not published. No identifiable data is even recoverable as it is removed before the file is transferred to the Evidence Generation team.
Participating agency affiliates know which report is their own, but they do not know the identity of other participating organizations. Reports include brief descriptions of programming offered by each affiliate, types of clients served, and the agency’s referral sources. This information helps provide context for comparing reports while preserving confidentiality.
These procedures provide affiliated agencies with previously unavailable, detailed information about long-term client recidivism. Agencies may then examine the relationship of their efforts to subsequent recidivism of their clients. As the Evidence Generation initiative accumulates more of these anonymous reports, we hope our affiliated agencies can begin to build evidence on their outcomes of interest. At a minimum, the anonymous report format supports simple, one-time comparisons, but each agency is encouraged to review their outcomes intermittently over several years, which is squarely in the spirit of the Evidence Generation initiative.
The Evidence Generation initiative plans to grow the scope and diversity of its DCJS criminal history projects. We invite all current agency affiliates and all potential affiliates to contact us in order to discuss the feasibility of new criminal history analysis efforts.