Learning by doing

What Demonstration Projects Can Teach
Us About Innovation and Criminal Justice

The Center for Court Innovation

Learning lessons
from the past
is not a particular strength of the criminal justice system.

Indeed, the history of criminal justice in the United States can be
read as a swinging pendulum, as policymakers have veered from
punitiveness to leniency and back again, without pausing to
remember why they initially favored one approach over the other.

Fear of failure

is, of course, part of human nature. Failure
is typically discussed only in hushed whispers in the world of civic affairs.

Men are greedy to
publish the successes of
[their] efforts, but meanly
shy as to publishing the
failures of men.

Men are ruined by
this one-sided practice
of concealment of blunders and failures.

–Abraham Lincoln

As Michael Scott of the Center
for Problem-Oriented Policing has
pointed out, police chiefs rarely
say, We had a great idea that just
didn’t work. We’re going back to the
drawing board to do it differently.

In other disciplines, most notably science,

solving problems
is viewed as an iterative process. Criminal
justice officials are rarely afforded the
opportunity to engage in a trial-anderror process because the results of failure
are so immediate: people can die, and
officials get fired.

That’s what
a scientist
would do.

But the typical
police chief doesn’t
feel that he has that
kind of latitude.

real consequences

But there are
when we fail to
talk about failure. Most obviously, it leads to an environment that stifles innovation. And without
innovation, it is hard to imagine solving difficult problems such as domestic violence, youth crime, and
chronic neighborhood disorder.

We need to change the
work environment so that it’s
not only physically safe but
intellectually safe for people to
learn and make mistakes.

Judith Sachwald

Former director of the Maryland Division of Parole and Probation

There are numerous reasons to encourage a more forthright dialogue about criminal justice and the challenges of reforming the system. Learning from mistakes is particularly important to those charged with implementing, evaluating, and funding demonstration projects.

Even initiatives that fall short of their goals can provide valuable information and guidance as innovators look to improve

criminal justice system in the future.

.,

the

The past generation has been a fertile period for criminal justice reform. Many new
initiatives have emerged in an effort to reduce crime and improve the functioning of the
system – COMPSTAT, drug court, HOPE Probation, Operation Ceasefire… The list
goes on and on. Almost all of these reforms began life as demonstration projects: smallscale pilots with a temporary life span. Numerous reports and case studies have been
devoted to culling the best practices from these projects. But what about the reverse? Are
there things to be learned from projects that struggled to achieve their goals?

Here are five such lessons:
1. Not all failures are alike.
2. Failure is in the eye of the beholder.
3. Things fall apart.
4. Context matters.
5. Beware of unrealistic expectations.

1. Not all failures are alike.
Failure is usually the product of a complicated chemistry involving a specific time, a specific place, and specific personalities. While
every programmatic failure has its unique elements, failures

generally fall into four distinct groups:

Failure of concept

1. 

(a bad idea)

Failure of implementation

2. 

(poor execution)
3.

Failure of marketing and politics
(an inability to attract the necessary money or manpower)

4.

Failure of self-reflection

(an inability to assess one’s own weaknesses and adapt to changes on the ground)

The St. Louis Consent to Search program, implemented in the 1990s, illustrates each of these types of failure.
Consent to Search was a reaction to the city’s alarmingly high
murder rate in the early 1990s. In an effort to get guns off
the street, the St. Louis Police Department conducted home
searches of young people previously arrested on gun charges –
with the consent of their parents and a pledge not to make
arrests based on what they found.

The program was surprisingly well-received
by the community, and over 500 guns were
seized in just 18 months.

However, despite promising results, after a change of leadership at the top
of the St. Louis police department, the program was scrapped. This was a

failure of politics,

with a small p: Consent to Search wasn’t shuttered based
on a hard analysis of the program’s benefits and costs.

Things started to fall apart
when the police chief resigned, which
was followed by a department shakeup. Almost overnight, no one in the
police department knew anything
about the program.

Richard Rosenfeld

Professor of Criminology and Criminal Justice at the University of Missouri – St. Louis.

The program was eventually re-started, but the pledge not to make an arrest, arguably
the key component of the program model, was eliminated. Not surprisingly, referrals
from the community dried up, fewer searches were conducted, and the number of
guns seized dwindled. The program was again discontinued, this time due to a

failure of concept:

the elimination of the
pledge was just a bad idea.

The program was re-started a third time, this time with the help of a federal grant and a local clergy group that
was brought on board to link families in crisis to needed services. Unfortunately, the police-clergy partnership
fell apart in mutual acrimony. The Consent to Search program soon followed suit. This illustrates a

failure of
implementation:
t he ideas were good, the project had adequate resources and local support, but the
execution was poor.

It is impossible to say if the errors of Consent to Search could have been avoided – hindsight is always 20/20. But it is
clear that the Consent to Search team lacked the institutional architecture to effectively analyze and document what they
were doing: these were street cops responding to challenges on the ground, not researchers. While understandable, this

failure of self-reflection
made it difficult for the project to respond as conditions on the ground changed.

2. Failure is in the eye
of the beholder.
Policymakers often have a pass-fail approach to evaluating social programs. The question they tend to ask is a
fairly basic one: “Does this program work or not?” But the truth is that Few

programs are utter

failures – or absolute successes.

D.A.R.E. (Drug Abuse Resistance Education) was launched in Los Angeles in 1983 to combat a rising drug
and alcohol abuse problem nationwide. The program trains police officers to lead educational sessions in local
schools, designed to help students resist peer pressure and live drug-free lives.

Unfortunately, numerous studies have documented that D.A.R.E. has no impact on teenage substance
abuse. Despite the negative findings,
like three out of four school districts in this country.

D.A.R.E. is still operating in something

Why is this? Carol Weiss and a team of researchers from Harvard sought to discover the answer. They found

that in many cases, local officials cited a range of other benefits from D.A.R.E., particularly
improved communication with local police, as a reason to keep the program.
The point here is that it is not so easy to divide the world into tidy piles marked “successes” and “failures.”
Even much-criticized programs like D.A.R.E. have supporters who can point to positive impacts.

3. Things fall apart.
Just because something works once, doesn’t mean that it will work forever.
An example of this phenomenon is Operation Ceasefire . The product of a collaboration among local
criminal justice agencies, street outreach workers, and scholars from Harvard University, Ceasefire was credited
with significantly reducing gang violence in Boston in the 1990s. However, after experiencing success on a scale
that few programs ever achieve – including trips to the White House and the cover of Newsweek – Ceasefire
fell apart. What

went wrong?

Teny Gross, a former Boston outreach worker who now runs the
Institute for the Study and Practice of Nonviolence, has commented on
Boston’s unraveling: “A decade ago, a young man in Dorchester told me,

‘You adults are the real gang members, easy to
feel slighted, fighting petty beefs, vying for attention and credit.’ It is the
beefs on the street that get the headlines. But the beefs in the offices and
agencies are now equally to blame for what is happening.”

Operation Ceasefire highlights the challenge of engaging in collaborative work , particularly
over the long haul. In the aftermath of Ceasefire’s success, bitter fights erupted among participants over credit
and public attention.
This discord was compounded by the departure of several key players from the coalition. The importance of
planning for succession, and building institutions and programs that do not rely entirely on heroic individuals,
cannot be overstated.

I’m the one
with the good
ideas here!

It’s my turn
to talk!

Who do you
think you are?

This is my stage,
Mister!

4. Context matters.
Program implementation is a tricky thing. What

works in Los Angeles might not work in
Chicago, let alone in a rural parish in Louisiana.

The biggest failure trap with anti-gun
violence programs is failing to appreciate
the different cultures related to guns
based on geographic region. Large urban
jurisdictions like Chicago, New York, and
Boston have very different attitudes
towards regulation of gun ownership
than places like Montana.
One common issue involves
something I call the “copycat”
problem. I’ve seen this across the
country, where a jurisdiction tries to
copy a successful program without
really thinking it through or having
regard for proper implementation.

Ed McGarrell

Director of the School of Criminal Justice at Michigan State University

Gary Hinzman

Former director of the Sixth Judicial District of the Iowa Department of Correctional Services

As Lisbeth Schorr, senior fellow at the
Center for the Study of Social Policy,
has noted, context is the most likely
saboteur of innovations. Just because
a program is wonderful doesn’t mean
that the surroundings won’t destroy it
when it is replicated in a new place.

t
i
k
l
o
o
T
One size
does not
fit all

5. Beware of unrealistic expectations.
the mismatch between
public expectations and what new programs can reasonably be expected
to deliver. Changing the lives of offenders, transforming crime-plagued neighborhoods, reengineering large
Perhaps the biggest challenge that criminal justice reformers must face is

bureaucracies…these are not easy goals to accomplish. Unfortunately, there are enormous pressures that make it
difficult to convey this message to elected officials, the media, and the general public.

There is nothing in our history
of over 100 years of reform
that says that we know how
to reduce recidivism by more
than 15 or 20 percent.

And to achieve those rather
modest outcomes, you have to get
everything right: the right staff,
delivering the right program, at
the right time in the offender’s life,
and in a supportive community
environment.

Joan Petersilia

Professor at Stanford Law School

Unless people promise
a lot, it’s hard to get a
program funded.

Todd Clear

Dean of the School of Criminal Justice at Rutgers University

Most criminal justice interventions
only work with people for a short period
of time. For example, a court-mandated
batterer intervention typically only
involves about 28 contact hours.
Changing behavior that has developed
over a lifetime in 28 hours is a tall order.

When you run an advertising
campaign for Toyota, changing
sales by a percentage point or two is
considered a huge success. The same
is true in running a big
election campaign.

Why is that different in
criminal justice?

Carol Weiss

Professor Emerita at Harvard Graduate School of Education

David Wilson

Chair of the Department of Criminology, Law and Society at George Mason University

There is no sure way to avoid the failure traps described in this publication.
But one reasonable step that demonstration projects can take is to invest in

action research
– using data to engage in an iterative process of testing new solutions,
analyzing the results, and learning from the successes and failures.

Self-examination is vital to the long-term health not just of reform efforts
but of the entire field of criminal justice. An ongoing commitment to
action research can help reformers make mid-course adjustments to flagging
programs. It can also help reduce the likelihood that today’s innovation
becomes tomorrow’s conventional wisdom that needs to be overturned.

I have not failed 5,000 times.
I have successfully discovered 5,000 ways that do not work and I do not need to try them again.

–Thomas Edison

Ideas

The Center for Court Innovation, with the support of the U.S. Department of
Justice’s Bureau of Justice Assistance, has embarked on a multi-faceted inquiry designed to promote trial and error in criminal justice
reform. Through case studies, roundtables, interviews, and site visits, researchers from the Center have examined demonstration projects
that attempted to improve the criminal justice system in one way or another. This includes both successes and failures. By analyzing these
experiments, the Center seeks to encourage honest self-reflection and thoughtful risk-taking among criminal justice agencies.
The Center for Court Innovation’s “trial and error” products include:

Trial and Error in Criminal Justice Reform: Learning from Failure, Greg Berman and Aubrey
Fox, Urban Institute Press, 2010.
Trial and Error in Criminal Justice Reform examines well-intended programs that for one reason or
another fell short of their objectives yet also had positive effects. The book encourages reformers to
learn from their predecessors, analyze their own foibles, and keep innovating.

Daring to Fail: First-Person Stories of Criminal Justice Reform, Aubrey Fox and Emily Gold, eds.,
Center for Court Innovation, 2010.
Daring to Fail is a collection of interviews with leaders in a variety of fields – prosecution, policing, community corrections, indigent defense, and others – about leadership, management and
innovation. While each interview is unique, taken together they offer vivid testimony that it is in
fact possible to make change happen within the criminal justice system – provided that officials are
encouraged to risk failure and given the time they need to engage in an iterative learning process.

.......
.......

“Lessons from the Battle over D.A.R.E.: The Complicated Relationship between Research and
Practice,” Greg Berman and Aubrey Fox, Center for Court Innovation, 2009.
“Lessons from the Battle over D.A.R.E.” offers an examination of the controversy over D.A.R.E.,
one of the most well-known and widespread crime prevention programs in the country, which has
thrived despite research showing less-than-inspiring results. The paper unpacks the complicated
relationship between research and practice, drawing lessons for future programming.

“Avoiding Failures of Implementation: Lessons from Process Evaluations,” Amanda Cissner
and Donald Farole, Jr., Center for Court Innovation, 2009.
“Avoiding Failures of Implementation” examines failures that occur during the implementation of
a new initiative, seeking to identify common sources of failure and to develop a basic list of considerations that may help practitioners avoid future pitfalls.

For more information and to see other “trial and error”
publications, visit www.courtinnovation.org/failure.

The winner of the Peter F. Drucker Award for Nonprofit Innovation,

The Center for Court Innovation
seeks to help the justice system reduce crime and improve public
trust in justice through demonstration projects, research, training,
and technical assistance.

CENTER
COURT
INNOVATION

520 Eighth Avenue • New York, NY 10018 • (646) 386-3100 • www.courtinnovation.org
Illustration and design by Danica Novgorodoff • www.danicanovgorodoff.com
Copyright © 2011 The Center for Court Innovation