Problem‐oriented policing for reducing crime and disorder: An updated systematic...

1 PLAIN LANGUAGE SUMMARY

1.1 Problem‐oriented policing (POP) is associated with reductions in crime and disorder

POP is associated with statistically significant reductions in crime and disorder. Place‐based POP programs are more likely to produce a diffusion of benefits into areas adjacent to targeted locations than to lead to crime displacement.

1.2 What is this review about?

POP is a proactive policing strategy developed by Herman Goldstein, who argued that the standard reactive model of policing was ineffective as it was overly focused on the means of policing (number of arrests, average response time, etc.) rather than the end goal of reducing crime and enhancing community safety. He suggested that police could be more effective if they were more proactive and researched root causes of crime, and developed tailor‐made responses.

1.3 What studies are included?

This review includes both randomized and quasi‐experimental evaluations of POP, where a treatment area or group received a POP approach while a control area or group received standard police services.

Thirty‐four studies are assessed in the review—an increase of 24 studies from the original review (Weisburd, Telep, Hinkle, & Eck, 2008; Weisburd, Telep, Hinkle, & Eck, 2010). All studies were published between 1989 and 2018. Most studies (28) were conducted in the United States, five in the United Kingdom, and one in Canada.

1.4 Does POP reduce crime and disorder?

Yes. The results of this updated systematic review suggest that POP is associated with a statistically significant overall reduction in crime and disorder of 34%.

There are positive impacts for POP across a wide variety of crime and disorder outcomes, among studies that targeted problem places and problem people, at a variety of different units of analysis and featuring a wide array of types of interventions. The effect size is smaller in randomized experiments and after accounting for publication bias.

POP had limited impacts on police legitimacy, fear of crime, and collective efficacy. Few studies incorporated cost‐benefit analyses, but those that did suggest POP can be cost‐effective and provide substantial savings through prevented calls‐for‐service and incidents.

1.5 What do the findings of the review mean?

Findings from this review support the notion that proactive policing strategies that identify specific problems, conduct analyses to determine underlying causes, and develop and deliver tailor‐made responses, are more effective in reducing crime and disorder than standard, reactive methods of policing. Moreover, in place‐based interventions, diffusion of crime‐reduction benefits are more likely than displacing crime to nearby areas. As such, police departments should incorporate the use of problem‐solving into their crime prevention strategies.

However, the impacts of POP on crime are highly heterogeneous. This result may reflect the tremendous variability in the types of problems identified and targeted and the types of tailored intervention strategies used, suggesting that more studies are needed to allow more robust analyses of factors that influence POP program impacts. In turn, future evaluations should be designed to capture more data about the problem‐solving process so that future reviews can more directly assess what types of problems seem most amenable to POP efforts and what characteristics of problem‐solving interventions are associated with larger effects.

1.6 How up‐to‐date is this review?

This authors of this review update searched for studies up to December 2018.

2 BACKGROUND

2.1 The issue

In an article in Crime & Delinquency in 1979, Herman Goldstein critiqued police practices of the time by noting that they were more focused on the “means” of policing than the “ends” or goals of policing. His critique drew from a series of recently completed studies that suggested that such standard policing practices as “preventive patrol” (Kelling, Pate, Dieckman, & Brown, 1974) or “rapid patrol car response to calls for service” (Kansas City Police Department, 1977) had little impact on crime. Goldstein suggested that the research evidence was not idiosyncratic but reflected a crisis in policing. To illustrate his concern, he referred to a newspaper article in the United Kingdom that reported on bus drivers in a small city that were passing bus stops waving and smiling but failing to pick up passengers. When questioned by a reporter, a representative for the bus company responded that “it is impossible for the drivers to keep their timetable if they have to stop for passengers” (Goldstein, 1979, p. 236). Goldstein termed this the “means over ends syndrome” and noted that police were particularly susceptible to this problem. Goldstein noted that the police too had become so focused on the means of policing—such issues as the staffing and management of police—that they had begun to ignore the problems policing was meant to solve. Goldstein saw this dysfunction as at the heart of the failures of the police to be effective in addressing community problems.

Goldstein called for a paradigm shift in policing that would replace the primarily reactive, incident driven “standard model of policing” (Skogan & Frydl, 2004; Weisburd & Eck, 2004) with a model that required the police to be proactive in identifying underlying problems that could be targeted to alleviate crime and disorder at their roots. He termed this new approach “problem‐oriented policing” to accentuate its call for police to focus on problems and not on the everyday management of police agencies. Goldstein also expanded the traditional mandate of policing beyond crime and law enforcement. He argued that the police should deal with an array of problems in the community, including not only crime, but also social and physical disorder.

He also called for police to expand the tools of policing much beyond the law enforcement powers that are seen as the predominant tools of the standard model of policing. In Goldstein's view, the police needed to draw upon not only the criminal law, but also civil statutes and rely on other municipal and community resources if they were to successfully ameliorate crime and disorder problems. As such, successful implementations of POP would be reliant on forming partnerships with other agencies, community organizations and community members to deliver non‐law enforcement responses. This would particularly be the case when the targeted problems do not necessarily involve law violations.

John Eck and William Spelman (1987) drew upon Goldstein's idea to create a straightforward model for implementing POP, which has become widely accepted. In an application of problem solving in Newport News, in which Goldstein acted as a consultant, they developed the SARA model for problem solving. SARA is an acronym representing four steps they suggest police should follow when implementing POP, which will be outlined in Section 2.2.1

A number of studies going back to the mid‐1980s demonstrate that problem solving can be utilized to address a variety of police concerns, including fear of crime (Cordner, 1986), violent and property crime (Eck & Spelman, 1987), firearm‐related youth homicide (Kennedy, Braga, Piehl, & Waring., 2001), and various forms of disorder, including prostitution and drug dealing (Capowich & Roehl, 1994; Eck & Spelman, 1987; Hope, 1994). As a further example of the proliferation of POP, the Center for Problem‐Oriented Policing (POP Center, https://popcenter.asu.edu/) documents a large number of case studies and evaluations of POP. For instance, there have been over 1,000 programs submitted for consideration for the Goldstein Award and more than 800 submissions to the Tilley Award. These submissions document the use of a wide array of problem‐solving responses to document crime, disorder and a host of other issues police are tasked with addressing, highlighting the utility of the POP model for a wide variety of problem types (see also, Scott, 2000; Scott & Clarke, 2020). As our review is focused on impacts on crime and disorder, we limit our discussion here to those outcomes.

There are also a number of experimental and other more rigorous examinations of POP. For example, a study in Jersey City, New Jersey, public housing complexes (Mazerolle, Ready, Terrill & Waring, 2000) found that a police problem‐solving model could be used to respond to violent and property crime in six housing complexes. In another example, Clarke and Goldstein (2002) report a POP project to reduce thefts of appliances from new home construction in Charlotte, North Carolina. Officers carefully analyzed this problem before working with construction firms to implement changes in building practices.

Two early experimental evaluations of applications of problem solving in crime hot spots (Braga et al., 1999; Weisburd & Green, 1995) suggested POP interventions, particular those implemented in crime hot spots, could be evaluated rigorously.2 In a randomized trial involving Jersey City violent crime hot spots, Braga et al. (1999) examined the impact of problem solving in 12 hot spots on property and violent crime. While this study tested problem‐solving approaches, it is important to note that focused police attention was brought only to the experimental locations. Accordingly, it is difficult to distinguish between the effects of bringing focused attention to hot spots and that of such focused efforts being developed using a problem‐oriented approach. The Jersey City Drug Market Analysis Experiment (Weisburd & Green, 1995) more directly tested the value added of problem‐solving approaches in hot spots policing. In that study, a similar number of narcotics detectives were assigned to treatment and control hot spots. Weisburd and Green (1995) compared the effectiveness of unsystematic, arrest‐oriented enforcement based on ad hoc target selection (the control group) with a treatment strategy involving analysis of assigned drug hot spots, followed by site‐specific enforcement and collaboration with landlords and local government regulatory agencies, and concluding with monitoring and maintenance for up to a week following the intervention. More recent experimental evaluations have also examined the impact of POP in crime hot spots (e.g., Braga & Bond, 2008; Groff et al., 2015; Taylor, Koper, & Woods, 2011).

In sum, POP has emerged as one of the most widely accepted and widely used strategies in American policing (Scott, 2000; Weisburd & Majmundar, 2018). This is indicated both by the adoption of POP by major federal agencies and national policing groups, the creation of national awards for effective POP programs, and the widespread adoption of the approach in American policing and throughout the world. For example, the U.S. federal agency, the Office of Community Oriented Policing Services (COPS) adopted POP as a key strategy, initially funding the Center for Problem‐Oriented Policing, which has developed over 70 problem‐specific guides for police. More recently, the Bureau of Justice Assistance has also funded the creation of problem‐oriented response and tool guides. The Police Executive Research Forum adopted POP as a “powerful tool in the policing arsenal,” in the 1980s and began to run a yearly national conference to promulgate and advance POP strategies (Solé Brito & Allan, 1999, p. xiii) that the POP Center still continues today. In 1993 the Herman Goldstein Award was created for “problem solving excellence.” In the United Kingdom, the Tilley Award for POP was created in 1999.3 To date there have been more than 1,800 submissions to these awards. Reflecting the wide‐scale adoption of POP by American police agencies, the 2013 Law Enforcement Management and Administrative Statistics (LEMAS) survey reported that 33% of all departments, and 74% of departments serving 100,000–249,000 citizens, reported actively encouraging officer involvement in problem‐solving projects (Reaves, 2015).4

While POP has been widely adopted and assessed, it is important to note that fully implementing POP has been challenging (Cordner & Biebel, 2005; Maguire, Uchida, & Hassell, 2015), and programs are often characterized by partial implementations of the SARA model that have been termed “shallow” problem‐solving (Braga & Weisburd, 2010). For instance, in his Stockholm lecture Goldstein (2018) noted that many early initiatives lacked fundamental understanding of the POP approach and that he did not adequately acknowledge the importance of having enough individuals with the requisite research and assessment skills when developing his model. However, he also noted that he has been impressed by improvements in areas such as focusing specifically on micro‐problems, the engagement of rank‐and‐file officers in problem solving, the use of a broad range of responses, and increasing engagement of the private sector in partnerships to share responsibility for public safety problems. Thus while POP still has a long way to go to be fully embedded in police agencies, much less to become standard police practice as Goldstein hoped, there is reason to believe that the model is both spreading and improving in quality over time.

2.2 POP in practice

Since its initial proposition, the POP model has been further articulated by Eck and Spelman (1987) whose work in Newport News produced the SARA model. SARA is an acronym representing four steps they suggest police should follow when implementing POP. “Scanning” is the first step, and involves the police identifying and prioritizing potential problems in their jurisdiction that may be causing crime and disorder. After potential problems have been identified, the next step is “Analysis.” This involves the police analyzing the identified problem(s) in‐depth using a variety of data sources so that appropriate responses can be developed. The third step, “Response,” has the police developing and implementing interventions tailored to what was learned in the “analysis” step and designed to solve the problem(s). The search for responses should be broad and not limited to law enforcement methods, and often should involve partnering with other agencies, community groups and/or community members depending on the type of problem and its causes. Indeed, the POP model stresses the need to shift and share responsibility for public safety, and this will require police to identify and mobilize partners (Goldstein, 2018; Scott & Goldstein, 2005). Finally, once the response has been administered, the final step is “Assessment” which involves assessing the impact of the response on the targeted problem(s).

For example, a police agency may determine that drug‐related crime is on the rise in their jurisdiction, constituting a problem in need of prioritization (Scanning phase). Further examination of the nature of drug‐related crime may reveal problem areas and times (Analysis phase). Based on this analysis, the agency may choose to direct increased patrol and enforcement to the specific areas deemed problematic at the specific times deemed problematic and to partner with community organizations to deliver substance abuse treatment programs (Response phase). After a period of time the agency may compare drug‐related crime in the jurisdiction as a whole, as well as in the targeted areas, from before and after the response was implemented (Assessment phase).

This process in general, rather than the specific problem or response chosen, represents the core concept of POP. Thus, a diverse set of variations in problems, responses, and length of interventions are possible across an array of targets (i.e., problem places of varying size or problem people may be the focus) and virtually any unit of analysis.

2.3 How might POP work?

POP is not concerned simply with the problem outcomes themselves but rather with the underlying processes that lead to problems emerging and developing. Addressing the underlying mechanisms that cause problems should lead to long‐term solutions and should lead police agencies to think and act in ways that go beyond their normal day‐to‐day operations. Furthermore, the assessment of results should lead to refinement and improvement in subsequent efforts.

Moreover, while the POP model does not favor any particular kind of intervention, one can still look beyond this literature and find support for the basic ideas behind the models by examining evidence for approaches commonly utilized in POP programs. For instance, evidence of the effectiveness of situational and opportunity‐blocking strategies, while not necessarily police based, provides indirect support for the effectiveness of problem solving in reducing crime and disorder. Moreover, POP has been linked to routine activity theory, rational choice perspectives, and situational crime prevention (Clarke, 1992a, 1992b; Eck & Spelman, 1987). Reviews of prevention programs designed to block crime and disorder opportunities in small places find that most of the studies report reductions in target crime and disorder events (Eck, 2002; Poyner, 1981; Weisburd & Telep, 2014; Weisburd, 1997),5 and many of these efforts were the result of police problem‐solving strategies. Further, a systematic review and meta‐analysis of situational crime prevention both supports its effectiveness and that such approaches do not merely displace crime to other areas (Guerette & Bowers, 2009). Lastly, hot spots policing and focused deterrence approaches that involve problem‐solving have been found effective in recent systematic reviews (Braga, Turchan, Papachristos, & Hureau, 2019; Braga, Weisburd, & Turchan, 2019).

2.4 Why is it important to do this review?

This is an update to an earlier Campbell systematic review of the effectiveness of POP that included studies through 2006 and identified a total of 10 studies that met the Campbell criterion for inclusion—4 randomized experiments and 6 quasi‐experiments (Weisburd et al., 2008, 2010). Overall, the findings of this review largely reinforced those of prior narrative reviews and more general assumptions of the effectiveness of POP. Specifically, the authors noted “[w]hether we used a more conservative mean effect size approach or examined the largest effects on crime and disorder reported, we found that POP approaches have a statistically significant effect on the outcomes examined. Importantly, the results are similar whether we look at experimental or nonexperimental studies” (Weisburd et al., 2010, p. 162).

However, the original review also noted that effect sizes were relatively modest, ranging between 0.10 and 0.20 (measured as Cohen's D) and were based upon only 10 experimental or quasi‐experimental studies. As such, an updated review may help to shed further light on the ability of POP to reduce crime and disorder problems by analyzing an increased base of empirical research on POP interventions by including studies up to the end of 2018 (12 years beyond the cutoff for the original review's search). Having more complete and current evidence on POP is especially important given an increasing focus on problem‐solving and other proactive policing approaches around the world (Weisburd & Majmundar, 2008).

We also add an additional approach to measuring effect sizes suggested by Wilson (in progress) that has statistical properties better suited to the nature of place‐based data and provides a more easily interpretable set of estimates of program outcomes. As empirical knowledge on POP's effectiveness increases, police agencies may be able to better determine ways to identify and respond to the various problems occurring in their jurisdictions.

3 OBJECTIVES

The objectives of this updated review are to extend the findings of the original review (Weisburd et al., 2008, 2010) by synthesizing the findings of published and unpublished evaluations of POP through December 2018 to assess its overall impacts on crime and disorder. Spatial displacement was also assessed for studies that provided data needed to calculate effect sizes for such effects. Finally, while too few studies included outcomes other than crime or disorder to allow for meaningful meta‐analyses, impacts on items such as police legitimacy and fear of crime are reviewed narratively, as well as findings about the financial cost/benefits of POP.

4 METHODS

4.1 Criteria for considering studies for this review

4.1.1 Types of studies

For studies to be considered in this review the evaluation had to include a target area or group that received a POP intervention AND a control area/group that received standard police services. The control condition could be either experimental or quasi‐experimental (Campbell & Stanley, 1966; Cook & Campbell, 1979; Shadish, Cook, & Campbell, 2002).

Unlike some Campbell reviews, we included studies with nonequivalent control groups; for example, studies that compared a target area to the rest of the jurisdiction. As the POP model requires police to identify specific problems in specific areas or populations, it will often be difficult for evaluators to create equivalent comparison areas/groups (Eck, 2006a). As such, we did not restrict our review to quasi‐experiments with equivalent control groups as we felt it important to be inclusive of studies that were representative of how POP is often carried out in practice. Thus any evaluation of POP that included a comparison group that did not receive the POP intervention was eligible for our review if it met our other inclusion criteria

4.1.2 Type of areas/groups

As noted above, POP is a general approach that calls for police to identify specific problems and develop specific responses to them based on potential underlying causes determined through problem analysis. As such, POP is not limited to any specific unit of analysis. For example, problems can be citywide, confined to small areas such as hot spots or can be individual offenders or groups of offenders rather than places. As such our review is not restricted by the type of target and includes problems at any unit of analysis that were addressed with a POP intervention.

4.1.3 Types of interventions

Given that the POP model calls for police to develop tailor‐made responses designed to address underlying causes of identified problems, a nearly limitless array of interventions can be associated with the approach. As such, our review is not restricted to any specific type of police response to crime or disorder problems. In this review we treat the use of the SARA model described above to identify problems, research underlying causes and develop and deliver specific responses to address them as the “intervention.” That is to say that our central question is whether using the SARA model to identify and respond to problems is associated with larger crime reduction compared with traditional reactive policing strategies. Further, we did not require that publications specifically mention the SARA steps (or even POP). We carefully read every potentially eligible study identified through our search and included studies if we could determine the interventions roughly followed the tenets of the SARA model.6

4.1.4 Types of outcome measures

The primary outcomes examined in this review, and included in our meta‐analyses, are measures of crime and disorder. By far the most commonly used measures of these outcomes in evaluations of POP are police recorded calls for service or incident reports. However, all measures of crime and disorder such as arrests, social observations or resident perceptions were coded. We also coded survey measures of other outcomes such as citizen perceptions/opinions of police, fear of crime, and collective efficacy where possible. We had hoped to get enough of these types of measures to conduct meaningful meta‐analyses on some of these types of outcomes. However, few studies reported on more than crime/disorder outcomes, and those that did are characterized by wide variation in measures used and data reported in study publications. As such we provide a narrative review and summary of the limited findings for such outcomes, as well as cost‐benefit analyses. We also conducted a meta‐analysis of displacement/diffusion effects, and the narrative summaries in Appendix A also discuss conclusions about these effects drawn in studies that did not provide data needed to calculate these effect sizes.

4.2 Search strategy for identification of studies

The search for this updated review was led by the Global Policing Database (GPD) research team at the University of Queensland (Elizabeth Eggins and Lorraine Mazerolle) and Queensland University of Technology (Angela Higginson). The University of Queensland is home to the GPD (see http://www.gpd.uq.edu.au), which served as the main search location for this review. The GPD is a web‐based and searchable database designed to capture all published and unpublished experimental and quasi‐experimental evaluations of policing interventions conducted since 1950. There are no restrictions on the type of policing technique, type of outcome measure or language of the research (Higginson et al., 2015). The GPD is compiled using systematic search and screening techniques, which are reported in Higginson et al. (2015) and summarized in detail in Appendices B and C. Broadly, the GPD search protocol includes an extensive range of search locations to ensure that both published and unpublished research is captured across criminology and allied disciplines.

Several additional strategies were also used to extend the GPD search. First, we performed forward citation searches for works that have cited seminal POP studies.7 Second, we conducted hand searches of 2017 and 2018 volumes of leading journals in the field to identify any recent studies that may have not yet been indexed in the GPD.8 Third, we reviewed the Center for Problem‐Oriented Policing website for all Tilley Award and Goldstein Award winners and submissions.9 Fourth, after finishing the above searches and reviewing the studies as described later, we e‐mailed the list to leading policing scholars knowledgeable in the area of POP (see list in Appendix D). This was aimed at identifying studies the above searches missed, as these experts may be able to refer us to eligible studies missing from our list, particularly unpublished pieces such as dissertations and smaller research reports.

Several strategies were used to obtain full‐text versions of the studies found through our search. First, we attempted to obtain full‐text versions from the electronic journals available through the George Mason University, Georgia State University, and Arizona State University libraries. When electronic versions were not available, we used print versions of journals available at the library. If the journals were not available, we made use of both the GPD team and the Interlibrary Loan Office (ILL) to obtain the journal from the libraries of other area schools. When those methods did not work, we contacted the author(s) of the article and/or the agency that conducted and/or funded the research to try to get a copy of the full‐text version of the study.

4.3 Data collection and analysis

Search results were given title and abstract review by Kevin Petersen, one of the authors of this review. Any studies that were not obviously eligible or ineligible were flagged. Flagged studies were reviewed by the other three authors of this review, who then discussed and voted on each study's eligibility. All inclusion/exclusion decisions were unanimous.

4.3.1 Details of study coding categories

The full coding sheet is provided in Appendix E. Kevin Petersen (one of the authors of the review) and another graduate research assistant at George Mason University independently coded each eligible study. Where there were discrepancies, Drs. Hinkle, Weisburd and Telep reviewed the study, had discussions and voted to determine the final coding decision.

4.3.2 Statistical procedure and conventions

We completed a meta‐analysis of the 34 eligible studies by calculating a standardized effect size for each included outcome and then estimating an overall random effect for the impact of POP on crime and disorder. We used Biostat's Comprehensive Meta Analysis 3.0 program for our analyses and to create the forest plots we present below.

Computation of effect sizes in the studies was not always direct. The goal was to convert all observed effects into a standardized mean difference effect size metric. None of the studies we examined calculated standardized effect sizes, and indeed, it was sometimes difficult to develop precise effect size metrics from published materials. This reflects a more general problem in crime and justice with “reporting validity” (Farrington, 2006; Lösel & Köferl, 1989), and has been documented in recent reviews of reporting validity in crime and justice studies (see Perry & Johnson, 2008; Perry, Weisburd, & Hewitt, 2010).

For many of our eligible studies, effect sizes could only be calculated using pre‐ and postintervention crime/disorder counts for the treatment and control group/area. A similar approach was used for some studies in our earlier review and is common in systematic reviews of policing interventions (e.g., Braga and colleagues’ [2019] recent update of their Campbell review of the effectiveness of hot spots policing).

The variance of the log of the RIRR (V(log RIRR)) was adjusted for over‐dispersion using the approach outlined by Farrington, Gill, Waples and Argomaniz (2007).10 This adjustment is calculated as the product of V(log RIRR) and D, with D = 0.0008 × N + 1.2. N is indexed as the mean number of incidents per case and is calculated as the total number of incidents (a + b + c + d) divided by the total number of treatment plus control areas/groups.

Finally, Cohen's D is obtained by multiplying the log of RIRR by √3/π, while its standard error is calculated by multiplying the adjusted V(log RIRR) by (3/π2; Hasselblad & Hedges, 1995).

While the Cohen's D approach allows us to compare our findings to the prior review, Wilson (in progress) argues that the Cohen's D approach fails to produce effect sizes that are comparable across studies when based on place‐based count data (the majority of studies in our review). Moreover, he has also pointed out that Cohen's Ds obtained through the above conversion are not comparable to those calculated directly through conventional means. As such, we also present meta‐analysis models where the effect size is the log RIRR and its standard error (which is the square root of adjusted V(log RIRR)).11 This approach also has an advantage in that the exponent of the log RIRR can be interpreted simply as the relative percent change in the treatment group compared with the control group.

4.3.3 Determination of independent findings

We first note that a few studies had multiple publications found through our searches. In these cases, the publication that provided the data used to calculate effect sizes was considered the main study and that is what is listed in tables, figures and the text. In cases where the effect size data were available in multiple publications, we treated the peer‐reviewed journal article as the main publication for the study (including in our coding of publication type). Secondary publications associated with the project that may have been used to help complete other items on our coding instrument are listed below the main publication in the list of eligible studies (via “see also” notes) in Section 5.1.1. There were no cases where unique crime/disorder outcomes for our main analyses were found across publications for the same study.

A common problem in conducting meta‐analyses in crime and justice is that investigators often do not prioritize the outcomes examined. This is common in studies in the social sciences in which authors view good practice as demanding that all relevant outcomes be reported. However, the lack of prioritization of outcomes in a study raises the question of how to derive an overall effect of treatment. For example, the reporting of one significant result may reflect a type of “creaming” in which the authors focus on one significant finding and ignore the less positive results of other outcomes. However, authors commonly view the presentation of multiple findings as a method for identifying the specific contexts in which a treatment is effective. When the number of such comparisons is small and therefore unlikely to affect the error rates for specific comparisons such an approach is often valid.

This is a particularly important issue for the current review. Given that POP calls for police to identify specific problems and develop tailor‐made solutions, it is important to include only outcomes likely to have been impacted by such focused responses. For example, in the Mazerolle et al. (2000) study, the authors noted that the Beat Health program “uses a variety of tactics to resolve drug and disorder issues” (p. 220). The authors present data on calls for service for disorder, drug crime, property crime, and violent crime. Because of their description of the intervention, we chose to include only drug and disorder calls as primary outcomes, and these were the outcomes we used for our mean effect size discussed below.

A primary outcome is defined in our review as one that was the direct focus of the POP intervention. The police needed to be specifically targeting the crime or call type in an outcome for us to identify an outcome as primary. We note that we erred on the side of being inclusive and only excluding reported crime/disorder outcomes in cases where the studies made it abundantly clear that only certain reported outcomes were the direct target of the tailored POP intervention. As such, for the vast majority of studies we include all reported crime/disorder outcomes.

We also note that it is important to examine variation in impacts across outcomes, and as such we analyze the studies using two approaches. The first is conservative in the sense that it combines all relevant outcomes reported into an overall average effect size statistic for each study. Second, to provide a range of effects, we also present separate models based on the largest and smallest effect for each study with multiple included outcomes. For studies with a sole outcome, or a clearly‐specified primary outcome, the same effect size is reported in all models. We also examined the impacts of POP across crime type.

In addition to providing a range, this approach is important as in some of the studies with more than one outcome reported, the largest outcome reflected what authors thought would be the largest program effect. This was true for the Jersey City Drug Market Analysis Experiment, which examined violent and property crimes, but assumed that the largest program effects, given the nature of the intervention, would be found in the case of calls for disorder (Weisburd & Green, 1995).

4.3.4 Treatment of qualitative research

Qualitative research on crime and disorder outcomes was not included in this review. Our goal was to summarize the findings of experimental and quasi‐experimental evaluations of the quantitative impacts of POP on crime and disorder. Purely qualitative studies do not meet the inclusion criteria of the GPD and would not have come up in our searches. The authors encourage other researchers to examine whether there is a sufficient amount of qualitative research on POP to warrant a systematic review.

5 RESULTS

5.1 Selection of studies

5.1.1 Results of the search

Search strategies in systematic reviews return a large number of results that must be screened for eligibility. Utilizing the GPD helped keep this number more manageable. Even though the GPD search strategy is much more comprehensive than those typically employed in searches by researchers conducting individual reviews, the studies included in the database have already been screened and confirmed to be policing evaluations that meet their methodological criterion (see above and the full details provided in Appendices B and C).

The initial steps of the review consisted of reviewing titles and abstracts to eliminate any duplicates and studies that were clearly not evaluations of POP. For any studies that could not be eliminated at this this step, we obtained the full‐text of the articles, reports, theses/dissertations or books for careful review to assess whether the interventions and evaluations met the eligibility criterion.

In total, the GPD searches and other strategies used in this review yielded a total of 2,464 results to review. Reviewing titles and abstracts eliminated 1,481 studies which were clearly not evaluations of POP. This left 983 studies which received full‐text review. Of these 39 met our eligibility criteria, and 24 provided the quantitative data needed to calculate effect sizes for our meta‐analyses. As the original Campbell systematic review of POP (Weisburd et al., 2008, 2010) included 10 experimental or quasi‐experimental studies in their main analyses we have a total of 34 studies included in our summary tables and meta‐analyses.

Of the 983 publications which received full‐text review, 746 studies were Goldstein or Tilley Award submissions (all available award submissions from 2006 to 2018 received full‐text review), 204 studies were GPD search results, and 33 studies were identified via forward citation searches.

Figure 1 provides a visual summary of the number of eligible studies by year of publication. As the graph highlights, there was a clear increase in evaluations of POP in the years after the 2006 cutoff for our original review. This uptick was relatively evenly spread over the 12‐year period, with each year other than 2014 having between one and five eligible studies. While it is possible that some of the increase is due to the use of the GPD for searches for this update, the data suggest that there has simply been an increase in experimental and quasi‐experimental evaluations of POP since 2006. For example, the original review only included 2 Goldstein/Tilley award submissions, while our update found 11 new submissions which met our inclusion criteria and included the data needed for effect size calculations.

Below we list the 49 studies that met our inclusion criterion. We first list the 34 studies which provided the quantitative data needed to be included in our meta‐analyses. We refer to these as “included studies.” As noted above, the publication that provided data for the effect size(s) for each study is listed, and any supplemental publications that were used to complete other parts of our coding of studies are listed in sub‐bullets via “see also” notes. Below this, we list the 15 studies that are not included in our analyses due to not providing sufficient data to calculate effect sizes for either of our approaches. Both lists are in chronological order.

Several studies that received full‐text review after the initial abstract screening were excluded after determining that they did not meet our inclusion criteria. These studies are noted in Appendix F.

5.2 Characteristics of selected studies

Table 1 provides an overview of the 34 studies that are included in our meta‐analyses. In terms of location, 28 studies (82.4%) were conducted in the United States, 5 (14.7%) in the United Kingdom and 1 (2.9%) in Canada. Studies were conducted in a total of 23 U.S. cities and 2 counties across 17 states. Lowell, MA, Jersey City, NJ and Philadelphia, PA all served as the jurisdiction for two studies. The U.K. studies included a total of eight jurisdictions, with Lancashire Constabulary serving as a study site in three evaluations. Vancouver was the setting of the Canadian study.

The study documents we identified were predominantly peer‐reviewed journal articles (N = 13, 38.2%) and submissions to the Goldstein Award (N = 13, 38.2%) for excellence in POP (no Tilley Award submissions met our inclusion criteria). There were also 4 (11.8%) research reports, 2 (5.9%) doctoral dissertations, and 2 (5.9%) master's theses. A few scholars served as an author or coauthor on multiple included studies, including David Weisburd (four studies and a coauthor of this review), Lorraine Mazerolle (three studies), Anthony Braga (two studies), and Brenda Bond (two studies).

In terms of rigor of research design, our sample of studies includes 9 (26.5%) randomized experiments and 25 (73.5%) quasi‐experiments. This is an increase from four randomized experiments and six quasi‐experiments in the original review and suggests a trend toward more rigorous evaluation of POP since 2006.

Turning to the unit of analysis for the POP interventions in these studies, 26 (76.5%) programs targeted problem places, 4 (11.8%) were targeted at place managers, and 4 (11.8%) targeted individuals. Three of the individual‐focused programs targeted problem offenders, while one intervention was aimed at potential victims. Eight studies of place‐based POP approaches quantitatively assessed displacement and diffusion effects, and we conduct a meta‐analysis on these effects.

Table 2a provides a quick overview of the type of problems targeted and the type of responses delivered, while Table 2b provides a detailed summary of the studies based on type/depth of scanning for problems and problem analyses used, the responses delivered and the research design. The latter columns in the table also highlight implementation problems and research design limitations where applicable. These are discussed in detail in the following two sections.

Finally, 15 of 34 (44.1%) studies reported significant reductions in at least one crime or disorder outcome, while another 17 (50%) studies reported raw differences favoring the treatment group for at least one crime or disorder outcome. Table 3 provides a summary of study conclusions about impacts on crime and disorder, as well as displacement and diffusion of crime control benefits where applicable.

5.3 Study implementation

While there was a relative lack of substantial complications reported, 67.6% (n = 23) of the 34 included studies did identify some degree of difficulty during implementation. Based on our coding criteria, the severity of these issues was classified as either minor (n = 15), more substantial (n = 7), or major (n = 1). There also appeared to be thematic consistencies across the studies in the types of issues reported. While such issues are in no way limited to POP interventions, this is perhaps indication that POP interventions are at greater risk of certain complications. Brief summaries of each study's implementation issues are also presented in Table 2b.

Given that POP is an iterative process, such that problems and responses are often continually changing, implementation issues may arise from problem instability. In the Philadelphia Policing Tactics experiment, Groff et al. (2015) note that nearly half of the POP intervention hot spots began focusing on nonviolent crime problems after determining that violent crime was no longer the primary concern in the area. Specifically, eight POP sites were noted to have targeted nonviolent or quality of life offenses, and at least four POP sites were noted to have focused on drug crime in addition to a violent or nonviolent crime problem. This is an issue from an evaluation standpoint as violent crime incidents were the only measured outcomes; thus it is possible that the interventions were effective in ameliorating the problems that they targeted, but this was unclear absent measurement of those outcomes. The RECAP experiment (Sherman et al., 1989) also suffered from issues related to problem instability. Specifically, call trends for many high‐crime addresses were remarkably heterogeneous from year to year, subsequently reducing the experiment's statistical power. The RECAP experiment and the Philadelphia Policing Tactics experiment both suffered from additional resource constraints as well. In the Philadelphia Policing Tactics experiment, POP officers were not dedicated to the intervention full‐time, but were instead drawn from patrol and expected to conduct POP activities during their free time. Similarly, in the RECAP experiment, there were likely too many addresses assigned to the experimental unit, thus spreading the unit's resources too thin and, perhaps, contributing to the lack of effectiveness in the second half of the intervention year.

The reality of resource constraints and other internal barriers to proper program implementation was not uncommon across these studies. Stokes (1996) reported that the safe travel corridor was poorly staffed during the afternoon hours, despite violence being more prevalent during this time. It was subsequently determined that this incongruence was due to officer shift changes and high numbers of outside calls during the afternoon hours. These factors created a gap in coverage and limited police resources toward the intervention (though the authors note that officer presence was adequate due to Temple University Police presence). It was also revealed that very few students were aware of the safety corridor, though it is not clear whether this was resource related, as school administration reported that the corridor was advertised over school announcements and letters that were distributed to students and parents. Stone (1993) also reported organizational and resource‐related constraints during the Atlanta public housing POP project. There was a relative lack of interest regarding the intervention within the department, little administrative support, and police training was minimal. These issues were compounded by the fact that the city of Atlanta had hosted the Democratic National Convention prior to the intervention, which forced officers to delay vacation during this time. Thus, when the POP project started, many officers opted to take time off and the project was chronically understaffed. In their evaluation of the Lowell (MA) Smart Policing Initiative (SPI, now referred to as Strategies for Police Innovation), Bond and Hajjar (2013) also noted that a common complaint from police captains was a lack of resources. It was suggested that such constraints prevented an increased level of proactivity during the intervention, though these issues appeared to be minor as the intervention was still considered to be effective. Minor issues with internal resistance were also noted by the Knoxville Police Department (2002) during their Public Safety Collaborative.

Police subversion concerns created barriers for both the Jersey City violent places POP study (Braga, 1997; Braga et al., 1999) and the Drug Market Analysis Program (Weisburd & Green, 1995). Partly as a result of officer resistance, the Drug Market Analysis Program achieved limited implementation in the first 9 months, with only nine hot spots receiving all program elements. This forced Weisburd and Green (1995) to increase the length of the intervention and develop a more detailed implementation schedule, and ultimately the program was fully implemented for the last 5 months of the intervention period. Braga (1997) noted similar resistance among officers in the violent places POP project, as well as a disconnect between middle management and department headquarters that threatened the integrity of the program and slowed progress during the first 8 months. Ultimately the intervention unit was placed under new leadership and protocols were established to document instances of subversion. Braga also noted significant organizational changes, such as an influx of retirement and scheduled vacations which strained resources and reduced the sample size of the experiment.

Another frequent implementation barrier was resistance from stakeholders that were intended to be involved in the intervention. In the Glendale SPI study, White and Katz (2013) indicated that in phase I of the response, the SPI team was largely unsuccessful in working with Circle K management to change the physical structure and operating policies of the stores. They note that, at this stage, communication between Circe K representatives and the SPI team suffered, and the intervention was forced in a different direction. Partly as a result of this, Dario (2016) expressed concern over treatment dosage, noting that treatment quality likely varied by store location due, in part, to differing levels of responsiveness. In their attempted intervention with Walmart management, Zidar et al. (2017) also reported resistance toward environmental and policy‐oriented intervention measures. This resistance dictated the future direction of the program, as the initial plan to partner with Walmart seemed futile. In response, Zidar et al. forced responsibility on Walmart leadership by forcing them to handle petit shoplifting incidents without police assistance; however, even after doing so they noted instances of Walmart loss‐prevention misrepresenting case facts to illicit police response.

The San Angelo Police Department (2006) cited heavy opposition from business owners to the implementation of an identification checking program, largely over concern that the program would inconvenience customers. This resistance forced the department to shift responsibility for the intervention toward the customer, though despite this, few businesses ever became willing to implement the program. The Motel Interdiction Team (MIT) program in Reno experienced similar complications (Elliott, 2007; Reno Police Department, 2006). Motel owners were concerned about the economic ramifications that would result from the eviction of criminal tenants. The Reno Police Department (2006) noted that it became difficult to educate these owners about recognition of criminal behavior and eviction processes, ultimately slowing the intervention's progress. In response to uncooperative property owners, the Houston Police Department (2012) increased code enforcement in their targeted intervention of the Antoine corridor. However, this response temporarily led to a backlog in the court system, and prosecutors began dropping charges (though this issue was subsequently resolved by use of specialized prosecutors). Lancashire Constabulary (2008) noted a similar delay in court processing based on their enforcement responses.

Several studies reported resistance from other outside stakeholders such as neighborhood residents, community organizations, and local government. Cooley et al. (2019) described an attempted POP replication in Canton, OH. However, while the initial intervention was successful in establishing partnerships with community residents and neighborhood groups, the attempted replication was not. Cooley and colleagues noted that there was a lack of community organizations available to partner with and that local residents were distrusting of and unwilling to cooperate with police. Local community meetings were unsuccessful at bridging the gap between law enforcement and neighborhood residents, officer morale was low, and ultimately the intervention showed limited effectiveness. Issues were encountered forming resident partnerships in the St. Louis County Hot Spots in Residential Areas experiment as well (Kochel & Weisburd, 2017). Specifically, program evaluation revealed that resident partnerships were less frequently established than had been originally intended. While residents appeared to be cooperative in the early stages of the Brightwood Interdiction project, the same residents were subsequently caught tipping off offenders to police surveillance during the intervention (Nunn et al., 2006). This slowed the evidence gathering process, but police were ultimately able to generate enough evidence to execute the planned warrant sweep. Lastly, as a result of varying levels of difficulty partnering with the community, Tuffin et al. (2006) reported that only two of six intended sites receiving full implementation, though the sites that did achieve full implementation showed strong results.

Gill et al. (2018) noted several aspects of the collaborative problem‐solving intervention in Seattle (WA) that were halted by local government resistance. In one intervention area, officers sought to implement a smoking ban, but were unable to do so largely due to a lack of political support. In another target area, officers were unable to implement environmental changes due to a city redevelopment plan that was operating on a different timeline. Gill et al. also noted that, despite the project's intention to be fully non‐enforcement, officers in one of the intervention areas felt that enforcement measures were necessary to stabilize the area.

Studies that targeted residential burglary incidents reported minor issues with environmental changes to the target area. London Borough of Enfield (2011) referenced issues with the installation of alley gates. Installation required 100% approval from area residents; however, several properties were rentals with out of town owners. The police were subsequently able to adjust the approval rate from 100% to 98% to circumvent this issue. Durham Constabulary (2017) noted similar issues installing security measures in private housing areas; however, they were eventually able to gain permission to do so. Lancashire Constabulary (2008) also documented issues implementing situational responses that altered the physical environment, and the Hollywood Police Department (2015) determined that closing access to alleyways was not financially feasible (however, they promoted the use of see‐through fencing instead).

Finally, there were some problems unique to certain studies. Sherman et al. (1989) encountered issues with hot spot selection, discovering that up to 15% of calls were “mirrors,” or duplicates created as a result of multiple 911 calls for the same incident. Additionally, several addresses that were originally believed to be independent were subsequently determined to correspond to the same building. This led to the inclusion of some buildings in both treatment and control groups, and required a series of pairwise deletions to modify the initial assignments. Bichler et al. (2013) expressed some concern over the possibility of unknown history effects, noting that at least one motel location was disqualified from their study after being the target of another policing intervention. Ultimately, however, there was no reported evidence to suggest that similar issues occurred at other intervention motels.

It is also worth noting that these studies may vary in their level of reporting validity. All interventions are likely to encounter obstacles; however, it is not necessarily the case that all such obstacles are accurately documented. At times there may be incentive to represent an intervention in the best possible light, perhaps at the expense of complete transparency. While this is certainly true of any form of research, it bears reminding that our determinations are limited to what was reported in study publications.

5.4 Risk of bias in included studies

Five main measures from our coding instrument (see Appendix E) were used to assess potential sources of bias in our included studies. These items included: (a) Were any sources of nonequivalence or bias reported or implied in the application of the intervention or its analysis (i.e., threats to internal validity)? (b) If yes, what sources of nonequivalence or bias were identified? (c) Did the researcher(s) express any concerns over the quality of the data? (d) If yes, explain. (e) If a quasi‐experiment, how was matching of groups achieved? The studies that reported issues along these dimensions and/or compared treatment groups to the rest of a jurisdiction or population are presented in Table 4. The remainder of our included studies reported no such issues and/or employed higher quality matching procedures.

Overall, only 14.7% (n = 5) of studies reported internal validity concerns and 5.9% (n = 2) of studies reported concerns over data quality. However, the validity of the matching techniques used in our sample of studies does need to be considered, as 40.0% (10 of 25) of the identified quasi‐experiments used the rest of a jurisdiction or population not receiving treatment as the comparison unit. Additionally, the remaining quasi‐experimental evaluations exclusively matched comparison units based on descriptive and demographic characteristics, or simple statistical tests of such characteristics. None of the included quasi‐experimental evaluations reported propensity or regression‐based matching techniques.

Of the randomized experiments (n = 9) included in this review, very few reported concerns over randomization procedures or other issues related to internal validity and bias. All experimental studies included some form of blocking or pair‐matching technique in addition to randomization. However, specific concerns were noted in a few of these experiments. In the RECAP experiment (Sherman et al., 1989, p. 17), there was possible contamination (or “spillover” effects) as treatment and control addresses were, at times, under shared ownership. Moreover, the instability in the call frequencies of particular addresses created additional variability between treatment and control groups. However, ultimately the groups were reported to be roughly equivalent. Despite the use of matched‐pair techniques in the Collaborative Problem‐Solving at Youth Crime Hot Spots study, the treatment and control locations were unable to be optimally matched, and Gill et al. (2018) noted some concern over the equivalence of these matched pairs.

The risks of bias in the quasi‐experimental evaluations are undoubtedly greater. However, of the 15 studies that matched treatment to comparison locations based on simple analysis of descriptive, social, or demographic characteristics, no major concerns were reported. It should be noted, however, that few of these studies (n = 4) were identified as providing a visual comparison of descriptive statistics between treatment and comparison areas. The remaining studies (n = 11) described the rationale and/or process for selection of comparison units, but did not provide further evidence that equivalence was attained.

The most notable concerns among our included studies were related to the use of particularly nonequivalent comparison groups. There were six studies that compared small geographic treatment areas (or collections of small geographic treatment areas) to city, district, or other population‐wide trends (Boston Police Department, 2008; Guseynov, 2010; Houston Police Department, 2012; Lexington Division of Police, 2009; London Borough of Enfield, 2011; White & Katz, 2013). Of note, four of these studies are Goldstein Award submissions, and while they do not report substantial concern over the comparability of the units, there are clear threats to internal validity caused by the use of such comparisons. There were another five studies that compared treatment units to the remainder of a population not receiving treatment, but where the size discrepancy between the groups was not as large (Baker & Wolfer, 2003; Elliott, 2007; San Angelo Police Department, 2006; Zidar et al., 2017).

In addition to the inherent threat to internal validity, several of the weaker quasi‐experimental studies reported unique issues. Comparison of descriptive statistics for the treatment and control areas in the CSTAR intervention (Guseynov, 2010) indicated statistically significant differences on measures of race, unemployment, poverty, single parent households, and population mobility. Both Guseynov (2010) and Elliott (2007) also reported data‐related concerns. Guseynov specifically notes that the Kansas City Police Department switched reporting systems during the beginning of the postintervention period. This switch resulted in the omission of 17 weeks of data, possibly leading to bias in the analysis. Zidar et al. (2017) suggested that the significant decline in reported shoplifting incidents indicated by their evaluation may have been the result of under‐reporting rather than true changes in the outcome. The intervention involved the implementation of a new reporting system for target Walmarts only. Thus, they imply that it is likely the observed outcome was the result of this measurement change rather than changes in the incident rate. In the Glendale SPI (Dario, 2016; White & Katz, 2013), the intervention locations were selected based on high pre‐existing crime rates. Dario (2016, p. 99) noted the inherent potential of regression to the mean when treatment units are selected in such a way, noting the possibility of bias in treatment selection.

5.5 Meta‐analysis of the effects of POP on crime and disorder

Our first meta‐analytic model presents the overall mean effects for 70 outcomes across the 34 included studies. As noted, above, many studies reported on multiple crime/disorder outcomes and for many the authors did not specify any one outcome as the primary target of their intervention. In our data, 13 of the 34 (38.2%) studies fit into this category. To avoid any “creaming” of results, we include all relevant outcomes from such studies. For studies with multiple outcomes, a mean effect size is used in this model; thus each study is only counted once in the analysis. This is the same approach used in our original review, as well as other recent Campbell reviews of policing strategies (e.g., Braga, Turchan, et al., 2019).

The results from the first model are presented in Figures 2a (Cohen's D model) and 2b (RIRR model). The forest plots show the standardized mean differences and log RIRRs, respectively, between the treatment and control groups, with the lines on either side representing the 95% confidence interval (CI). Effects to the right of 0 are supportive of reductions in crime/disorder, while effects to the left would suggest backfire effects where problems increased in the treatment areas relative to the controls. A random effects model was estimated based on an a priori assumption of a heterogeneous distribution of effect sizes (and the Q statistics for our models confirm this assumption).

The overall mean effect size for Cohen's D approach is 0.183 (p < .001). The largest effects were found in the studies conducted by the London Borough Enfield (0.841), Thomas (0.771), the Knoxville Police Department (0.664) and the San Angelo Police Department (0.654)—all four were submissions for the Goldstein Award. Three studies reported negative overall effects, Stone (−0.001), Taylor et al. (−0.012), and Stokes et al. (−0.203).

The overall mean effect is considered a small effect by conventional standards developed by Cohen (1988). However, Lipsey (1990) describes effects in this range as small but meaningful impacts that could “easily be of practical significance” (Lipsey, 1990, p. 109). It is also important to note here that the studies clustered immediately around the mean effect size are randomized experimental evaluations of place‐based versions of POP in which the authors reported notable reductions in a variety of crime and disorder outcomes. (e.g., Weisburd & Green, 1995; ES = 0.147, Sherman et al., 1989; ES = 0.192; Braga & Bond, 2008; ES = 0.206; Braga et al., 1999; ES = 0.233). In this regard, Wilson (in progress) has raised strong concerns in interpreting place‐based effect sizes similarly to person‐based effect sizes. While we follow standard practice here in reporting effect sizes, we think that caution should be used in interpretation of what magnitudes mean.

For example, the largest impact in the study by Nunn et al. (2006) was an effect size of 0.200 for robbery calls for service (see Figure 3). This is an effect at the criterion of 0.20 set by Cohen (1988) for a small effect. Looking at the raw changes in the data, we see that the relative actual reduction in the proportion of crime in the treatment condition was 30.4% percent, while the control area saw no change in robbery calls. Similarly, in Gill et al.'s (2018) study the largest impact was on calls for service in the retail treatment site. Our effect size for this outcome of 0.187 is a bit below the 0.20 threshold, yet the raw change shows a 10.3% decrease in calls compared with a 25.9% increase in the comparison areas. Our point is that standard small effect sizes may translate to very meaningful crime prevention outcomes at places.

As noted above, David Wilson (in progress) has argued that Cohen's D fails to produce effect sizes that are comparable across studies when based on place‐based count data and has also shown that the process of converting RIRRs to Cohen's D is problematic. As such, we also present meta‐analyses for all of our models using the Log RIRR as the effect sizes. These analyses are of 33 of 34 included studies (and 69/70 outcomes). The study by Stone (1993) is not included in the RIRR models as the data and methods used did not allow us to calculate an RIRR. This is not a major concern as this study is a near zero effect in Cohen's D model (D = −0.001, p = .992). The overall RIRR model is shown in Figure 2b.

The results of the RIRR show an overall effect of 0.291. This can be interpreted as a relative present change by taking the exponent of the effect (which is a log RIRR) and then subtracting 1 from that value and multiplying by 100. For the overall model this shows that there was a 33.8% reduction in crime/disorder in the POP treatment areas/groups relative to the controls. Some caution is needed in interpreting this effect size as analyses presented below show smaller (though still positive and statistically significant) effects in randomized experiments and after accounting for publication bias. Nonetheless, this finding supports our illustration above of how Cohen's D often understates the magnitude of effects for place‐based studies (the majority of our sample) and provides further evidence that Wilson (in progress) is correct that Cohen's D is not the most appropriate effect size for these types of studies. We continue to present Cohen's D models throughout to allow for comparison to our prior review.

Regarding the meaning of effect sizes, we think it important to also note here that scholars have argued that approaches like POP that alter the characteristics of high‐crime places may reduce more crime in the long run than approaches such as temporarily increased police presence or crackdowns (Braga, Turchan, et al., 2019; Braga & Weisburd, 2010). This is the case as lasting changes to places made through identifying and solving problems may reduce crime over the long term through reducing opportunities for crime or other mechanisms (see above). Thus finding a relative reduction of 33.8% may be suggestive of even larger impacts on crime/disorder in targeted areas in the long run.

Moving beyond the overall effect size, perhaps the most striking finding is that the overall trend of mean effect sizes per study skews very heavily toward studies that produced findings in the direction of POP being effective. Specifically, 31 out of 34 studies (91.2%) in Cohen's D model have positive effect sizes, with 13 of them (38.2%) being statistically significant effects in favor of the treatment group. In the RIRR model, 30/33 studies (90.9%) reported positive impacts and 12 (36.4%) were statistically significant. Only one study (Stokes et al., 1996) had a statistically significant backfire effect, and that was a project that was plagued by implementation and research design limitations as reviewed in Section 5.3. Excluding this study from our overall model slightly increases the mean effect size from 0.183 to 0.195 in Cohen's D model and increased the relative reduction in crime from 33.8% to 36.6% in the RIRR model.

Before moving on, it is also important to recall that we had 15 eligible studies that are not included in any of our analyses as they lacked data needed to calculate effect sizes. Goldstein and Tilley Award submissions accounted for 13 of these 15 studies. As one would expect for programs submitted for award consideration, all 13 of these studies discuss findings favorable to POP's effectiveness. The other two studies (Carson & Wellman, 2018; Wolfe et al., 2015) failed to find evidence of positive impacts, but also did not note any backfire effects and pointed to challenges with fully implementing the interventions. As such, there is no reason to believe that the absence of these studies from our meta‐analyses would alter any of our conclusions.

Given that the overall models present mean effect sizes for studies reporting on multiple outcomes without a clear primary outcome specified, we felt it important to also estimate models including only the largest and smallest effect sizes for each study. For studies with a single outcome, or a clearly stated primary outcome, their effect sizes remain the same in all models. This approach provides an upper and lower bounds for the overall standardized mean effect. Figure 3a,b presents the largest effects models and Figures 4a,b the smallest effects models.

Following the logic of inclusion, the overall random effect is larger when only including the outcome with the largest effect size for each study. While the mean effects Cohen's D model had an overall effect of 0.183, the largest effects model has an overall standardized mean effect of 0.271 (p < .001)—an increase of 0.088 (48.1%). Similarly, for the RIRR model the mean effects approach had an overall effect of 0.291 (33.7% relative reduction), the largest effects model had an overall effect of 0.357. This corresponds to a 42.9% relative reduction of crime/disorder in the treatment group.

Turning to the smallest effects, we see a lower bound for the standardized mean effect of 0.135 (p < .001)—a decrease of 0.048 (−26.2%) from the overall mean effect model. For the RIRR model we see a lower bound of the overall effect of 0.223, corresponding to a 25.0% relative reduction in crime/disorder. We think that this approach gives a sense of the range of effects that can be expected in POP programs. Specifically, using the RIRR results, the overall effect ranges from a 25% relative reduction when using the smallest effects to 42.9% relative reduction when using the largest effects. Importantly, our overall conclusion about the effectiveness of POP remains consistent. Regardless of the type of effect size and whether we examine the overall mean effect or look at the largest or smallest effect size, the results suggest that POP has a significant meaningful effect (Lipsey, 1990) in reducing crime/disorder.

Finally, we felt it important to examine variation in effect sizes across crime types. Table 5 summarizes the mean effects sizes for violent crime, property crime and disorder offenses for both our Cohen's D model and the RIRR approach. Studies that reported on aggregated crime counts that included more than one of these categories are not included in these analyses, nor are other types of outcomes that did not fit into those groupings (and lacked enough cases to perform a meaningful meta‐analysis).

The results show that while POP had significant impacts on property crime (31.0% relative reduction) and disorder offenses (18.9% relative reduction), the overall effect for violent crime did not reach statistical significance (p = .156 in the RIRR model, p = .218 in Cohen's D model). However, the effect is still in the positive direction (9.5% relative reduction) and 13 of the 18 violent crime outcomes were positive. Future research should further explore the potential reasons for the heterogeneous impact of POP across crime types.

5.5.1 Moderator analyses

We also conducted moderator analyses to examine heterogeneity in effect sizes across three dimensions—(a) experiments versus quasi‐experiments, (b) studies with nonequivalent groups versus all others, and, (c) the type of publication (scholarly publications vs. Goldstein Award submissions).

Study design is important to assess as it is well known that more rigorous designs are more likely to produce null findings (Rossi, 1987). Figure 5a,b shows the moderator results for randomized experiments versus quasi‐experiments.

The moderator analysis of the impact of study design shows that quasi‐experiments have a larger overall effect size than randomized experiments. For Cohen's D model (see Figure 5a) the quasi‐experimental studies have an effect size of 0.212 (p < .001), while the randomized experiments have an effect size of 0.107 (p < .001). The difference between groups was statistically significant (Q = 4.914, df = 1, p = .027) and the moderated effect size is 0.147 (p < .001). Turning to the RIRR models (see Figure 5b) we see the same pattern. For quasi‐experiments, the effect of 0.377 (a relative reduction of 45.8%) was larger than the effect of 0.089 (a relative reduction of 9.3%) for randomized experiments. The difference between groups was again statistically significant (Q = 14.171, df = 1, p < .001) and the moderated effect size was 0.183—a relative reduction of 20.1%.

These results are consistent with Weisburd, Lum, and Yang's (2003) proposal that experimental designs more generally show smaller impacts in crime and justice research (see also, Welsh, 2016). The results show that while there is a bias toward finding stronger effects in studies with weaker research designs, the overall finding of a significant meaningful effect for POP is supported across study types, as well as by the moderated effect size.

Our second methodological moderator analysis examined the impact of the nonequivalent research designs highlighted in Table 4. Specifically, we compared the studies that are listed in Table 4 as having nonequivalent control groups to studies with better matching methods. The results here suggest that studies with nonequivalent control groups did not bias our conclusions. Indeed, effect sizes were actually slightly larger in the studies with better matching methods. For Cohen's D models, the 11 studies with nonequivalent control groups had an overall effect size of 0.178 (p < .001), while the effect for the 23 studies with more rigorous matching approaches was 0.190 (p < .001). The difference between groups was not statistically significant (Q = 0.034, df = 1, p = 0.854) and the moderated effect size was 0.184 (p < .001). Similarly, for the RIRR models, the nonequivalent control groups had an overall effect size of 0.263 (p < .001; a 30.1% relative reduction), while the studies which used better matching methods had an effect of 0.309 (p < .001; a relative reduction of 36.2%). The between‐groups difference was again not statistically significant (Q = 0.231, df = 1, p = .631) and the moderated effect size was 0.289 (p < .001; a 33.5% relative reduction).

Next, we examined the impact of the type of publication. This is important as the award submissions are inherently biased toward successful outcomes and likely also toward larger effects. The reasoning here is simple—police departments are not going to submit a program that did not work for consideration for an award and are probably most likely to submit when a project has a larger impact. As such, our moderator analysis here compares the mean effects for the award submissions to those for scholarly publications (journal articles, research reports, theses and dissertations in our current sample). These results are shown in Figure 6a,b.

As expected, the award submissions have a larger overall mean effect size than scholarly publications. Examining Cohen's D model (Figure 6a) we see that the award submissions had an overall effect of 0.362 (p < .001) while the effect for scholarly publications was 0.101 (p = .001). The difference between groups was statistically significant (Q = 13.702, df = 1, p < .001) and the moderated effect size is 0.149 (p < .001). The results from the RIRR model (Figure 6b) are very similar, with the effect for award submissions being 0.580 (a 78.6% relative reduction) compared with an effect of .166 (an 18.1% relative reduction) for the scholarly publications. The between‐groups difference was again statistically significant (Q = 12.329, df = 1, p < .001) and the moderated effect size was 0.228 (a 25.6% relative reduction).

These results raise possible concerns regarding the exclusion of police‐initiated POP programs that were evaluated in some way but were not submitted for award nominations. We conduct analyses regarding publication bias below, and note the biases there. Nonetheless, because our analyses without the award submissions remain statistically significant, this finding does not alter our overall conclusion of a significant crime prevention outcome for POP. In turn, these award studies do provide additional information about successful interventions and a sense of the upper range of POP impacts that are to be expected.

As a final summary and sensitivity analysis, Tables 6a and 6b below summarize the main effect sizes outlined above, and also present results from models using the smallest and largest effects in each applicable study for randomized experiments, quasi‐experiments, award submissions and scholarly publications that were not presented above to save space. The range of effects support our conclusion of a meaningful effect of POP in reducing crime in all but two of the models. The overall effect is not significant in the two most conservative models—smallest effect outcomes for randomized experiments and scholarly publications—for both Cohen's D and RIRR models. All the other models produced statistically significant overall effects. For Cohen's D models (see Table 6a), those effects ranged from 0.101 (mean effect for randomized experiments) to 0.415 (largest effect outcomes for award submissions). The mean effects shown in the “Combined ES” column provide the overall summary of effects across all outcomes for each model and show that across all models the average overall effect of POP on crime/disorder ranges from 0.101 to 0.362. Similarly, looking at the RIRR models summarized in Table 6b (again excluding the smallest effects for the randomized experiments and scholarly publications which are not significant) shows that crime in the POP group relative to the controls was reduced between 9.3% (mean effect for randomized experiments) and 81.5% (largest effect for award submissions). The mean effect ranged from the 9.3% relative reduction in randomized experiments to 78.6% for award submissions.

As such, our review provides strong and consistent evidence that POP is an effective approach to reducing crime and disorder. However, there is a great deal of heterogeneity in the magnitude of effect sizes across factors such as study type, study rigor, and crime type. This will be discussed more in the discussion and conclusion sections of this report.

5.5.2 Meta‐analysis of displacement and diffusion effects

Many of our studies were place‐based approaches to POP that may have had potential to either displace crime/disorder to surrounding areas or to see the benefits of the intervention diffuse to areas that were not targeted (see Weisburd et al., 2006). Eight of our studies provided data for a total of 19 outcomes that allowed us to create effect sizes and conduct a meta‐analysis of displacement and diffusion effects.

These effect sizes were calculated using pre‐ and postintervention counts for target, control, and buffer areas. This was done following the approach used by Telep, Weisburd, Gill, Vitter, and Teichman (2014) in their meta‐analysis of displacement and diffusion effects of interventions in large‐scale geographic areas (see also Bowers, Johnson, Guerette, Summers, & Poynton, 2011; Braga, Turchan, et al., 2019). Effect sizes were calculated using the relative incidence risk ratio approach described above. Pre‐ and postintervention counts from the treatment buffer area(s) are compared with pre‐ and postintervention counts either from a control buffer area or to the control area itself in studies that did not have a catchment area for the control site. Figure 7 shows the mean overall effect for displacement/diffusion. Effects to the right of zero indicate evidence of diffusion of crime control benefits, while effects to the left suggest displacement effects.

Looking at Cohen's D model (see Figure 7a), the overall model provides no evidence of displacement. Seven of the eight studies have positive effect sizes, while the sole negative effect for Taylor et al. (2011) was very small and not statistically significant (−0.050, p = .765). Moreover, the overall random effect for the model of 0.089 (p = .023) is suggestive of diffusion of benefits across these eight studies, though caution is needed given the small effect size and limited number of studies. We also estimated displacement/diffusion models using the largest and smallest effects to provide a range for our overall estimate. This showed that when using the smallest effects, there is no evidence of either displacement or diffusion (0.029, p = .345), while when only including largest effects the evidence in favor of diffusion of benefits is moderately stronger than in the mean effects model (0.154, p = .006). The results from the RIRR model lead to the same conclusions. The overall model (see Figure 7b) shows a relative reduction of 17.5% (p = .005) in favor of diffusion effects and the largest effects model shows a larger relative decline of 34.6% (p = .004). The smallest effects model shows no statistical evidence of displacement or diffusion, with a nonsignificant relative difference of 4.6% (p = .327) in favor of diffusion.

5.5.3 Narrative review of impacts on noncrime/disorder outcomes

While the primary question of our review is concerned with crime and disorder outcomes of POP, we also collected data, when available, on the cost effectiveness of POP, as well as secondary outcomes of POP programs, including impacts on police legitimacy, fear of crime, and collective efficacy. Because only a small number of included studies focused on each of these outcomes and inconsistency in measures across studies, we felt a narrative review of findings would be more useful for synthesis than a meta‐analysis. Additionally, because our included studies generally prioritized crime control outcomes, the information provided on these secondary outcomes is not always sufficient for calculating effect sizes. We are also more cautious in interpreting these findings, since we did not search for, and excluded any studies we did find, that focused exclusively on fear of crime or other noncrime outcomes. A future study should systematically search for studies of POP focused on impacting outcomes other than crime and disorder.

We include summary information on each outcome below and an examination of findings by study in Table 7. Our findings overall suggest POP can be cost‐effective, but tends to have limited impacts on police legitimacy, fear of crime, and collective efficacy, although those outcomes are often not assessed in our included studies.13

Financial cost‐benefit analysis

Eight studies assessed cost or hours savings as a result of the POP project. These were generally based on cost estimates for how much time would have been spent on calls for service or incidents that were prevented by the POP project. In most cases, estimates were just based on police time and cost, but two studies (Bichler et al. 2013; London Borough of Enfield, 2011) also included estimates for time saved by other agencies. In all cases, the POP project was associated with a substantial cost savings. We recognize though that POP projects without significant impacts on crime would be less likely to include a cost‐benefit analysis.

Two motel‐based studies in the U.S. looked just at hours saved. Bichler et al. (2013) examined just savings in hours, finding the motel ordinance program saved 1,253.4 hours per year in patrol time on calls for service, more than a 51% reduction. Time spent by other city agencies on motel‐related issues also dropped 92.4 hours per year, on average. The Reno Police Department (2006) did not provide precise estimates for their entire project, but noted that impacts had saved the department approximately 1,750 officer hours per year.

Two other U.S. studies estimated cost savings based on calls prevented in retail locations. White and Katz (2013) estimated the cost to respond to calls at targeted convenience stores dropped from $43,685 to $25,403. They argue these are conservative estimates, since they do not account for other criminal justice system and business costs. Zidar et al. (2017) found a program to reduce responses for low‐level theft led to $26,884 less in police department manpower costs. The department saved about 35 hours per month by responding to fewer calls at Walmart.

Four studies by U.K. agencies also estimated savings, using Home Office estimates on the costs of crime. Durham Constabulary (2017) found that burglaries prevented equated to a savings of £3,640 just in police‐related costs, with even higher estimates when accounting for all system and victim costs. Lancashire Constabulary (2008) estimated cost savings across multiple crime categories as a result of their POP project. Burglary savings were estimated at $62,100, criminal damage savings were $72,420, and antisocial behavior incident savings totaled $51,770. A second project by Lancashire Constabulary (2012) found a significant reduction in arrests were associated with a total cost savings of £82,000. The London Borough of Enfield (2011) estimated project cost savings at £934,000, accounting for both police and social costs of crimes prevented.

Police legitimacy/satisfaction

Six studies used measures of resident perceptions of police procedural justice and/or legitimacy to assess impacts of POP on trust in the police. Results here were not entirely consistent, but generally suggest POP has limited impact on police legitimacy. There is no evidence here, however, that problem‐oriented approaches, even when applied in hot spots, damage police trust.

Stone (1993) saw no change over time or across treatment and control public housing sites in Atlanta in whether residents were satisfied with police and thought police treated them with respect. Braga and Bond (2008) found no differences in perceptions of police in pre and post interviews with respondents who were likely to have had contact with police during the Lowell hot spots experiment. Cooley et al. (2019) similarly found no change over time or between treatment and control neighborhoods in perceptions of whether police are doing a good job. Results were similar in the two most rigorous assessments. Using a mail survey, Ratcliffe et al. (2015) found no differences in perceptions of procedural justice or satisfaction with police among residents of control spots versus those receiving POP in Philadelphia. Using an in‐person resident survey, Kochel and Weisburd (2017) found procedural justice perceptions improved over time in problem solving hot spots, but no more than they did in control hot spots. There was a small nonsignificant decline in problem solving hot spot resident perceptions of legitimacy in the short‐term follow‐up, but legitimacy had rebounded to similar levels to control respondents by the long‐term follow‐up. Tuffin et al. (2006) found the only evidence of enhanced perceptions of police, although we note that this was an evaluation of reassurance policing, so building trust was a major component along with problem solving. Here, treatment site residents relative to control site residents saw 15% net improvements in confidence in police and 8% net increases in feeling that police are willing to listen and respond.

Fear of crime

Eight studies assessed changes in resident fear of crime as a result of a POP project. Findings here were not entirely consistent, but most studies found no impact of POP on resident fear of crime, and for studies that did find an impact, effects were generally small. Stone's (1993) public housing surveys, Braga and Bond's (2009) resident interviews, Ratcliffe et al. (2015) mail surveys, and Cooley et al's. (2019) resident surveys, for example, suggested no pre–post change in fear in treatment relative to control sites. Tuffin et al. (2006) saw limited impacts of reassurance policing on fear of crime. There was some net improvement in feelings of safety after dark in treatment compared with control sites, but for particular crime types, there were no consistent effects. Stokes et al. (1996) saw, if anything, negative impacts of the safe route to school program on student fear of being attacked, not surprisingly given the overall backfire effects of the intervention. Similarly, Kochel et al. (2015) found increases in victimization risk and decreases in feelings of personal safety among residents of problem solving hot spots relative to controls in the short‐term, but there were no differences across groups by the second postintervention survey. Baker and Wolfer (2003) found more substantial impacts of the intervention on fear of crime among residents living near the target site, particularly for feelings of safety in the park during the day; however, even here the findings were mixed. Control group respondents were more likely than treatment group respondents to say they felt safe due to crime prevention efforts in the postsurvey, even though only treatment group residents had received a crime prevention intervention.

Collective efficacy

Three studies included pre‐ and postintervention measures of resident perceptions of collective efficacy. Findings across the three studies were inconsistent and mixed. None of the studies showed large impacts of POP on collective efficacy. Stone's (1993) survey showed no difference in perceptions of informal social control over time or in treatment versus control housing complexes. Tuffin et al. (2006) also found no difference in collective efficacy perceptions among residents of reassurance policing areas relative to control sites. There were also no significant changes in perceptions of social cohesion, but treatment sites did show improvements relative to comparison sites in the percentage of residents saying they trust many or some people in their area. Kochel and Weisburd (2019) found no overall change in resident perceptions of social cohesion or overall collective efficacy in problem‐solving hot spots relative to controls. They did find some long‐term improvements in perceptions of informal social control among POP hot spot residents, with about a 7% improvement compared with baseline levels. Kochel and Weisburd (2019) suggested the limited community involvement in most implemented problem‐solving projects may explain the modest impacts.

5.5.4 Publication bias

Publication bias presents a strong challenge to any review of evaluation studies (Rothstein, 2008). Campbell reviews, such as ours, take a number of steps to reduce publication bias, as represented by the fact that 21 of the 34 (61.8%) included studies in our main analyses came from unpublished sources (13 Goldstein Award Submissions, 4 research reports, 2 doctoral dissertations, and 2 Master's theses). Wilson has argued that there is often little difference in methodological quality between published and unpublished studies, suggesting the importance of searching the “gray literature” (Wilson, 2009). For our review, there may also be a bias in unpublished studies that are nevertheless available for review, since 13 studies were identified through the Goldstein Award competition. As noted earlier, award submissions are inherently biased toward successful programs. This was evidenced by our moderator analyses, which showed effect sizes were significantly larger for award submissions versus the other publication types.

Here we focus on an overall comparison of the 13 studies published in scholarly journals versus the other 21 studies (20 for the RIRR approach) through use of moderator analysis. For Cohen's D model, the mean overall effect size for studies published in scholarly journals is 0.156 (p = .002; Q = 46.482, df = 12, p < .001) and for unpublished studies the average effect is 0.199 (p < .001; Q = 111.367, df = 20, p < .001). Moreover, the moderated effect size is 0.184 (p < .001) and the between models heterogeneity test was not statistically significant (Q = 0.468, df = 1, p = .494). Similar findings are seen in the RIRR model. The mean effect for the scholarly journal studies shows a relative decline of 29.7% (p = .005; Q = 63.518, df = 12, p < .001) while the unpublished studies show a relative decline of 35.1% (p < .001; Q = 141.113, df = 19, p < .001). The moderated effect size shows a relative decline of 33.8% and the between models differences test was again not statistically significant (Q = 0.145, df = 1, p = .703). The lack of significance for the between‐model tests, along with the similarity between the mean overall effect sizes, suggests that publication bias may not have major impact on the outcomes of this review.

To more formally assess publication bias we generated a funnel plot to examine for possible selection bias in our results. This is shown in Figure 8a,b. A visual inspection indicates some asymmetry with more studies with a large effect and a large standard error to the right of the mean than the left of the mean. We used the trim‐and‐fill procedure developed by Duval and Tweedie (2000) to examine how our estimates would change in the absence of this asymmetry. The trim‐and‐fill procedure determined that nine studies should be added to create symmetry.

For Cohen's D model, these additional nine imputed studies slightly reduced the mean effect size from 0.183 (95% CI = 0.0124–0.241) to 0.106 (95% CI = 0.043–0.170). For the RIRR approach, 11 studies were imputed and this reduced the overall effect from 0.291 (95% CI = 0.202–0.379) to 0.132 (95% CI = 0.040–0.223). Put into relative risk reduction terms, the imputed studies decreased the overall relative reduction from 33.8% to 14.1%.

Along with the moderator analysis above, this suggests that while there is some potential for publication bias in our sample, especially given the nature of including police award submissions, it does not alter our overall conclusion of POP having an overall significant, meaningful impact on crime and disorder.

6 DISCUSSION

6.1 Summary of main results

The results of this updated review provide strong evidence that POP is effective in reducing crime. Across a large array of analyses, we find statistically significant impacts of POP. Overall we find that there was a 33.8% reduction in crime/disorder in the POP treatment areas/groups relative to the controls. At the same time, the effect sizes we observe are strongly heterogeneous and the overall effect is likely overstated as smaller effects were found when looking only at the randomized experimental evaluations and after accounting for publication bias. Nonetheless, the findings of those models still show that POP is associated with meaningful and statistically significant declines in crime/disorder in treatment groups relative to controls. Such heterogeneity across models is very common in meta‐analyses in criminology (Lösel, 2018), but points to the importance of going beyond what works in POP to what are the most effective strategies for specific problems.

Overall, our findings show that police following the tenets of the SARA model to identify specific problems, conduct analyses to examine underlying causes, and develop and deliver tailor‐made responses is an evidence‐based approach to crime prevention. This is especially true for property crime and disorder offenses based on our results. POP is also an approach that fits well with hot spots policing, another tactic that has been found effective in reducing crime through Campbell reviews (Braga, Turchan, et al., 2019). Some of our studies overlap with those in that review as they involve applying problem‐solving tactics at small hot spots of crime/disorder, and the authors of the hot spots review note that their strongest effects were associated with programs that involved problem‐solving efforts rather than just increased police presence/activity. The positive findings of both reviews suggest that combining the two approaches is likely a fruitful endeavor for crime‐control efforts.

As a number studies involved place‐based POP programs, it was important to also examine the potential for spatial displacement and diffusion of crime control benefits. Eight of our 26 place‐based studies reported data that allowed us to calculate effect sizes for displacement/diffusion. There is no evidence of spatial displacement of crime/disorder in these studies. Indeed, there is evidence of a small diffusion effect when looking at the mean effect across outcomes. This finding, along with a similar finding of a small diffusion effect in the hot spots policing review (Braga, Turchan, et al., 2019), suggests that place‐based policing efforts do not simply cause crime to “move around the corner” (see Weisburd et al., 2006).

There was some evidence that research design moderated the magnitude of the impact of POP on crime/disorder. The effect size for quasi‐experiments was somewhat larger than that for randomized experiments. However, the mean overall effect is significant for both models (as well as the moderated effect size). The same was true when examining the award submissions, which are inherently biased toward success and larger effects, as those had larger effects than the other types of studies. Nonetheless the overall effect for the non‐award submission studies was statistically significant. Adding to this the fact that nearly all of the studies were weighted to the prevention side of the distribution across analyses, we have additional confidence in our overall conclusion.

Finally, as noted above, if underlying causes at problem places are successfully addressed, the crime‐reduction benefits at targeted locations may be longer lasting and more meaningful in terms of overall crimes prevented compared with a similar effect size for a temporary police crackdown on an area or an intervention delivered to individual offenders. In plain language, there may be more “bang for our buck” when lasting changes are made at places, versus crackdowns that see deterrent effects erode when the program ends or person‐based programs that have to be continually delivered to different individuals over time. Unfortunately, existing studies do not examine crime prevention benefits in the long run, and are generally limited to follow‐up within a year or less (see also Weisburd & Majmundar, 2018 for a similar critique of short‐term follow‐up periods for POP studies). Future evaluations should include longer follow‐up periods so that later updates to this review can quantitatively assess the potential lasting impacts of POP.

6.2 Overall completeness and applicability of evidence

The findings of this review have widespread applicability to policing and crime prevention given the consistency of the conclusions drawn across all of our models. This review also represents a large increase in the available body of evidence in the time since the original review, which included only 10 studies (4 randomized experiments and 6 quasi‐experiments) and 16 outcomes. The current review includes an additional 24 studies (5 randomized experiments and 19 quasi‐experiments) published through December 2018. Our overall model now provides a meta‐analysis on the impact of POP on a total of 70 crime and disorder outcomes across 34 studies. The inclusion of these additional studies reaffirms the conclusion of the original review about POP's effectiveness in reducing crime and disorder with support from a much larger number of tests. While most studies (82.4%) were conducted in the United States, the fact that 5 were conducted in the United Kingdom and 1 in Canada offer initial support that POP is applicable in different contexts. However, this is still limited and caution may be needed when trying to generalize these findings to contexts outside of North America and the United Kingdom. We also were unable to perform a meaningful meta‐analysis on noncrime outcomes. With the current data we could only provide a preliminary, narrative summary of study findings due to the small number of studies that report on the same noncrime outcomes captured through similar measures. The same was true for cost‐benefit analysis.

6.3 Quality of evidence

The overall quality of evidence has improved from the original review with the inclusion of 5 new randomized experiments and 19 quasi‐experiments. However, as we discussed above and assessed with our moderator analyses, POP remains an area that needs more rigorous evaluations. The majority of studies are still quasi‐experiments, and several are weaker designs with unmatched control groups, comparisons of a target area to the rest of a jurisdiction and so forth. While we have confidence in our conclusions as the main finding of a significant effect for POP holds when looking only at the most rigorous studies, caution is needed in individually interpreting the larger effects from the weaker studies—especially the award submissions which are inherently biased toward positive outcomes.

We note that while more randomized experiments, and quasi‐experiments with matched control groups/areas, would improve the quality of evidence on POP, the existing body of evidence is largely a function of the nature of the POP model. The approach calls for identifying specific problems to be researched and addressed and it is often not going to be possible to create a well‐matched comparison area in a study jurisdiction (Eck, 2006a, 2006b). Similarly, the POP model calls on police to identify, analyze and respond to problems, and to then assess whether their efforts are successful. In this sense, the award submissions are evidence of the SARA model in action and are important to include in reviews such as this.

Moreover, the fact that 11 additional award submissions were eligible for this updated review (only two were included in the meta‐analysis in the original review) is indicative of an increase in rigor as more agencies are using control groups, even if unmatched, instead of simple pre–post case study designs. As such, these studies are important evidence and simply require caution in interpreting individual effect sizes and acknowledgment that there is a “file drawer” problem here as agencies are not going to submit (or even write up research reports) for programs that failed. On that front, we retain confidence in our findings as our analyses above suggest that publication bias was not a major concern in our study.

6.4 Limitations and potential biases in the review process

In general, there are no specific limitations or biases in the review process used in this study beyond those inherent to the systematic review process. Namely, all reviews will have a “file drawer” problem to some extent, and the threat may be slightly higher for POP than other approaches due to the assessment step of the SARA model asking police to test whether their efforts were effective. As discussed above, these findings are unlikely to be written up (much less submitted for award consideration) when efforts fail. Moreover, the use of the Global Policing Database is a huge asset to the current review. The traditional and gray literature sources searched to compile that database are far more exhaustive than those used in individual reviews (see Appendices B and C), including the original version of this review. Lastly, there were 15 (13 of which were award submissions) potentially eligible studies of POP that we could not include as we could not calculate standardized effect sizes due to insufficient or inadequate information being presented (see Appendix F). As noted above, we do not feel the absence of these studies biased our conclusions as the 13 award submissions discussed positive impacts of POP and the other two studies reported null effects, but no backfire effects.

6.5 Agreements and disagreements with other studies or reviews

The findings of this review reaffirm those of the earlier review (Weisburd et al., 2008, 2010) and existing narrative reviews that conclude that current evidence suggests that POP is one of the most promising police approaches to preventing crime (Skogan & Frydl, 2004; Weisburd and Majmundar, 2018). Moreover, the limited evidence on displacement and diffusion confirms the findings of other studies of place‐based crime prevention efforts by showing evidence of diffusion of benefits (Bowers et al., 2011; Braga, Turchan, et al., 2019; Weisburd & Majmundar, 2018; Weisburd et al., 2006). This finding is contrary to arguments made by others that displacement is the likely outcome of place‐based interventions (Blattman, Green, Ortega, & Tobón, 2017; Reppetto, 1976).

7 AUTHORS’ CONCLUSIONS

7.1 Implications for practice and policy

Evidence from this review suggests that POP is an effective crime prevention approach. While there is a great deal of heterogeneity in effect sizes across studies and outcomes, 31 out of 34 studies (91.2%) have effect sizes in favor of a treatment effect. Moreover, the overall effect is positive and significant in all of our mean effect size models. In short, the findings suggest that POP is a promising approach to reducing a variety of types of crime and disorder in a variety of contexts (the 34 included studies included 29 unique jurisdictions in 3 countries).

Our findings suggest that it is important for police departments to be fully behind POP efforts if they are to succeed. For instance, the lone backfire effect in the study (Stokes et al., 1996) involved a program that was barely implemented as two‐thirds of students at the target school were unaware of the existence of the school safety corridor and the corridor was poorly staffed in the after school hours due to the timing of police shift changes and limited police resources. Similarly, a null effect was reported in the study by Stone (1993), who noted that the police department did not seem entirely interested in implementing POP and that study officers did not view problem solving as “real” police work. This and other factors led to the program being chronically understaffed.

However, our results also highlight the fact that POP efforts can be successful even if the SARA approach is not delivered in its ideal version. This is important as studies have noted that it is difficult to fully implement the ideal model (Braga, Turchan, et al., 2019; Weisburd & Braga, 2006; Cordner & Biebel, 2005; Eck, 2006b; Maguire et al., 2015). Our findings show that even though the SARA model is often loosely followed, with the problem analysis often being shallow rather than in‐depth (see Table 2b), the approach is still found to lead to crime reductions when compared with control areas that received standard police services. This adds support to arguments that even “shallow” problem‐solving efforts can be lead to significant reductions in crime (Braga & Weisburd, 2010).

POP can also be fruitfully combined with other police tactics that have been found effective in recent Campbell reviews. In particularly, the POP approach has been shown to be effective when combined with hot spots policing. Braga, Weisburd, et al. (2019) found larger effect sizes for POP approaches at hot spots than approaches which simply increased police presence/activity at target areas. Elements of POP often also underlie the focused deterrence approach, which has been found effective and could perhaps be enhanced with more in‐depth problem solving in future programs rather than largely replicating the existing “pulling levers” model (Braga, Weisburd, et al., 2019).

There is also potential for combining POP with the popular Compstat model. Indeed, “innovative problem solving” is one of the key elements in the ideal form of Compstat. While evaluations of Compstat in the United States suggest that the emphasis on holding commanders accountable through Compstat meetings has limited innovative problem solving in the field (Weisburd, Willis, Mastrofski, & Greenspan, 2019), a recent Israeli program suggests that innovative problem solving can be encouraged in a Compstat‐like reform (see Weisburd et al., Unpublished manuscript). In that study, evidence‐based policing practices were strongly encouraged in the context of a national POP reform program, and the message of the program was communicated more directly to the rank and file. Robust prevention benefits were identified in moderate and large size police agencies in quasi‐experimental analyses of property crimes.

7.2 Implications for research

Our study identified 70 tests of POP in 34 included studies. Our meta‐analyses suggest that overall there is a significant effect of the approach in reducing crime and disorder. Our moderator analyses showed that effects are larger for the quasi‐experiments compared with the randomized experiments. Nonetheless, this does highlight the need for more rigorous evaluations of POP in order for a future update of this review to provide a more robust estimate of overall effect size.

This is not in any way meant to discourage quasi‐experimental evaluations of POP, or even pre–post case studies. As discussed earlier, the nature of the POP model means there may sometimes only be a single area with the problem being treated, and even if there are multiple locations it will often be difficult, and sometimes impossible, to identify suitable control areas—much less to identify enough sites to create a randomized experiment with sufficient statistical power. As such, knowledge from less rigorous research remains important. That said, given the similarity of our current findings to those of our original review, we view it as unlikely that future updates will shed new light on our knowledge of POP's effectiveness in the absence of more rigorous evaluations.

In our view, the question at this point is not whether POP works. The vast majority of the studies we reviewed show prevention benefits. This conclusion is also supported by past narrative reviews (Skogan & Frydl, 2004; Weisburd & Majmundar, 2018) and the summary of simple pre–post case studies provided in our original review (Weisburd et al., 2008, 2010). The key remaining questions surround what characteristics are associated with larger impacts of POP on crime. To truly assess this, future evaluations need to not only be more rigorous, but also must capture and report more data about the problems targeted, the level of problem analysis applied, the specific responses actually delivered and report outcomes more often by crime type than aggregate categories. The current sample of studies have a combination of lack of detailed information on many of these factors and tremendous heterogeneity in what is reported, which makes it difficult to draw strong conclusions about what makes POP most effective.

But irrespective of reporting practices, to build an evidence base that will be useful for practitioners in the field it is important for there to be a robust evidence base that is related to specific problems and specific interventions. This is a limitation more generally in the crime and justice field, but is particularly important to address when we have strong evidence of crime prevention effectiveness of a strategy, as we do here. Practitioners want to know what works in what situations, and which practices are most cost‐effective. Building such an evidence base would take a major federal or foundation effort to advance the practice of POP, and literally hundreds of studies testing practices in regard to specific types of problems. In turn, we have limited cost‐benefit analysis data from the studies we reviewed. The studies that did examine cost savings generally used limited data to estimate both costs and benefits and rarely did any systematic analysis. For practice it is not simply whether something works, it is equally important to provide a sense of what cost for what benefit. Answering this question should be a major focus of future studies.

The authors of this review plan to do a deeper dive and coding of the eligible studies to see if more light can be shed on these matters in a follow‐up publication. The prospects of succeeding in this effort with existing data are unclear, and was beyond the scope and timeline for this funded review.

On this front, it is important for more future studies to evaluate the impact of POP on outcomes beyond the standard crime and disorder outcomes examined in the studies included in this review. These studies tested impacts on aggregate crime/disorder, violence, property crime, disorder, drug sales/use and related outcomes such as probation/parole success or failure. POP was proposed as a flexible model that can be applied to a wide array of problems and our understanding of the model's potential would be enhanced through studies that assess its impact on issues such as cyber‐crime, human trafficking and other issues increasingly of interest to criminologists and criminal justice practitioners. Additionally, more future studies should be designed to examine its applicability in reducing resident fear of crime, improving citizens’ opinions of the police, and bolstering collective efficacy. Too few existing evaluations report on such outcomes to allow for a meaningful meta‐analysis, but our narrative review of the existing evidence suggests mixed and inconsistent findings across studies. This suggests the need for further research, particularly on POP interventions that include close partnership with and involvement of the community, which might be expected to have the greatest impacts on these perception‐based outcomes.

Lastly, this updated review also added the approach of performing meta‐analyses using log RIRRs as the effect sizes. This was done based upon in‐progress work by David Wilson which argues both that Cohen's D fails to produce effect sizes that are comparable across studies when based on place‐based count data and that the conversion of RIRR to Cohen's D is problematic. We still reported Cohen's D, including a majority (27 out of 34) of effects that were converted from RIRR, as we wanted to be consistent with the approach used in our original review (Weisburd et al., 2008, 2010) and other recent Campbell Reviews such as the updated hot spots policing review (Braga, Turchan, et al., 2019). This also allowed us to compare the two approaches.

Our results show that while similar conclusions would be reached about POP's effectiveness using either approach, it does appear based on our comparisons and examples discussed above in Section 5.5 that the Cohen's D approach may understate the impact of place‐based interventions, and that the RIRR approach appears to generate effect sizes that are more in line with actual changes reported in the studies themselves. Moreover, being able to convert the log RIRR to relative change in the treatment group versus the control group makes effect sizes more intuitive for researchers and practitioners alike. For instance, for our Cohen's D model our mean overall effect of 0.183 is not going to immediately tell a police leader much about the effectiveness of POP. However, using the RIRR approach allows us to more simply state that the relative reduction in the POP groups versus the controls was 33.8%. This is much more easily interpretable to practitioners, which is an important aim of Campbell Systematic Reviews.

Given that the RIRR approach is both more informative for practitioners and, based upon David Wilson's (in progress) work, more appropriate for place‐based studies (while also avoiding the problematic conversion of RIRR to Cohen's D) we encourage future meta‐analyses of place‐based interventions to adopt this method.

CONFLICT OF INTEREST

Weisburd is an author on four of the included studies and has been author or coauthor of several studies that have found POP and other proactive policing approaches effective, coauthored a book on POP with Anthony Braga and served on National Academy of Science panels which concluded that POP is a promising approach for crime prevention. Telep has coauthored a problem‐solving guide for the POP Center and has coauthored narrative reviews of policing strategies and helped design the Evidence‐Based Policing Matrix which suggests POP and similar approaches are effective based on existing evidence. Weisburd and Telep do not have any ideological bias toward the effectiveness of POP. Nonetheless, the inclusion of additional authors without prior work in this area reduces unconscious biases. Hinkle and Petersen have not conducted evaluation research or published on the effectiveness of POP outside of this Campbell Review (including the original version for Hinkle).

ROLES AND RESPONSIBILITES

SOURCES OF SUPPORT

This updated review was supported by the Campbell Collaboration through funding provided by Problem Solving and Demand Reduction Programme, hosted by the South Yorkshire Police. The original review was supported by Award 2007‐IJ‐CX‐0045, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice and the Nordic Campbell Centre.

PLANS FOR UPDATING THE REVIEW

Joshua Hinkle will coordinate the next update of this review with support from David Weisburd, Cody Telep and Kevin Petersen. We plan to update this review every 5 years, in accordance with Campbell Collaboration Guidelines. As the search strategy of this review depends upon the GPD, the plan is to carry out an update when five additional full years of studies (work through December 2023) have been fully indexed into the database.