Mimesis Law
7 December 2019

No Country For Old Criminal Defense Lawyers

Mar. 7, 2016 (Mimesis Law) — As some of you know, I am fascinated by empiricism. For example, I wrote that data derived from the field of criminology could and should be put to far more use at sentencing. See my 2015 article in the Federal Sentencing Reporter here (under “Other Items of Interest”).

As I was thinking about my next post for Fault Lines, I began to wonder about how prosecutors and criminal defense lawyers would score on the Minnesota Multiphasic Personality Inventory, the most widely used and researched standardized psychometric test of adult personality and psychopathology. I couldn’t find anything on the subject.*

So, I began to look for other information about profiles of lawyers who live their lives dealing with criminals. There is remarkably little out there.  But, I did stumble across a particularly fascinating article that forms the basis for the question I present at the end of this post. See Ronald F. Wright and Ralph A. Peeples**, Criminal Defense Lawyer Moneyball: A Demonstration Project, 70 Washington & Lee Law Review, 1222 (2013).

Here is the abstract from a draft of this article that appears on the Social Science Research Network:

The book and movie “Moneyball” portray the iconoclastic general manager of a baseball team. When drafting new players, this GM de-emphasized the insights of baseball scouts as on-the-scene evaluators of a player’s talents, and looked instead to statistical measures of player quality. We take this idea from baseball into the criminal courts. In this article, we argue that criminal defense organizations could meaningfully evaluate the skills of their attorneys through the use of metrics, rather than relying so heavily on the in-person observation of their work in the courtroom. Statistical performance-based rankings could support better leadership in defense attorney organizations.

Rather than simply assert that a rating system is possible, we attempt in this paper to show its feasibility. We employ data from the North Carolina courts as a demonstration project to illustrate how an office might develop a rating system for the attorneys who work there. Our attorney ratings are based on the bottom line: sentencing reductions those attorneys achieve for their clients, principally through plea negotiations. We then use our tentative quality ratings to address the question of structural causes. What makes one attorney noticeably more or less effective than the typical defense lawyer? Our most surprising discovery is that experience actually has a negative correlation with performance after the first eight years: the more time an attorney has spent in the profession, the more likely that her clients will obtain a more severe sentence. We close with some reflections on other potential users of a statistical rating system, concluding that managers of defense organizations are better situated than judges, prosecutors, or clients to make wise use of ratings data.

Ronald F. Wright & Ralph A. Peeples , Criminal Defense Lawyer Moneyball: A Demonstration Project, SSRN (December 17, 2012) (emphasis added).

For purposes of this post, I don’t want the reader to be overly concerned with the adequacy of the research design or the statistical validity of the model constructed by the professors. Suffice it to state that the professors’ work has all the hallmarks of a sound empirical inquiry. Their work included plenty of R-squares, T tests, P tests and all the rest.

For now, I am interested in the surprising finding that at a certain point the experience level of criminal defense lawyers is negatively correlated with outcomes for the client.  The authors write:

Another variable that initially attracted our attention is years of experience. Intuitively, it is difficult to know whether experience would move the performance score up or down. On the one hand, attorneys with more experience might learn from past efforts and become more effective over time for their clients. On the other hand, attorneys might devote less energy to their cases after many years on the job and may invest less time in learning about any changes in criminal law and practice—particularly for complex changes in the law, such as the arrival of sentencing guidelines. Similarly, more experienced attorneys may over time come to accept local norms about acceptable outcomes and stop pressing so hard against that accepted courthouse culture. This phenomenon might be called the “Sam Rayburn effect,” recalling the former Speaker of the House of Representatives’ advice: “If you want to get along, go along.”

The data from North Carolina offers tentative support for the latter unhappy story about experience. The average case performance score remains positive for the first 5 years in practice; the average case performance for almost every experience level above that is negative. Figure 1 summarizes the trend.

kopf

Ronald F. Wright and Ralph A. Peeples, Criminal Defense Lawyer Moneyball: A Demonstration Project, 70 Washington & Lee Law Review at 1257-1259.***

Now, I am about to ask a question. But before I ask it, and request that you answer it, please view the following short clip from the opening scene of No Country For Old Men. The words that you hear spoken are by Tommy Lee Jones as Sheriff Ed Tom Bell, a laconic, soon-to-retire county sheriff.****

Now here is the question: Is the practice of criminal defense no country for old men?

Richard G. Kopf
Senior United States District Judge (NE)

*I have a hypothesis, however. I bet the L scale for more than a few prosecutors and criminal defense lawyers would be high. Know what I mean? By the way, when the L score is elevated for cops, that’s a bad thing. See Peter A. Weiss and James E. Vivian, Exploring the MMPI-2 L Scale Cutoff In Police Selection (2010).

**Ronald Wright is the Needham Y. Gulley Professor of Criminal Law at Wake Forest University; Ralph Peeples is a Professor of Law at Wake Forest University.

***In Figure 1, the average performance score for the category 0–5 years is based on 2393 cases and 45 attorneys; for 6–10 years, 2041 cases and 45 attorneys; for 11–15 years, 2225 cases and 42 attorneys; for 16–25 years, 1991 cases and 40 attorneys; and for 26 or more years, 1,847 cases and 42 attorneys. Id. at 1258 & n. 113.

****Here are the words the Sheriff speaks:

I was sheriff of this county when I was twenty-five. Hard to believe. Grandfather was a lawman. Father too. Me and him was sheriff at the same time, him in Plano and me here. I think he was pretty proud of that. I know I was. Some of the old-time sheriffs never even wore a gun. A lot of folks find that hard to believe. Jim Scarborough never carried one. That the younger Jim. Gaston Boykins wouldn’t wear one. Up in Commanche County. I always liked to hear about the old- timers. Never missed a chance to do so. You can’t help but compare yourself against the old timers. Can’t help but wonder how they would’ve operated these times. There was this boy I sent to Huntsville here a while back. My arrest and my testimony. He killed a fourteen-year-old girl. Papers said it was a crime of passion but he told me there wasn’t any passion to it. Told me that he’d been planning to kill somebody for about as long as he could remember. Said that if they turned him out he’d do it again. Said he knew he was going to hell. Be there in about fifteen minutes. I don’t know what to make of that. I surely don’t. The crime you see now, it’s hard to even take its measure. It’s not that I’m afraid of it. I always knew you had to be willing to die to even do this job. But I don’t want to push my chips forward and go out and meet something I don’t understand. You can say it’s my job to fight it but I don’t know what it is anymore. More than that, I don’t want to know. A man would have to put his soul at hazard. He would have to say, okay, I’ll be part of this world.

20 Comments on this post.

Leave a Reply

*

*

Comments for Fault Lines posts are closed here. You can leave comments for this post at the new site, faultlines.us

  • Robert Beckman
    7 March 2016 at 4:49 pm - Reply

    While I’m also a big proponent of empiricism, it’s often misused for conclusions that don’t really hold up. This is a common problem in my field (analytics), because so much of the math that used to be hard has now become trivial, so we have studies that appear to be far more rigorous than they really are.

    To pick on this study, the researchers didn’t propose a theory that would have disproven their hypothesis and that could be tested – they only proposed theories that support what they otherwise found. For example, if more experienced attorneys take harder cases (because they’re better attorneys) then we would expect to see a drop in the sentence reduction precisely because the best-outcome is worse than the prior-expected-outcome.

    To give a concrete hypothetical: an inexperienced lawyer will get a large sentence reduction on a weak case, and an experienced lawyer will get a larger reduction in that same case. But the inexperienced lawyer will get a tiny reduction in a hard case, while the experienced lawyer will get a small reduction in the hard case. If the relative volume of hard-to-weak cases is different between the two, then the inexperienced lawyer (who only takes easy cases) will appear better than the experienced lawyers who takes disproportionately hard cases.

    The problem is that we don’t know a priori which types of cases go to which attorneys – how do you measure how hard of a case it is in advance?

    • Richard G. Kopf
      8 March 2016 at 8:04 am - Reply

      Robert,

      I respectfully suggest that you read the entire article again. Two things.

      First, this effort was intended as a demonstration project to show how a manager of a Defender program or an authority who appoints private lawyers or a criminal defense lawyer might gauge the performance of attorneys in his or her program. Thus, it did not start with a hypothesis in the normal scientific sense of that word.

      Second, if you look at the gauge–essentially the difference between the charged crime and the plea-bargained offense that forms the basis for the sentencing outcome–it should make no difference whether the case was serious or relatively minor to start with. The question was did the lawyer obtain a reduction in the charged offense, serious or not. And, most importantly, the negative correlation between experience and outcome was stark, that is, it was all in one direction for all levels of experience after five years.

      All the best.

      RGK

      • Robert Beckman
        8 March 2016 at 2:13 pm - Reply

        Judge Kopf,

        I may have been too harsh on these researchers in suggesting that their findings aren’t useful, as they themselves call out the dearth of data from which to analyze these results.

        That said (unless I misread their analysis) I see the same problem here I’ve seen in healthcare (my field), where comparative outcomes are used as a proxy for effectiveness.

        To try to make this clear, I’ll illustrate one finding in healthcare, then try to bridge to the legal analog and see if you agree (note that this is far more complex than any of the examples we’ll use, as I’m sure that law is more complex as well)*.

        Take two patients in the same car accident who both receive identical diagnoses. At presentation to the emergency department, both will receive the same triage, and both go into simultaneous surgery. One will survive, the other will not. The patient who survived will have been treated by the junior surgeon, with the apparent result that a more senior surgeon results in worse patient outcomes, but is that really the case? In the studies I’ve done, the patient with the worse chance coming out of triage will get the more experienced surgeon, so that while both had the same diagnoses, their chances of survival were not actually the same, even though most of the medical record data that is readily available would suggest that they are (this is a common issue with Medicare data sets – some other sets augment what Medicare calls for which makes this level of analysis possible).

        With that example as one of my priors, I’m thinking of the legal analog like this:
        Two people are charged with the same crime (same diagnosis), and are each represented by a different lawyer (surgeon). Based on this, we should expect to see both** receive the same sentence, but instead we see that the person with the more experienced attorney gets a worse result (from the study) and conclude that there is a negative correlation between outcome and experience.

        If, instead, when two people are charged with the same crime at a single firm the more experienced attorney represents the person with a worse fact pattern (even though the same charges) then they will appear to have worse outcomes compared to a less experienced attorney on the same charges simply because the junior attorney gets the “easy” cases within the same class of charges.

        This of course then raises the question: are more experienced attorneys getting the harder cases within a single charge category (as in the doctor example) or are they getting equivalently difficult cases and are performing worse***? This study doesn’t show one or the other, but does indeed show that this information is available and should be used (which I think we agree on).

        * if you disagree with this analysis, I’d love to understand why so I can apply that insight back into my own field, as I’m a legal dabbler

        ** over a large number of cases – the individual outcome, of course, is a crapshoot

        *** for example, should we conclude that attorney Stephen Jones (randomly picked attorney with infamous clients) is less effective than a younger attorney, or merely that he tends to pick up cases with heinous fact patterns?

        • Bryan Gates
          8 March 2016 at 3:21 pm - Reply

          I think the authors assumed that the cases were randomly assigned to a public defender in a particular office. That is likely how it works. When the PD’s office is appointed, there is not much to go on about the difficulty of a case other than the charges. For example only the most experienced APD’s are assigned to class F felonies and above on a rotating basis to keep the caseloads evened out. Under that system, over time, each could be assumed to get a proportionate share of easy and difficult cases. Now if someone in the PD’s office made some more detailed assessment of which cases to assign to which APD, judging the APD by sentence length would be unfair. Even if the assignment is random, the differences in sentence length might have more to do with vagaries other than the skill of the assigned lawyer.

          There is a fairly large PD’s office in my county, which is in North Carolina where the data originated. I do not work for the PD’s office.

          The article reminded me of the book “Homicide: A year on the killing streets” and the series based on it. The homicide detectives were judged primarily on the clearance rate — how many of their cases they solved. When a detective was in line for the next case, they prayed for a “dunker,” a case that was easily solved. They dreaded a “stone whodunnit” such as a homeless person pulled from a river. A detective who draw a series of “stone whodunits” would see his clearance rate plunge.

        • Richard G. Kopf
          8 March 2016 at 4:01 pm - Reply

          Robert,

          If you compare board certified colon/rectal surgeons with board certified colon/rectal surgeons and you focus in on a specific cancer and control for the age of the patient and other confounding variables I would guess that you would agree that those numbers might be useful if only to the patient seeking a surgeon. The authors of the law review article certainly did not have controls approaching the foregoing. But, they did a pretty good job of controlling for outcomes if you read their design methodology carefully.

          After all, the law review article was intended as a demonstration project. While one can certainly find fault with the research design for a variety of reasons, the professors did us a solid by trying to bring empirical vigor to the question of the effectiveness of criminal defense lawyers.

          More subjectively, I wanted to touch a nerve, as one commentator has suggested. We lawyers/judges don’t like to be measured, and we sure as hell don’t like to be told that we are too old to do our work. I thought it would be useful to “poke the bear” if only as a thought experiment.

          All the best.

          RGK

          • Robert Beckman
            9 March 2016 at 5:23 pm -

            Judge Kopf,

            Here we entirely agree. Doctors also don’t take kindly to performance metrics, even setting aside some studies that find that more experienced doctors (in some fields) have worse outcomes.

            Pulling this example from memory as I don’t have the research at hand: patients have statistically better outcomes when experienced cardiologists are NOT present than when they are (the study used a major cardiology conference as a natural experiment to reduce the availability of cardiologists in hospitals) as junior doctors were less likely to try more invasive interventions, and a more conservative treatment across the board yielded better results (with the obvious caveat that some patients would have benefited from stronger interventions and didn’t receive them).

            Bryan,

            The random assignment of PDs to defendants may be the biggest explainer here, as that’s often* not how doctors are assigned, in the narrow case of emergency departments where there is more than one doctor available at a time.

            *there are some horrifying cases where senior doctors are assigned “easy” patients because they can control their own work load and choose to do less work, and some not-horrifying cases that look similar where a junior Doctor treats the more seriously injured patient because the senior doctor thinks they have almost no chance and can either choose to save one life (the less injured patient) while condemning the other to almost certain death, or probably lose two patients because the junior Doctor couldn’t save the less injured patient, and the senior Doctor couldn’t save the more injured patient. Sadly these two scenarios look almost identical – this isn’t my specific area, so I haven’t studied this in depth, and only share the conflating factors when people are involved in their work selection.

  • Cornflake S. Pecially
    8 March 2016 at 1:28 am - Reply

    Deontologicaly speaking the depths of this sort of rabbit hole devolution, if true, is nearly enough to make me want to wallpaper the consequentialism nightmares of every CDL over thirty five, who has ever shrugged his or her shoulders, with the most horrific house of mirrors wallpaper ever imagined.

    Are you trying to get that unkindness of ravens that likes to hangout around the courthouse to talk again judge?

    • Richard G. Kopf
      8 March 2016 at 8:01 am - Reply

      CSP,

      I don’t want unkindness. But, yes, I do want the ravens to talk.

      All the best.

      RGK

  • Raymond Rigat
    8 March 2016 at 8:10 am - Reply

    Absolutely no country for old men (or old women) for that matter. But you don’t have to be old, even if you’re 92. Too many are old at 40. I think the trick is to stay inspired. this is the greatest profession in the world. I really believe that. I don’t want to do anything else. Never did. And nothing is better than standing up I court defending a criminal defendant who doesn’t have a friend left in the world against the Leviathan of the state. I mean, if that doesn’t get your blood running, heck, your not old- you’re dead.

  • Roger
    8 March 2016 at 8:40 am - Reply

    Respectfully, I suggest you read Mr. Beckman’s comment again. He spoke to the difficulty in gauging whether the study adequately accounted for more experienced attorneys taking “harder” cases, but your reply changed that into “serious” cases.

    In my younger days, I prosecuted rapes where a stranger was caught in the act of assaulting his violently-protesting victim and I prosecuted what today we would call date rape. They all carried the same range of punishment of five years to life, but no serious observer would have thought that a reduction in charges and a 3-year sentence for a he-said she-said date-rape case showed better lawyering than a 5-year sentence on a stranger rape case with a battered victim. Date rape cases at that time were hard for the state to win, while stranger rape cases with any kind of decent evidence were basically lay ups where juries not only convicted a very high percentage of the time but also gave extremely long sentences.

    I read Mr, Beckman’s criticism as noting the study’s failure to address the possibility that more experienced lawyers handle more difficult cases. This criticism may or may not have legs, but it is a different argument than you responded to.

    • Richard G. Kopf
      8 March 2016 at 4:21 pm - Reply

      Roger,

      Without seeing the actual data on easy v. hard cases it is difficult to answer your question. Here is what I thought: If you have only two variables, the charged offense v. sentenced offense, one would have to assume that all lawyers 5 years and under as CDLs get easy ones, and all lawyers over 5 years as CDLs get harder ones. If you only had two age subgroups–like 5 or less and 6 or more–that wouldn’t be too surprising. But the slope of the line is downward (worse) for EACH age subgroup, with each EACH age subgroup receiving poorer scores than the preceding one. Thus, the findings are at least cause for further investigation as an empirical matter.

      All the best.

      RGK

      • Richard G. Kopf
        8 March 2016 at 4:28 pm - Reply

        Roger,

        I slightly misstated the slope of the line in my prior reply. Sorry. There is an uptick for subgroup 11-15, but the correlation remains negative. All the best.

        RGK

  • Judge Kopf, Criminal Defense Is No Country For Metrics
    8 March 2016 at 9:14 am - Reply

    […] 8, 2016 (Mimesis Law) — Yesterday, Judge Kopf asked the question: is the practice of criminal defense no country for old men? He read a law review article called […]

  • Alan
    8 March 2016 at 9:36 am - Reply

    I’ll admit I didn’t read the original article, but from your description, Judge, it seems like you run into the underlying problem of all metrics: you can’t measure what you want to improve, so you improve what you can measure.

    • Richard G. Kopf
      8 March 2016 at 4:30 pm - Reply

      Alan,

      I agree, sorta. Empirical stuff is hard and can take you only so far. But, on balance, I remain convinced that an empirical approach can still teach us something of value.

      All the best.

      RGK

  • Jim Burke
    8 March 2016 at 2:45 pm - Reply

    Several ways to approach this. Simple – many of us get lazy as we age/get experience/get tired. But some of us keep the fire and improve. A step up: As already pointed out, really hard to quantify criminal defense work. More complicated answer: As some of the younger criminal law academics have recently been writing, with all of the collateral consequences attendant to misdemeanors, defense lawyers who really get into the client’s case might well end up with a resolution that, on the face of it, seems worse (more time, higher level charge) but on digging in, is actually a better deal for client since it avoids a worse collateral consequence than the more commonly negotiated lower charge. So on a simple analysis (lower result-better result) they might score worse.

    Defending criminal cases is complex and intricate – far more so than prosecuting. Being now into the “older lawyer” category myself (I have been eligible, if I wanted, for social security for some time now ) I do on some days feel that it is a younger person’s game. Ah, and then there 9is the day I can really help a client.

    • Richard G. Kopf
      8 March 2016 at 4:37 pm - Reply

      Jim,

      Thanks for taking the time to comment. I am 69. Some days I think I’m pretty good, and others not so good.

      Example: I must have done two thousand or more first appearance/arraignments when I served as a Magistrate Judge between 1987 and 1992 (the height of the crack, Crips and Bloods plague in Omaha). Several weeks ago, our MJ got conflicted out at the last moment, and she asked me to do the first appearance/arraignment. I did it including a detention hearing. As I was about to leave the bench, I always ask “is there anything else we take up.” The AUSA said, rather meekly, “Judge perhaps you ought to ask the defendant how he pleads.” Oh, shit!

      All the best.

      RGK

  • losingtrader
    9 March 2016 at 11:32 am - Reply

    It’s nice you have the time to do this stuff since, as a senior status judge, you get paid for doing nothing–or, rather, don’t get paid for doing something

  • Peter Gerdes
    13 March 2016 at 11:44 pm - Reply

    First let me say the aim of the paper was not to defend any particular statistical relationship about attorney performance, only sketch how it might be done. Having said this there are several worrying aspects to the analysis that cast the purported relationship between experience and poor outcome into doubt.

    (And EVEN IF you took it at face value it would only suggest that the better attorneys tend to migrate out of the criminal defense trenches to judgeships, professorial positions, or management)

    The authors perform a straightforward regression that controls for far too few potential confonders. Worse, their methodology itself introduces even greater potential confounders.

    1) Biasing data selection:

    Consider how they select their sample of attorneys: “for each of these districts, we identified the defense attorneys with the most case dispositions.” In other words they aren’t looking at a representative sample of all attorneys but just the ones who handled the most cases. This presents a serious sources of error:

    A defense attorney with a successful career should expect to handle fewer cases per year as their career progresses as they are assigned/gravitate towards more serioues harder, more visible cases supported by greater client resources…all factors that decrease how many cases they handle.

    As a result defense attorneys with many years of experience who are still taking on lots of cases are more likely to be doing poorly in their profession. The fact that they include caseload and charging severity in their regression doesn’t control for this effect as the bias resides in their criteria for data inclusion (the attorneys who work only big murders aren’t even in the data). This is a critical flaw.

    2) Naive use of ordinary least squares

    The authors force a linear model onto the data without providing any plots of residuals or other information that might let one check the supposed linearity assumption. Offense class is almost surely not a linear effect.

    Also no tests were done to check if the results were robust when extreme results (like those from district 21) were removed.

    3) The relationship between years experience and performance appears non-linear (figure 1) and is dominated by the extreme low performance of those with 25+ years on the job.

    As linear regression is known to be quite sensitive to extreme values it’s quite likely the result would no longer be significant if we excluded defense attorneys with 25+ years experience.

    4) Too little data for the number of independent variables

    Some of the districts have as few as 2 attorneys

    5) A bevy of uncontrolled for potential confounding effects:

    *) Both guilt and social undesirability correlate with having friends who need attorneys. If attorneys gain clients through some combination of random choice and referral this implies that clients gained through referral will be more likely to be the sort that are unsympathetic to judges.

    *) Good experienced attorneys are more likely to be patronized by prior clients with more serious priors. This could mean the benefits of experience are being subsumed into the prior record coefficient leaving the experience coefficient to capture the effect of being unable to generate repeat customers on results.

    *) The practice of handing the easy cases off to the inexperienced or other non-random assignment.

    I could continue but the long and short of it is that it’s a toy example not a real attempt to demonstrate a negative relationship between attorney quality and experience.

    • Richard G. Kopf
      14 March 2016 at 3:07 pm - Reply

      Peter,

      Thank you very much for your deep dive into the research methodology. I appreciate it and agree with some of your criticism. But I would hardly call the effort a “toy.”

      As you recognize, this was a demonstration project. It came out with a counter-intuitive result. I am interested how you would structure the research methodology to avoid the faults you focus upon but yet test whether the negative correlation is real rather than a false artifact of weak methodology.

      All the best.

      RGK