Mimesis Law
20 January 2021

Judge Kopf, Criminal Defense Is No Country For Metrics

Mar. 8, 2016 (Mimesis Law) — Yesterday, Judge Kopf asked the question: is the practice of criminal defense no country for old men? He read a law review article called Criminal Defense Lawyer Moneyball: A Demonstration Project which claims there is a negative correlation between a lawyer’s experience and client results. In other words, it suggests the older a lawyer gets, the crappier a job he does. If that’s the case, yeah, let’s get rid of these old guys before they screw everything up.

Of course, it’s not the case. Delving into the application of statistics to criminal defense lawyer performance really only proves one thing. Statistics are no way to measure a criminal defense lawyer.

The Moneyball article is interesting. Two law professors decided there was some value to rating criminal defense attorneys through statistical performance. In an experiment only an academic could love, the professors came up with a way to determine which attorneys were doing a good job.

If you really want to get into the weeds with some scientific formulas, read the article. But here is the lowdown. The professors used North Carolina for their sample, which has a sentencing guidelines system. They weighted cases based on the charge at the time of the arraignment and the defendant’s criminal history category. Whoever got the biggest sentence reductions got the higher scores as a criminal defense attorney.

Great! We can plug in what every criminal defense lawyer does and see which one sucks, right? Assign each lawyer a number score based on performance and no one ever has to worry about hiring a crappy lawyer again.

The experiment revealed another surprise that triggered Judge Kopf’s post. Once a lawyer passes 8 years in practice, performance falls.

Our most surprising discovery is that experience actually has a negative correlation with performance after the first eight years: the more time an attorney has spent in the profession, the more likely that her clients will obtain a more severe sentence.

Sounds like experience is bad for a criminal defense lawyers. The professors kindly added a few potential reasons, which Judge Kopf noted in his post:

[A]ttorneys might devote less energy to their cases after many years on the job and may invest less time in learning about any changes in criminal law and practice—particularly for complex changes in the law, such as the arrival of sentencing guidelines. Similarly, more experienced attorneys may over time come to accept local norms about acceptable outcomes and stop pressing so hard against that accepted courthouse culture.

Before all you old guys start packing up your offices and you young guys start signing up for capital murder cases, it might be worth taking a look at why this whole idea of statistically rating criminal defense lawyers is a bunch of nonsense.

Criminal defense is as personal as it gets. What’s happening doesn’t lend itself to a slick academic formula or a fancy business metric. The government is pointing its finger at one of its citizens and accusing the citizen of violating the law. It wants all their money, or their liberty, or their life.

The Moneyball study went off the rails immediately by failing to recognize the differences in cases. As a base level, it used the charged crime and the criminal history category. Neither of which tells them a thing about what the person has done and what the prior criminal record really looks like.

Take an armed robbery, for example. One guy is a drug addict who sticks his finger in his pocket like it’s a gun and tries to rob the convenience store so he won’t die from heroin withdrawal. The other guy takes a machine gun into a bank with a crew and executes a well-planned heist. Same crime. In fact, the bank robber may have less of a criminal record than the drug addict.

How do you compare the outcome of cases like that? How do you determine if a lawyer did a good job? How can you tell if the sentence each defendant got tells us anything about the lawyer’s skill? The answer is easy. You can’t.

Every single case that walks into a criminal defense lawyer’s office is different. Each client is different. Some are smart. Some are dumb. Some are mean. Some are pitiful. Some are desperate. Some are sociopathic.

The crimes they are accused of are just as unique. Some are guilty. Some are innocent. Some are guilty of something, but not what they are accused of. Some are guilty, but had a good reason for doing it.

This combination of unique clients accused of unique crimes is why there will never be a metric to measure the performance of a criminal defense lawyer. It’s a job with a strange measure of what is victory.

A person can get sentenced to life without parole in prison. Sounds like a loss. Unless they were facing execution and weren’t ready to die. Then it’s a win. Maybe they walk out of court free, with nothing more than a little probation. Sounds like a win. Unless that felony conviction just ruined their future. Then it’s a loss.

The same muddled measure of wins and losses casts doubt on the idea that more experienced lawyers don’t perform as well. In fact, the answer is so clear it’s strange two law professors didn’t notice it.

Remember all those different cases and clients? Well, sometimes a really bad guy is really screwed. Or a not so bad guy who just hasn’t been able to stay out of trouble. Who do you think that guy hires? The rookie out of law school? Or maybe the more experienced lawyer?

That’s another part of lawyering that can’t be measured. The better you are, the tougher the case that hires you. So as you gain experience, and become a better lawyer, you get worse results. No wonder criminal defense lawyers are going crazy.

At the end of the day, the older, more experienced lawyer is not only who clients usually need to hire, but who younger lawyers should look up to.

A few years ago I spent 8 weeks in a federal courtroom trying a racketeering case with a lawyer named Herb Louthian. He is in his 80s and has been practicing law for over 50 years. Herb’s lawyering was brilliant. His experience had taught him when to be quiet and when to make a point. The result? His client walked away from the case a free man after the judge granted a verdict of acquittal.

So is there room for old men in the practice of criminal defense law? Yes. There is just no room for statistical performance metrics. Because this job isn’t measured in control groups and variable groups. Its measured in real people and real cases. The more experience you have, the more that makes sense. Whether that goes for judges, Judge Kopf, I can’t say.

13 Comments on this post.

Leave a Reply



Comments for Fault Lines posts are closed here. You can leave comments for this post at the new site, faultlines.us

  • Alan
    8 March 2016 at 10:11 am - Reply

    Josh, I mostly agree with your conclusion, but I don’t think your arguments are that strong. A lot of fields use the ‘special snowflake’ argument to claim that theirs isn’t subject to statistics. Law may truly be the exception, but I don’t think anecdotal evidence is the way to establish that. (Yes, every case is truly unique, but the same can probably be said for medicine.)

    That said, my problem with Judge Kopf’s post is that the study he refers to uses carefully picked metrics to judge a lawyer’s effectiveness. But what does that mean? What are we trying to approximate? ‘Justice per lawyer’? Do sentencing reductions accurately measure that? More importantly, would maximizing those metrics also maximize ‘justice’? I’m not a CDL (clearly), just an avid Fault-Lines reader, but from this pessimist’s viewpoint you’d actually see an increase in innocent people taking plea deals. Afterall, going for a ‘risk-free’ (from the lawyer’s POV) 5 year to 1-year plea bargain is better than a time-consuming, risky 5-year to 0-year acquittal.

    • Josh
      8 March 2016 at 10:17 am - Reply

      So do you think its the fields that are the problem or the use of statistics? If you can make the same argument in medicine (which I completely agree you can) then are statistics really going to tell us anything that useful about medicine? There are plenty of fields that statistics should not apply to. But Judge Kopf and I work in the law, so that is the focus here.

      Its not about special snowflakes – its about the need to put everything into numbers. If we can just analyze that bottom line, we can fix every problem. But that doesn’t always work.

  • Furslid
    8 March 2016 at 10:51 am - Reply

    I’m pretty sure I’ve seen similar arguments for doctors. Who is a better doctor, the head of Oncology at a major hospital or the doctor at an urgent care storefront who treats strep throat and sprained ankles? Who has a higher portion of their patients die?

  • Bryan Gates
    8 March 2016 at 1:16 pm - Reply

    I was surprised at the correlation between experience past eight years and more severe sentences. I expected that more experience would make attorneys more effective, in general. I figured the increase in effectiveness would level off after 10 years or so, but I was surprised it goes down.

    Criminal defense attorneys have been evaluated as long as they have existed. The evaluations have been based on an ad hoc assessment of reputation. In my county and state there are criminal defense attorneys who are known as the go-to man or woman for a someone accused of a crime who has the means to hire an attorney. Attorneys with good reputations tend to be good attorneys in my experience, but I wonder how accurate my assessment is of them is. After all, my opinion is based on anecdote, some observation and personal contact. Sometimes the middle-of-the pack defense lawyers gripe that one of the go-to guys is not any better than the rest of us – that he is someone who charges a large fee, puts on a show, and then pleads his client out. Is the lawyer perceived as better actually more effective?

    What makes a criminal defense lawyer effective is a hard question. Measuring sentence length is a sensible, but limited, metric. Many of my criminal defense clients tell me that getting the lowest possible sentence is a priority for them. Sentence length is a hard number that does not get fogged up with subjective factors. Acquittal rate (the percentage of not-guilty verdicts) might be another, but since jury trials are rare, there may not be enough data to make any conclusions.

    I am a numbers guy. I disagree that there is no role for statistical metrics in evaluating lawyers. The question is whether that system is any better than what we do now. If I were the chief public defender and there were statistics showing that Beth’s class C criminal history level 2 felon clients got 10 percent less time than Marilyn’s same felons consistently, I’d be interested in finding out what was going on there. Maybe Marilyn is more aggressive about taking cases to trial while Beth leans hard on her clients to take a plea. If a “trial penalty” causes Marilyn’s clients to get more prison time on average, that is something worth thinking about. The answer might be that Beth and Marilyn have different styles and are both following their instincts, intuition, etc. and neither should change. Criminal defense lawyers should be cautious about metrics (police departments have found a lot of ways to manipulate and misuse them), but we should not dismiss them outright.

    Note: I am a defense lawyer (non go-to guy) in North Carolina. I attended Wake Forest where the authors of the article are professors.

    • Richard G. Kopf
      8 March 2016 at 3:39 pm - Reply


      If you can’t measure it you can’t manage it.

      In my view, and despite the subjectivity of the endeavor, lawyers, even criminal defense lawyers, should seek out metrics to judge their own performance. Certainly, if they are in a public defender’s office or a law firm, the head cheese–assuming he or she is a lawyer–should want this type of information also.

      Here is an example applicable to federal district judges:

      Our court is the only district court in the nation that publishes each judge’s individual sentencing statistics. What do they show for me? That I’m a mean son-of-a-bitch. Yes, everyone knew that, but now they have the evidence to prove it.

      More seriously, as I slip into my dotage, those statistics, coupled with the functional death of the Guidelines, has caused me think harder about sentencing disparity writ large. Anecdotes wouldn’t help me to compare my sentencing behavior to my local colleagues or my colleagues nationally.

      All the best.


  • Richard G. Kopf
    8 March 2016 at 3:23 pm - Reply

    Dear Josh,

    As for Article III judges, we were perfect when appointed and remain so for life. Didn’t you learn a f–king thing after spending 8 weeks in a federal courtroom trying a racketeering case before an anointed one?

    Truly, all the best. Besides, I get your point.


    • Josh
      8 March 2016 at 3:26 pm - Reply

      I often marvel at the perfection of an Article III judge, especially here in these public forums. There is no way I would have some different opinion in a less public forum and you can count on not being able to ever prove I did…

      • Richard G. Kopf
        8 March 2016 at 3:40 pm - Reply


        Got me! All the best.


  • Peter Vanderwaart
    8 March 2016 at 8:19 pm - Reply

    Speaking a statistician, and not a lawyer (which I’m not), it’s too soon to get excited about this. When study gives rise to an unexpected hypothesis, the same study can not be used to validate it. Statistical methods require that the hypothesis precede, or otherwise be independent of, analysis of the data. I remember there was a ballyhoo a study that linked coffee to cancer. Subsequent studies failed to confirm it. Besides, the sample is not that big. It’s about the same size as all those polls with a “3% margin of error.”

    Any sample has unexpected and unreproducible patterns within it. If it wasn’t young vs old, it could have been city vs town, or male vs female, or Harvard vs UNC.

    • Richard G. Kopf
      9 March 2016 at 11:08 am - Reply


      Yes, what you write is true, but it is beside the point. If a well-designed study produces an unexpected result, you don’t ignore it. You test it again. Keep in mind that the study at issue was explicitly designed as a demonstration project.

      All the best.


  • Peter Gerdes
    13 March 2016 at 8:42 pm - Reply

    There are valid reasons to worry about confounding factors in studies like this. It seems plausible there is some reason (not effectively controlled for) why people facing more damning evidence tend to choose more experienced counsel. Maybe more experienced lawyers pass the easier cases off to their less experienced colleagues.

    Having said this the fact that each case is unique is totally irrelevant. Each child and their interaction with the teacher is unique as well but if those kids randomly assigned to teacher A average a 4.5 on their AP exams while those assigned to teacher B average a 2.5 it’s safe to say that teacher A is better than teacher B. If teacher B tries to explain this difference in outcomes by saying each child is unique it just begs the question of why teacher B disproportionately had students that were unique in ways that gave low scores while teacher A had students who were unique in ways that gave good scores.

    Statistics is just a fancy application of the commonsense rule that phenomena demand good explanations. Individual case differences doesn’t give a good explanation of why the cases that do better should go to the inexperienced lawyers.

    • Josh
      13 March 2016 at 9:19 pm - Reply

      Each kid may be unique, but the AP test is the same. So there is some standardization. Not the same with criminal cases. There are never two cases that are the same. So you have uniqueness on both sides, as opposed to only one side like your example.

      Statistics is a way of measuring standard things. Nonstandard things don’t lend themselves to the same measure.

      I am not particularly wedded to this idea. Its just that no one can come up with a good explanation regarding how statistics should apply to practicing criminal defense. I would love to plug in the facts of a case and have a formula to tell me what is going to happen. But that’s fantasy.

  • jr72023
    22 March 2016 at 9:34 pm - Reply

    Another case of what I call the, “Google Affect.” Thinking that any problem can be solved with an algorithm. Perhaps algorithms could solve all problems but, only if all of the variables are identified (not likely) and properly rated (even more unlikely.) When humans come into play this becomes nearly impossible. But, don’t try to tell the intellectuals that. We can all be reduced to numbers…and ad revenue. Just ask Google.