Barriers To Entry and Innovation

Malcolm Gladwell revisits the mismatch problem in the New Yorker.  (Only now he calls it “The Quarterback Problem”).  Whatever.  The main thing you need to know is that for many professions, there are gates and gatekeepers.  The gatekeepers use the gates–in the form of tests, degrees, or diplomas–to help them weed out the worthless and forecast who will make a great fill-in-the-blank (take your pick: lawyer, NFL quarterback, teacher, financial advisor).  The problem is that the gatekeepers have no idea what they’re doing.  Their gates are useless filters.  In some cases, they achieve the exact opposite of their purport.  Instead of helping gatekeepers discover the productive and worthy, these gates turn them away.  Worse, the gates entrench the power of the mediocre and raise the cost for those who need their products.  Gobbits worth pondering: 

A group of researchers—Thomas J. Kane, an economist at Harvard’s school of education; Douglas Staiger, an economist at Dartmouth; and Robert Gordon, a policy analyst at the Center for American Progress—have investigated whether it helps to have a teacher who has earned a teaching certification or a master’s degree. Both are expensive, time-consuming credentials that almost every district expects teachers to acquire; neither makes a difference in the classroom. Test scores, graduate degrees, and certifications—as much as they appear related to teaching prowess—turn out to be about as useful in predicting success as having a quarterback throw footballs into a bunch of garbage cans.

The NEA won’t tell you that, though.  Neither will the No Incompetent Teacher Left Unhired law either.  Anyway, Gladwell offers some policy prescriptions: 

[Teaching] needs an apprenticeship system that allows candidates to be rigorously evaluated. Kane and Staiger have calculated that, given the enormous differences between the top and the bottom of the profession, you’d probably have to try out four candidates to find one good teacher. That means tenure can’t be routinely awarded, the way it is now…

An apprentice should get apprentice wages. But if we find eighty-fifth-percentile teachers who can teach a year and a half’s material in one year, we’re going to have to pay them a lot—both because we want them to stay and because the only way to get people to try out for what will suddenly be a high-risk profession is to offer those who survive the winnowing a healthy reward.

I wish we lived in a world where this kind of experimentation could get its chance.  Imagine towns full of dispersed venture capital–in the form of vouchers for each child–and then imagine the educational entrepreneurs devising ways of staffing classrooms with eighty-fifth percentile teachers.  Nice fantasy, huh?

The Mismatch Problem Corroborated

His new book Outliers isn’t out for another month, but today Malcolm Gladwell might justly bask in that warm glow arising from confirmation bias. The mismatch problem, you’ll recall, is that we have no idea what we’re doing when we try to measure talent before it’s been put to actual use. Scouts fail to identify diamonds in the rough among draftees because they believe they can correlate success with a set of easily tested attributes–how high you jump, how fast you run and so on. The problem is, these attributes have little or no correlation with superior future performance. Or, as William Blake so eloquently put it, the tygers of wrath are wiser than the horses of instruction.

Now the ABA Law Journal has some good news for Gladwell’s thesis. A study by an unnamed top 25 law firm suggests that law school rank and GPA are worthless as predictors of future big law success. Harvard Law? Who cares! Yale? A nice credential to fulfill your quest after years and years of application-review servility, but now…who cares! As one of the study’s authors, Ron Paquette, tells the ABA Journal, “The Harvard attorneys do not perform any better than those at the 30th-ranked law school.” (The study defined success as longer tenure at the firm, higher productivity, and being a good cultural fit.) 

However, the authors do claim to have found some counterintuitive correlates for success. Among them are involvement in community groups or even participation in college athletics. In some ways this is obvious. I’ve always thought the frat house prez would make a great lawyer.

Gladwell’s Mismatch Problem

Malcom Gladwell has a thought-provoking video up, this time on something he calls the mismatch problem. In a nutshell, we think we know how to measure talent. The problem is that we have no idea what we’re doing. Our measurements appear to bear no relation to future performance. In Gladwell’s words, it’s a misjudgment that occurs when the criteria we use to assess someone’s ability to do a job is radically out of step with the demands of the job itself.

Since S. recently heard about a consulting firm that gives a battery of tests to help you find a career that matches your strengths and weaknesses, and since I’ve taken an interest in something called the Signaling Theory of Education, it’s worth rehearsing some of Gladwell’s points.

Every year, for a week or weekend, before professional sports teams draft the incoming rookie class, they hold what are called combines, a sort of mini-camp where prospective draftees go through a dog and pony show for all the scouts. The combine’s purpose is let scouts collect objective data about who these kids are and what they can do. Millions and millions of dollars are at stake after all, so naturally teams want some reliable way to predict how good of an investment they’re making in whomever they draft. Will this guy help us win? Is he worth it? In the NBA combine, these kids jump, they run, they lift weights, they take IQ tests, run drills–anything and everything to predict how well these kids will play.

But do you know who D.J Strawberry is? I certainly don’t. But last year his scores at the NBA combine were the highest of any incoming rookie. And yet he was awful. Strawberry averaged just two points a game during the entire year for the Phoenix Suns. All the top scorers at the combine turned out to be awful. Some didn’t even play in the NBA. But what about Kevin Durant, who won the rookie of the year award? He ranked 78th at the combine. In fact, the top five draft picks all performed poorly at the combine. It’s a remarkable waste of time–the combine and all of its tests are poor predictors for how well anyone plays basketball.

Or consider the Wonderlic test, an intelligence test given to NFL rookies in their combine. Being a quarterback in the NFL requires great cognitive demands and sophisticated decision making. Thousands of plays have to be memorized. Quarterbacks watch 100 hours of video every week to study the offense they plan to run and how the defense of the opposing team will react. So again, you might think that measuring intelligence with the Wonderlic might be important. But it turns out that among the 7 worst Wonderlic scorers in history are two of the best to ever play the game–Terry Bradshaw and Dan Marino. On the other hand, none of the seven best come close–Drew Henson, Eli Manning, Tony Romo…the list is laughable. So if in an industry where millions of dollars are at stake, where scouts are paid hundreds of thousands to measure talent in any quantifiable way, if in these multi-billion dollar industries our methods to predict productivity fail, then how about in other areas of work where measurements of productivity are even more vague and elusive?

When everything we use to assess talent has no correlation, or worse, a negative correlation with our actual performance on the job, it’s time we start rethinking some of these institutions. Consider more from Gladwell:

How do we ensure which teachers we hire are the best? Currenlty teachers are required to have a BA, teaching training, license from the state, academic work associated with your specialty. Does this ensure a high quality teacher? Not at all. Do any of these requirements correlate with increasing student performance? Not at all. So it’s the Combine all over again.

What about lawyers? Surely lawyers would know what correlates with what makes a good lawyer. But of course they don’t. The University of Michigan has an extremely generous affirmative action program, meaning minority applicants with lower GPAs, test scores and so on are accepted more frequently than white candidates with those same scores. This program then set up a natural experiment: measure the success across time of all UM Law graduates–30 years worth–and compare the success of those graduates who tested well with those minority graduates who did not. As it happens, on any measure of success, Gladwell claims there was no correlation (according to UM’s research). So, yet again, we have a mismatch problem. The criteria, which we think are associated with being a good lawyer–high LSATs, high GPA–have no bearing on how good of a lawyer you end up becoming.

So why do mismatch problems occur? Gladwell says it’s a madness for imposing certainty. A deep-seated need for clear and reliable statics, a hard-wired impulse for rational plan making. Also, he says, the complexity of jobs has increased. The cognitive demands required to succeed have multiplied and we can’t track them. So the world has changed, but the way we hire people hasn’t.

But there’s only one reliable way to measure productivity: to wait until someone’s on the job.