Baseball Musings
Baseball Musings
January 15, 2005
Are you Positive?

In a comment to this post, Al Bethke wrote:

I'm just not worried about false positives. I am worried about the integrity of the game, and with this strict policy, it is stronger than ever.

I'm very concerned about false positives. As Jayson Stark points out:


The worst part of testing positive would be getting that label Steroid User stamped on your forehead. That's a scarlet letter that these players would have to wear for the rest of their lives. If you don't believe their reputations will be tainted forever, just ask Jason Giambi -- if you can find him.

For a high-profile player, that means not just a life sentence of boos and insults. It means having everything he ever accomplished thoroughly discredited. And you sure don't want to be a utility infielder who tests positive. You'd be looking at playing the rest of your career in Korea.

So you can't evaluate this deal without remembering there are two levels of penalties -- formal and informal. There's a price to be paid to baseball -- and a price to be paid in the real world.

False positives happen because drugs tests are presented to the world as Bernoulli random variables (player was positive or negative) when in reality the tests measure some indicator of drug use in a continuous fashion, and use a cut off to indicate positive or negative. Depending on where you place the cut off, you're going to get more or less false positives. And it's a tradeoff. In general reducing false positives means increasing false negatives, and vice-versa.

A Receiver Operating Characteristic (ROC) curve is used to determine the cut off. The name derives from signal processing, the first use of this methodology. You can find a nice explanation of ROC curves at Steve's Attempt to Teach Statistics (StATS).

To understand an ROC curve, you first have to accept the fact that MDs like to ruin a nice continuous outcome measure by turning it into a dichotomy. For example, doctors have measured the S100 protein in serum and found that higher values tend to be associated with Creutzfeldt-Jakob disease. The median value is 395 pg/ml for the 108 patients with the disease and only 109 pg/ml for the 74 patients without the disease. The doctors set a cut off of 213 pg/ml, even though they realized that 22.2% of the diseased patients had values below the cut off and 18.9% of the disease-free patients had values above the cut off.

The two percentages listed above are the false negative and false positive rates, respectively. If we lowered the cut off value, we would decrease the false negative rate probability, but we would also increase the false positive rate. Similarly, if we raised the cut off, we would decrease the false positive rate, but we would increase the false negative rate.

Let's take testosterone as an example. It's a banned substance under the CBA (see page 160 of the CBA, page 171 of the PDF). Here's a research paper on the subject of developing a new way of testing for exogenous testosterone use. You see, you can't test for testosterone (T) directly, because we all make testosterone naturally. The standard test looks at the ratio of testosterone to epitestosterone (E). The ratio (T/E) should be about 1.0. The IOC used a cut off of 6.0 for the Los Angeles Olympic games. But, as the paper reports:

The overall incidence of urinary T/E in the general population of healthy males not abusing steroids is <0.8%

In other words, .8% is the upper bound of how many people are going to test positive for testosterone abuse falsely. In other words, if you test 1000 baseball players using this criteria, 8 may come up positive, even if no one is using steroids!

Maybe 8 isn't a lot to Al, but it sure is a lot to me. But to get that number down to eight, you have to throw out almost half of the people who really are injecting testosterone! Take a look at this table from the paper. Using T/E >= 6.0 as a cut off, 25 of the 46 subjects getting injections of testosterone test positive, while 21 test negative. Even if you lower the cut off to 1.2, just slightly above normal, you still only catch 38 of the 46, while the false positive rate goes up five times.

And people do use this test. The National Center for Drug Free Sport, Inc. offers a package of anabolic steroid tests, and lists the cut offs here (last table). The T/E cut off they use is 15:1; they're obviously afraid of returning a false positive, but they are clearly tossing out a lot of true positives as well.

Where is the integrity in this? Half the players abusing these drugs are going to get away with it, while a small number of innocents are going to be branded cheaters. The union understands this, which is the reason they resisted testing for so long. The MLBPA's failure was not recognizing that their own rank and file didn't like their brothers using these drugs. Now those players have put their reputations and paychecks at risk to try to drive these substances from the game. I hope it turns out to be worth the risk.

P.S. If anyone has the information, I'd like to know if the T/E test is used, and where the cut off is set. If anyone has the actual ROC curve for this ratio, I'd love to see that, also.


Posted by David Pinto at 10:13 AM | Cheating | TrackBack (2)
Comments

I'll bet you that this new policy allows for a retest BEFORE the first test is made public. Making the test results public before confirmation of a second test would never be allowed by the Players Union for the exact reasons that you give - the label Steroid User (or, more correctly, Performance Enhancing Drug User) would stick with you for the rest of your career (and life).

Posted by: sabernar at January 15, 2005 12:13 PM

Great work David. I added some of my own observations in a plug/continuation. You've really been doing some awesome work lately.

John

Posted by: John at January 15, 2005 02:50 PM

Thanks so much for this post-- if only traditional sports reporting were so rational. I ain't never voting for John McCain... :-)

Posted by: john swinney at January 15, 2005 04:12 PM

It seems like Al has become David's muse, in a twisted sort of way. Or maybe "easy target" is a better word for it. Either way, excellent post.

Posted by: Joe at January 15, 2005 04:26 PM

Yeah, my gut is that there's a provision for a second test to verify the first's results before public identification is made. The problem of false postitive is substantially reduced by the second test.

Posted by: Allen at January 15, 2005 04:38 PM

But will a second test yield a different result? Maybe I misread this, but it sounded to me like the problem was that some people just naturally score high T/E ratios.

Will they always test positive even if not taking anything? Or is the high T/E just an occasional high fluctuation, in which case retesting them will "clear" them?

Posted by: Arthur at January 15, 2005 06:09 PM

Firstly, thanks for putting some time into something I was too lazy to do myself. We MUST presume innocence of a player until conclusive evidece is presented, and the margin for error in this test leaves doubt. That's the way our legal system is supposed to work, and the reason is to protect those 8 out of 1000. Better to let the 100 who are testing negative get away with it than to falsely accuse the 8 on the other side who are innocent.

I have one other question, to which I think I already know the answer. But I'm wrong about 83 % of the time, so here it is. A person's testosterone production varies greatly enough between the ages 18 to 45, and from person to person, so that, if we were able to chart Bonds's testosterone production through his years, a sudden spike would not mean that he started 'roiding up, right?

Posted by: Mr. David M. Beyer at January 15, 2005 08:43 PM

David, I think you're too concerned about the false positives issue. Of course, I simply look at like I do Olympic testing, it's not 100%, but it's as good as they can do.

Also, Lance Armstrong has been labeled a "cheater" for a couple years now by many, and he's never been caught once. It's a never ending cycle.

Also, I'm not exactly sure what you are attempting to say...testing is bad? Players should be able to test positive a few times before suspension?

Most of us work in situations where a positive test can end employment, no questions asked. I have no fear of it, because I'm clean. I think the players, at least the vast majority, feel the same way.

Posted by: Al at January 15, 2005 09:20 PM

For those of you searching around for more info on ROC, false positives, etc., another good search term would be "theory of signal detection". Hope that helps.

As for the idea of a second confirmation test, I HAVE to believe there is a provision for such a thing.

Posted by: Tom G at January 15, 2005 10:30 PM

"Also, I'm not exactly sure what you are attempting to say...testing is bad? Players should be able to test positive a few times before suspension?"

How can you not be sure what he's saying?

He's saying that people are going to test positive even though they did nothing, and that's absolutely something you should be concerned about.

"Better one hundred guilty men go free than one innocent man be condemned." - Thomas Jefferson

Posted by: Larry Mahnken at January 15, 2005 10:59 PM

Al- you are comparing apples to oranges. The testing that most working people have to pass is a test for illegal drugs like cocaine and pot. Those substances don't occur naturally in the body. So you are basically looking for any amount of the compound. It's a lot easier to set a cutoff that will catch most of the users without ANY false positives. Also a false positive is more likely to be the result of testing error, not the bodies natural level of the compound, so a second test is more likely to rule out false positives.

Al if your workplace would test for steroids and set the bar to catch more than 75% of the cheaters, you should be afraid, even if you're clean.

Besides, the tone of your posts reminds me of roid rage, you sure you're clean? :)

Posted by: Ivan at January 16, 2005 03:04 AM

An interesting note is that looking around the internet, most agree the testing program is too lenient, a rather small penalty for the 1st offense. David seems to be one of the few saying it is too tough. I have to agree with the majority in this case.

Posted by: Al at January 16, 2005 09:29 AM

yeah but how many of thie majority understand any of what David discussed in this post???


Posted by: Ivan at January 16, 2005 10:14 AM

"I have to agree with the majority in this case."

That's because you're a sheep.

Posted by: Larry Mahnken at January 16, 2005 01:57 PM

I can't imagine the Players Union agreeing to a testing plan that is going to condemn innocent players as Drug Users. I guess we'll have to see how it all plays out.

Posted by: sabernar at January 16, 2005 03:05 PM

SO AL -
You said I think you're too concerned about the false positives issue...Most of us work in situations where a positive test can end employment, no questions asked. I have no fear of it, because I'm clean.

But would think false positives were okay if you were clean but the number on your test suggested you were guilty and it not only got you let go, but affected your ability to do the same job anywhere else?

You're clean. Fine. But you're counting on not being one of the unlucky ones who is clean and at the same time indicated as unclean by a fallible test.

So what that leaves me wondering is, as I said before, what would you think/feel if you had a false positive and it was you who was being fined, outed or even let go? Would that seem like a reasonable cost?

Or do you tend to agree with the ex-governor of Texas who once said, "Better to execute a hundred innocent murderers than to let one guilty one go free"?

Posted by: jeff angus at January 16, 2005 03:43 PM

First of all Larry, allow me to commend you for writing such intelligent counter arguments as:

"That's because you're a sheep."

I have spoken my opinion and see no need to go into more specifics. I doubt if the drug testing issue will come up much, as it exists in other major sports, and very rarely does. I repeat my feeling that it is a good thing for the game. I do, however, wonder if the "false positives" are such an issue, why no other sports has had this problem.

I will wait for David to answer my original question, if he is proposing giving everyone a free positive test, or what exactly he is suggesting.

Also, David and I may not always agree (he would likely give Bud Selig little credit for the 2004 campaign, which arguably was the most successful in MLB history, with 2005 sure to be even better, with DC replacing Montreal), but he is always respectful.

Posted by: Al at January 16, 2005 06:19 PM

Al-

David has stated many times what he thinks should happen..... We should legalize steroids. His point with this post is that there isn't a good testing system. You either get too many false positives or too many false negatives. There isn't any middle ground.


On the respectful side, you made a patently silly arguement (The majority agrees with me). Larry's reply was an intelligent, if snarky.

Posted by: Ivan at January 16, 2005 07:59 PM

Actually, I thought the system that was in place was fine. The first offense put the player into a drug abuse program but did not make the offense public. So if it was a false positive, there was no real harm. And the player had time to prove he was clean. Now, he'll get suspended and tarred as a cheater immediately.

And yes, I would prefer to see steroids legalized, but that's not going to happen soon. Therefore, I'd rather see a testing program that gives the benefit of the doubt to the players.

Posted by: David Pinto at January 16, 2005 09:54 PM

David, why would you like to see a medical substance legalized? While the harm is not known for long-term use, they certainly are not good for you. At best, they don't harm you that much.

I'm not sure how much they help either, especially in the way most people feel they do, giving you the strength to hit for more power. If they are a performance enhancer, I would guess they increase stamina more than anything else.

But, if they even help a little bit, it creates a playing field that is not level. If they cause even a small bit of harm, I'm glad they are now banned. Finally, I'm curious if you feel all drugs should be legal, or just steroids, and how you came to that conclusion. Looking forward to your reply, David.

Posted by: Al at January 16, 2005 11:30 PM

Well he** David, the criminal justice system sucks too. I mean there are people who are falsely convicted and the jury system is far from binary, it's "beyond the reasonable doubt" of twelve imperfect human beings, so in order to spare people from being falsely convicted let's just say to he** with the whole thing, and we'll let everybody get away with everything. Because it's an imperfect world after all.

How many NFL, NBA or Olympic athletes have had their lives ruined by a false positive? Really I'd like to know.

I'm all for libertarinaism, but you do need rules somewhere.

And Ivan, Al's argument was not that "the majority agrees with him." He said he'd go with the majority. but he never said that because it was the majority opinion made it the right one. But kudos for using "patently" it's so rare one gets the perfect opportunity to use that word.

Finally, most people (not most baseball bloggers but most fans of teams other than the Giants) think Barry Bonds is on the juice, and he hasn't returned a false positive. So testing policy or no, public opinion, which is really what David seems concerned about here, is going to make scapegoats out of someone testing or no testing. Publicized results or not.

Posted by: steve at January 17, 2005 02:46 AM

Why not let NFL players get jazzed on PCP before big games? Can you imagine the fear it would instill in a guy with the ball when he sees a horsed-up defender closing in on him? The hits would also be like near-death experiences. Awesome, dude.

Posted by: U and Sully at January 18, 2005 09:37 AM
Post a comment









Remember personal info?