A few months ago we looked at a paper that purported to show that watching TV causes autism. Well, that paper has finally been peer reviewed, and therefore published. It's…

From Sense about Science comes Celebrities and Science Review 2008. This article in The Independent explains it in more detail (originally linked on Instapundit). The Sense about Science report looks…

This is just a rough prototype meant to be easy to play with inside a post. READ the help and guidebook! Suggestions for new canned examples welcome—the hard part is deriving historical performance data.

Rules

1. Read the Decision Calculator guidebook below!
2. Fill in the Performance Table, or click on one of the predefined examples.
3. Fill in the Cost Comparison Table, or click on one of the predefined examples. You do not need to calculate the total: that’s done automatically.
4. Click Calculate (or Reset between examples).
5. Accuracy comparison rates are given between the Expert System and the Naive Guess.
6. Cost results are found in the Expected Cost Comparison Table.
7. Finally, a solution saying which option you should choose is given. Skill should be > 0!
8. Important: Use this software at your own risk. No warranties of any kind are given or implied. Always consult a competent medical professional. .

Preset examples:

See Below	HELP	HELP	HELP

 

1. Fill in the Historical Performance Table.

	Present	Absent
Test +
Test –

 

2. Fill in the Cost Comparison Table.

False Positive Costs	Score	False Negative Costs	Score
Total:		Total:

 

3. Click calculate (or Reset between examples).

 

4. The Optimal Naive Guess is to:

 

5. Accuracy (%) Comparison Table

Test	Accuracy
Expert System
Naive Guess

 

6. Expected Costs Comparison Table

Test	Expected False Positive Cost	Expected False Negative Cost	Total
Expert System
Naive Guess

 

7. The skill score is:

It should be greater than 0 for a skillful test!

 

8. The solution:

 

GUIDEBOOK

This article provides you with an introduction and a step-by-step guide of how to make good decisions in particular situations. These techniques are invaluable whether you are an individual or a business.

These results hold for all manner of examples—from deciding whether to have a PSA test or mammography, to get a vaccine, to finding a good stock broker or movie reviewer, to situations that require intense statistical modeling, to financial forecasts, to lie detector usefulness. Any situation that has a dichotomous outcome can use these techniques.

Many people opt for precautionary medical tests—frequently because a television commercial or magazine article scares them into it. What people don’t realize is that these tests have hidden costs. These costs are there because tests are never 100% accurate. So how can you tell when you should take a test?

When is worth it?

Under what circumstances is it best for you to receive a medical test? When you “Just want to be safe”? When you feel, “Why not? What’s the harm?”

In fact, these are not good reasons to undergo a medical test. You should only take a test if you know that it’s going to give you useful information. You want to know the test performs well and that it makes few mistakes, mistakes which could end up costing you emotionally, financially, and even physically.

Let’s illustrate this by taking the example of a healthy woman deciding whether or not to have a mammogram to screen for breast cancer. She read that all women over 40 should have this test “Just to be sure.” She has heard lots of horror stories about breast cancer. Testing almost seems like a duty. She doesn’t have any symptoms of breast cancer and is in good health. What should she do?

What can happen when she takes this (or any) medical test? One of four things:

1. The test could correctly indicate that no cancer is present. This is good. The patient is assured.
2. The test could correctly indicate that a true cancer is present. This is good in the sense that treatment options can be investigated immediately.
3. The test could falsely indicate no cancer is present when it truly is. This error is called a false negative. This is bad because it could lead to false hope and could cause the patient to ignore symptoms because, “The test said I was fine.”
4. The test could falsely indicate that cancer is present when it truly is not. This error is called a false positive. This is bad because it is distressing and could lead to unnecessary and even harmful treatment. The test itself, because it uses radiation, even increases the risk of true cancer because of the unnecessary exposure to x-rays.

This table shows all the possibilities in a test for the presence of absence of a thing (like breast cancer, prostate cancer, a lie, AIDS, and so on). For mammograms, “Present” means that cancer is actually there, and “Absent” means that no cancer is there. For a PSA test, “Present” means a prostate cancer is actually there, and “Absent” means that it is not. For a Movie Reviewer, “Present” means you liked a movie, and “Absent” means you did not.

Test Table

	Present	Absent
Test +	Good: True Positive	Bad: False Positive
Test –	Bad: False Negative	Good: True Negative

“Test +” says that the test indicates the test said the thing (cancer) is present. “Test -” says that the test indicates the absence of the thing. For the Movie Reviewer example, “Test +” means the reviewer recommended a film.

There are two cells in this graph that are labeled “Good,” meaning the test has performed correctly. The other two cells are labeled “Bad,” meaning the test has erred. Study this table to be sure you understand how to read it because it will be used throughout this article.

 

Error everywhere

The main point is this: all tests and all measurements have some error. There is no such thing as a perfect test or perfect measurement! Mistakes always happen. This is an immutable law of the universe. Some tests are better than others, and tables like this are necessary to understand how to rate how well a particular test performs.