Is this a good spam filter?

For every 100 emails, 98 are identified accurately, 2 are identified inaccurately.

For every 100 emails, 99 are normal, 1 is spam.

We can represent this with a chart (essentially one with $100 \times 100 = 10000$ entries):

The only emails we care about for this question are the emails identified as spam.

Out of $10000$ emails, normal mail is identified as spam $99 \times 2 = 198$ times.

Out of $10000$ emails, spam mail is identified as spam $98 \times 1 = 98$ times.

Since $198 > 98 ,$ it's more likely a spam-identified email is actually not spam!

Let $x$ be the total number of Rakesh's e-mails.

If $2\%$ of $99\%$ of Rakesh's good e-mails are erroneously marked as spam, the number of good e-mails in Rakesh's spam folder is $0.02 \cdot 0.99 \cdot x = 0.0198x$ .

If $98\%$ of $1\%$ of Rakesh's spam e-mails are correctly marked as spam, the number of spam in Rakesh's spam folder is $0.98 \cdot 0.01 \cdot x = 0.0098x$ .

Since $0.0198x > 0.0098x$ for all positive values of $x$ , there are more good e-mails than spam in Rakesh's spam folder.

Relevant wiki: Bayes' Theorem and Conditional Probability

From the question we know that the prior probability an email is spam, $p(\text{spam}) = 0.01$ .

Furthermore we know the accuracy on any email ( whether or not it is spam ) is 98%. Let $y = 1$ denote the prediction of an email of spam and $y = 0$ denote the prediction of an email as not spam. Then we can also say $p(y = 1|\text{spam}) = p(y = 0|\text{not spam}) = 0.98$ .

The degree of belief that an email is spam given the filter flags it as so is $p(\text{spam}|y=1)$ .

This can be written by Bayes' rule as $p(\text{spam}|y=1) = \frac{p(y = 1|\text{spam}) \cdot p(\text{spam})}{p(y = 1|\text{spam}) \cdot p(\text{spam}) + p(y = 1|\text{not spam}) \cdot p(\text{not spam})}$ .

$p(y = 1|\text{not spam}) = 1 - p(y = 0|\text{not spam}) = 0.02$ . Similarly $p(\text{not spam}) = 0.99$ .

Plugging in the numbers we get $\frac{0.98 \cdot 0.01}{0.98 \cdot 0.01 + 0.02 \cdot 0.99}\ \approx 0.332 \approx 33.2\%$ .

Given this is less than 50%, Rakesh should not believe the results with high confidence.

Relevant wiki: Bayes' Theorem and Conditional Probability

Suppose there are 10,000 emails.

9,900 are normal.

100 are spam.

Out of the 9,900 normal emails, 198 would be identified as spam, 9702 as normal.

Out of the 100 spam emails, 98 would be identified as spam, 2 as normal.

(0.01 * 0.98 = 0.0098) is spam and marked as spam

(0.01 * 0.02) is spam and marked as not-spam (false negative)

(0.99 * 0.98) is not-spam and is marked as not-spam

(0.99 * 0.02 = 0.0198) is not-spam and is marked as spam (false positive)

More likely to be not-spam (0.0198 > 0.0098)

For convenience, let us denote the events - the filter marking a mail as spam by $f = 0$ , the filter marking a mail as not spam by $f = 1$ , a mail actually being a spam by $s = 0$ and a mail actually being a non-spam mail by $s = 1$

We are told that $P(s = 0) = \dfrac{1}{100} \implies \overline{P}(s = 0) = P(s = 1) = \dfrac{99}{100}$

We are also told that the accuracy of Rakesh's spam filter is 98%.

From this, we deduce that $\begin{aligned}P(f=0|s=0)P(s=0) + P(f=1|s=1)P(s=1) &= \dfrac{98}{100} \\ \\ \implies P(f=0|s=0)P(s=0) + (1-P(f=0|s=1))P(s=1) &= \dfrac{98}{100} \\ \\ \implies P(f=0|s=0)P(s=0) - P(f=0|s=1)P(s=1) &= \dfrac{98}{100} - P(s=1) = \dfrac{-1}{100} \hspace{20mm} \text{equation }(1)\end{aligned}$

Notice that assuming $P(f=0|s=0) = P(f=1|s=1)$ is not warranted.

Now, we are to compare the following conditional probabilities (conditioned on the spam filter saying the mails a spam) $P(s=0 | f=0) \text{ and } P(s=1 | f=0)$

Using the Bayes Theorem, we see that $\begin{aligned} P_1 = P(s=0 | f=0) &= \dfrac{P(f=0|s=0)P(s=0)}{P(f=0)} \\ \\ \text{and similarly }P_2 = P(s=1| f=0) &= \dfrac{P(f=0|s=1)P(s=1)}{P(f=0)}\end{aligned}$

Consider the difference $D = P_1 - P_2$ .

$\begin{aligned}D &=\dfrac{P(f=0|s=0)P(s=0)}{P(f=0)} - \dfrac{P(f=0|s=1)P(s=1)}{P(f=0)} \\ \\ &= \dfrac{-1/100}{P(f=0)} \hspace{25mm} \text{from equation }(1)\\ \\ &< 0 \end{aligned}$

We therefore conclude - given that the spam filter says the mail is a spam, the probability that the mail is actually a spam is less than the probability that its actually a normal mail.

Here is another solution by drawing a tree diagram. If the mail is spam it goes into Junk folder. If the mail is not spam it goes into Rakesh's Inbox.

We know the following:

1. From all the messages Rakesh receives, 1% is spam and 99% is normal mail.
2. We also know that his spam filter is 98% accurate both ways. This means that the probability that a normal mail will be sent to Inbox is 98%, while the probability that a normal mail will be sent to Junk folder is 2%. Similarly, the probability that a spam mail will be sent to Junk folder is 98% and the probability that a spam mail will be sent to Inbox is 2%.

We can know draw the diagram:

As you can see from the diagram, the probability that a message is both spam and it ends up in Junk folder is 0.98% and the probability that a message is both normal and it ends up in Junk folder is 1.98%.

Therefore, if Rakesh gets an e-mail in his junk folder it is more probable that this message is actually normal mail mistakenly identified as spam than the message actually being spam.

Why is that so? Well, in this problem the spam filter is equally effective (or innefective) in identifying both spam and normal mail. However, Rakesh receives very little spam mail (only 1%). Just because 99% of his messages are normal, it is more likely that a normal mail will be flagged as spam than actually receiving a spam message.

Interestingly enough, if exactly 2% of all messages Rakesh receives are spam, than a message which ends up in junk folder is equally probable to be either spam or normal mail (1.96%). Try and calculate this for yourselves.

It's the paradox of the false negative. If the percentage of accuracy of the test is less than the probability that a perfect test would return negative, then the test returns more false positives than necessary. Lets say that there is a disease, called Math Deficiency Disease, that affects 6.9 percent of people. Now, scientists have made a way to test for MDD, which is correct 93 percent of the time. What is the percentage of people who had tests return positive who do not have MDD? A 1% B 0.1% C 51.8...% D 50.3...%

What drew me to doing this question was that in my math course, my professor did a question with us just like this but was looking at mammography results instead of spam emails.

We know that the spam detection is 98% accurate and that 1% of all the emails are actually spam

Let's assume there are 10 000 emails we assume that a true positive is correctly identifying spam email as spam and true negative is correctly identifying a true email as a true email

1% of emails are spam, therefore in his inbox there are: spam emails = 10 000x0.01 = 100 true emails = 10 000x0.99 = 9900

let's work with the spam first with 98% detection ability, of those 100 spam emails: 100x0.98= 98 will be correctly identified as spam (true positives) 100x0.02= 2 will be falsely identified as true emails (false negatives)

now let's work with the true emails with 98% detection ability, of those 9900 true emails: 9900x0.98= 9702 emails will be correctly identified as true emails (true negative) 9900x0.02= 198 emails will be falsely identified as spam emails (false positives)

And back to the original question, Which is more likely to be true if he receives an email that his filter predicts is spam?

well if he is told his email is spam and assuming there are 10 000 emails:

296 emails in total are identified as spam where: 98 of the emails identified as spam are actually spam 198 of the emails identified as spam are actually true emails

Therefore, the email is most likely NOT to be spam and is most likely to be a true email.

thanks for coming to my TedTalk.

P.S: shout out to my math prof for teaching me how to solve questions like this

I didn't even use math... I just thought about my friends and I. We send spam even in our IMPORTANT messages. (I broke the system, guys)

I had a spam filter with 99.5% accuracy, it marked everything as spam. Only 1/200 mails were non-spam; all non-spam were marked as spam. Yay for neural networks.

Two edgecases, out of 100 mail:

Case 1: 0 from the spam and 2 from the non-spam are incorrect.

Case 2: 1 from the spam and 1 from the non-spam are incorrect.

(Out of 100 mail, the possibility of 2 spam isn't there. SImplify.)

Case 1: All mail marked spam is non-spam (alternative 2)

Case 2: Half the mail marked spam is non-spam (not an alternative)

The answer is probably between case 1 and case 2 (alternative 2).

Intuitively, the overwhelming probability is that the email will not be spam, regardless of the result of the filter test. This can be shown with truth tables, as others have demonstrated.