Posted inCulture Statistics

Disparity is inevitable: a counter argument to filing discrimination lawsuits

Introduction

Know a lawyer who is involved in a discrimination lawsuit? Particularly one in which the plaintiff alleges discrimination because actual disparities are found in company hiring practices? Were you aware that, just by chance, a company can be absolutely innocent of discrimination even though they actually are found to have under-hired a particular group? No? Then read on to find out how.

What are diversity and disparity?

We discussed earlier that there are (at least) two definitions of diversity: one meaning a display of dissimilar and widely varying behaviors, a philosophical position that is untenable and even ridiculous (but strangely widely desired). The second meaning is our topic today.

Diversity of the second type means parity in the following sense. Suppose men and women apply in equal numbers and have identical abilities to perform a certain job. Then suppose that a company institutes a hiring policy that results in 70% women and 30% men. It can be claimed that that company does not properly express diversity, or we might say a disparity in hiring exists. Diversity thus sometimes means obtaining parity.

Disparity is an extraordinarily popular academic topic, incidentally: scores of professors scour data to find disparities and bring them to light. Others—lawyers—notice them and, with EEOC regulations in hand that call such disparities illegal, sue.

And it’s natural, is it not, to get your dudgeon up when you see a statistic like “70% women and 30% men hired”? That has to be the result of discrimination!

Of course, it was in the past routinely true that some companies unfairly discriminated against individuals in matters that had nothing to do with their ability. Race and sex were certainly, and stupidly, among these unnecessarily examined characteristics. Again, it’s true that some companies still exhibit these irrational biases. For example, Hollywood apparently won’t hire anybody over the age of 35 to write screenplays, nor will they employ actors with IQs greater than average.

Sue ’em!

It’s lawsuits that interest us. How unusual is a statistic like “70% women and 30% men hired”? Should a man denied employment at that company sue claiming he was unfairly discriminated against? Would we expect that all companies that do not discriminate would have exactly 50% women and 50% men? This is a topic that starts out easy but gets complicated fast, so let’s take our time. We won’t be able to investigate this topic fully given that it would run to a monograph-length document. But we will be able to sketch an outline of how the problem can be attacked.

Parity depends on several things: the number of categories (men vs. women, black vs. white, black men vs. black women vs. white men vs. white women, etc.; the more subdivisions that are represented, the more categories we have to track), the proportion those categories exist in the applicant population (roughly 51% men, 49% women at job ages in the USA; we only care about the characteristics of those who apply to a job and not their rates in the population), the exact definition of parity, the number of employees the company has, and the number of companies hiring. That last one is the one everybody forgets and is the one that makes disparities inevitable. Let’s see why.

Men vs. Women

Throughout all examples we assume that companies hire blindly, that they have no idea of the category of its applicants, that all applicants and eventual hires are equally skilled; that is, that there is no discrimination in place whatsoever, but also that there is no quota system in place either. All hires are found randomly. Thus, any eventual ratio of observed categories in a company is the result of chance only, and not due to discrimination of any kind (except on ability). This is crucial to remember.

First suppose that there are in our population of applicants 51% men and 49% women.

Now suppose a company hires just one employee. What is the probability that that company will attain parity? Zero. There is (I hope this is obvious) no way the company can hire equal numbers of men and women, even with a quota system in place. Company size, then, strongly determines whether parity is possible.

To see this, suppose the company can hire two employees. What is the probability of parity? Well, what can happen: a man is hired first followed by another man, a man then a woman, a woman then a man, or a woman followed by another woman. The first and last cases represent disparity, so we need to calculate the probability of them occurring by chance. It’s just slightly over 50%.

(Incidentally, we do need to consider cases where men are discriminated against: in the past, we could just focus on cases where women were, but in the modern age of rampant tort lawyers, we have to consider all kinds of disparity lawsuits. For example, the New York Post of 12 May 2009, p. 21, writes of a a self-identified “white, African, American” student from Mozambique who is suing a New Jersey medical school for discrimination.)

Now, if a woman saw that there were two men hired, she might be inclined to sue the company for discrimination, but it’s unlikely. Why? Because most understand that with only two employees, the chance for seeming, or false discrimination is high; that is, disparity resulting by chance is pretty likely (in fact, 50%).

So let’s increase the size of our company to 1000 employees. Exact parity would give us 510 men and 490 women, right? But the probability of exact parity—given random hiring—is only 2.5%! And the larger the company the less it is likely exact parity can be reached.

Posted inStatistics

Lazy Sunday links

As I try to track down the RSS problem, here's some things to read... * Longtime reader Bernie Cullen recommends this analysis over at Pielke Jr's place on those Australian…
Posted inStatistics

Wishcasting the McCain-Obama presidential election

Update: 2 March 2009. I am withdrawing this post, in the sense that I no longer think it is strictly accurate. See this post to see why. But I want to leave this post up for others to see how not to do statistics.

Background

Right after the close of the Republican and Democrat conventions (in 2008), I asked readers to participate in a study where they could guess who would win the presidential election. They could also indicate who they wanted to win, and if they ordinarily skewed Conservative or Liberal.

The idea was to search for traces of wishcasting, which is what happens when people let their desires influence their judgment about what will happen.

A wishcasting sales manager might suppose his will make a higher sales number next month than he probably will because he wants to save his job. A climate activist might forecast a higher probability of doom because he wants mankind to be responsible for various real and imagined ills. A sports fan will similarly put too much weight on a victory for his favorite team.

Or a voter might guess too high a probability for his candidate’s victory given that he desires him to win.

It is important to recognize wishcasting traits in yourself—or in your company—so that you can reduce or eliminate them and thus produce more accurate, and thus more valuable, forecasts. Recognizing wishcasting in your forecasts is the first step to making better predictions.

Results

Please remember that these guesses were made right after the conventions, well before the end-game of the campaign. At that point, the information on the candidates was roughly equal. Both media reports and personal conversations indicated that both candidates had a chance to win. That would obviously change later in the campaign, but a crude parity was in place after the conventions.

We received 624 legitimate responses (see the Data Caveats section for more details). 79% thought McCain would win, 21% Obama. Obama won 53% of the popular vote, McCain 46%. Given the information they had at the time, voters guessed too high for McCain and too low for Obama.

Not surprisingly, given the nature of this blog, 83% wanted McCain to win, 15% wanted Obama, and 2% chose not say. This was the same breakdown for philosophy: 83% reported Conservative, 15% Liberal, and 2% unknown.

80% of the participants were men. The median age was 51 (same for men and women), with a range between 16 and 89.

The tricky part of this analysis is that it is impossible to say exactly how much wishcasting was done. It might be true that each of the respondents did not let their desires influence their guess at all. Given what we know of human nature, this is probably false, but we cannot say with certainty. Everything below, therefore, is just an estimate.

We also cannot say about the level of wishcasting in any person: all estimates can only be for the “average” voter, that is, the type of voter who would take part in an internet study like this. If we had repeated guesses from one person, then we could hone in on his wishcasting component, but we only have one guess per person, so we can only say something about the group.

This table gives a first indication. It estimates the probability of who wins given who you wanted to win:

Who win?
McCain Obama
Want win?    McCain 89% 11%
Obama 25% 75%
None 67% 33%

Of those who wanted McCain to win, 89% thought he would win, and only 11% thought Obama would. Likewise, of those who wanted Obama to win, 75% thought he would. Of the small number who didn’t express a desire, two-thirds thought McCain would win. That 89% and 75% are pretty big and indicate that some kind of wishcasting has taken place. Why?

If there were no wishcasting, and assuming the 600-some people’s actual votes did not influence the election unduly (presumably everybody who wanted a candidate to win voted for him), we would expect that who you wanted to win would have no bearing on who you thought would win. People should be able to separate their desire from their judgment.

Since overall 79% thought McCain would win, if no wishcasting took place, we would expect that 79% of those who wanted McCain to win to say he would. But 89% did. That 10-percentage point difference is the amount of wishcasting that took place among McCain supporters.

Since overall 21% thought Obama would win, then if no wishcasting took place, we would expect that 21% of those who wanted Obama to say he would. But 75% did. Thus, the 54-percentage point difference is the amount of wishcasting among his supporters.

For the people who didn’t say who they wanted, we would have expected a 50/50 split, but we saw a slight leaning towards McCain, 66/34. We shouldn’t read too much into this, as this was only for 9 people.

First finding

In other words, Obama supporters, as a group, wishcasted at a rate five times higher than McCain supporters. This result is not now surprising, given how the election played out, particularly in the media.

We cannot dismiss the element of doomcasting, either. This is when people guess what they don’t want to happen. Doomcasters have a counter-balancing effect on wishcasters, pulling the data back to where we would expect it had people not let their desires influence their guesses in a positive direction. Since we see a wishcasting effect, we cannot reliably estimate any doomcasting effect from the above table.

But we can say more. The next tables are just like the first, but broken up by those who identified themselves as Conservative or Liberal.

Estimated probability of who wins given who you want to win by philosophy:

  Conservative          Liberal
Who win?        Who win?
McCain Obama        McCain Obama
Want win?    McCain 89% 11%        Want win?    McCain 89% 11%
Obama 35% 65%        Obama 23% 77%

For Conservatives, the amount of wishcasting for McCain was the same: a 10-percentage point bias. But for Obama supporters, the wishcasting fell 10-percentage points to only a 44-percentage point bias.

For Liberals, the amount of wishcasting for McCain was the same. But for Obama supporters, it increased a slight amount, to a 56-percentage point bias.

There were not enough people who did not specify a desired candidate to break down the numbers into philosophical groups.

Second finding

In other words, Liberal Obama supporters wishcasted at a rate higher than Conservative Obama supporters. The amount of McCain wishcasting remained the same regardless of philosophy, indicating more consistency. These results are also not surprising.

Third finding

If we break the data down by age into two groups (tables not shown): greater or lesser than the median 51 years, we find the McCain wishcasting bias remains the same, but for the older Obama supporters it goes up to a 65-percentage point bias. This was somewhat surprising given that we heard during the campaign that younger people who preferred Obama were more zealous. The younger group had just about the same percentage point bias as before (49 points).

Fourth finding

Among males, the McCain wishcasting bias stayed the same. But among females it increased just slightly to 13-percentage points. The amount of Obama wishcasting bias was the same for both males and females.

There was not enough data to reliably break the data down any further: for example, age by philosophy, sex by age, and so on.

Conclusion

Wishcasting almost surely took place in the McCain-Obama presidential election. This conclusion is conditional on the poll giving usable data (as to that, see the next section).

We must remember that these results are relevant for people who would come to a blog like this, during the specified time, and for elections like the one we had. At the least, we have said something about our small community; at the most, we have said something about average web browsers. We have probably not said much about the general population.

We should understand that any given Obama voter should not have necessarily subtracted 50-percentage points from his guess that Obama would win. That 50-percentage point bias was for the group and not necessarily for any individual. The implies that some people should have subtracted more, some less. The only way to say something about an individual is to collect more data on that individual so that we can estimate his typical bias.

Who wishcasted? Well, your author took part in the study: I thought McCain would win and also wanted him to. Wishcasting was probably there to some extent.

Other McCain supporters wishcasted, too, but not by very much on average. It didn’t seem to matter if they were young or old, male or female, or whether they identified themselves as Conservative or Liberal, the amount of McCain bias was about the same. Again, this was characteristic of readers of this blog, so we should be careful to say the same is true for all McCain supporters.

Those who wanted Obama to win really let that desire influence their guesses. Liberal Obama supporters wishcasted the most on average, a 56-percentage point bias; those who listed themselves as Conservative were more temperate, but not as tempered as McCain supporters. Sex didn’t make any difference to the results, but age did: older Obama supporters wishcasted more than younger ones did, a finding that goes against conventional wisdom. Once more, this result is for readers of this blog, or people like them.

Suppose you are, or were, an Obama supporter and you say, “So what if I say I wanted him to win. He did win, didn’t he? What does wishcasting have to do with anything? I made the right guess.” Yes, you did. This time.

The result—at the time of the conventions—was by no means a forgone conclusion. McCain might have won. You were making a guess about an uncertain future. You got it right this time, but you might not get it right next time. In making predictions of this type, if you are like the typical Obama supporter in this study, you are letting your desires influence your judgment too much, and over the course of many of guesses you will make more mistakes than the folks who do not wishcast. You’ll be losing either money, or prestige, or something.

There are probably plenty of formal studies, of which I am not aware, that show that the more controversial the matter the more people let their desires influence their predictions of the future. These findings do support the anecdotal perception that Obama supporters were more emotional than were McCain voters. It’s not that McCain supporters did not let their feelings influence them—again this data can say nothing about any individual, just the average behavior about a group of persons—but that influence was not sharp.

Obama supporters were, as everybody knows, more passionate (particularly read the next section). They let this passion sway them more. It worked out for them this time, but that passion can easily work against them the next time they make a prediction.

————————————————-
Data Caveats