Statistics are persuasive. So much so that people, organizations, and whole countries base some of their most important decisions on organized data. But there's a problem with that. Any set of statistics might have something lurking inside it, something that can turn the results completely upside down. For example, imagine you need to choose between two hospitals for an elderly relative's surgery. Out of each hospital's last 1000 patient's, 900 survived at Hospital A, while only 800 survived at Hospital B. So it looks like Hospital A is the better choice. But before you make your decision, remember that not all patients arrive at the hospital with the same level of health. And if we divide each hospital's last 1000 patients into those who arrived in good health and those who arrived in poor health, the picture starts to look very different. Hospital A had only 100 patients who arrived in poor health, of which 30 survived. But Hospital B had 400, and they were able to save 210. So Hospital B is the better choice for patients who arrive at hospital in poor health, with a survival rate of 52.5%. And what if your relative's health is good when she arrives at the hospital? Strangely enough, Hospital B is still the better choice, with a survival rate of over 98%. So how can Hospital A have a better overall survival rate if Hospital B has better survival rates for patients in each of the two groups? What we've stumbled upon is a case of Simpson's paradox, where the same set of data can appear to show opposite trends depending on how it's grouped. This often occurs when aggregated data hides a conditional variable, sometimes known as a lurking variable, which is a hidden additional factor that significantly influences results. Here, the hidden factor is the relative proportion of patients who arrive in good or poor health. Simpson's paradox isn't just a hypothetical scenario. It pops up from time to time in the real world, sometimes in important contexts. One study in the UK appeared to show that smokers had a higher survival rate than nonsmokers over a twenty-year time period. That is, until dividing the participants by age group showed that the nonsmokers were significantly older on average, and thus, more likely to die during the trial period, precisely because they were living longer in general. Here, the age groups are the lurking variable, and are vital to correctly interpret the data. In another example, an analysis of Florida's death penalty cases seemed to reveal no racial disparity in sentencing between black and white defendants convicted of murder. But dividing the cases by the race of the victim told a different story. In either situation, black defendants were more likely to be sentenced to death. The slightly higher overall sentencing rate for white defendants was due to the fact that cases with white victims were more likely to elicit a death sentence than cases where the victim was black, and most murders occurred between people of the same race. So how do we avoid falling for the paradox? Unfortunately, there's no one-size-fits-all answer. Data can be grouped and divided in any number of ways, and overall numbers may sometimes give a more accurate picture than data divided into misleading or arbitrary categories. All we can do is carefully study the actual situations the statistics describe and consider whether lurking variables may be present. Otherwise, we leave ourselves vulnerable to those who would use data to manipulate others and promote their own agendas.
Statistika je uverljiva, toliko da ljudi, organizacije i čitave države zasnivaju neke od svojih najvažnijih odluka na organzovanim podacima. Međutim, tu imamo problem. Svaki statistički skup može da ima nešto skriveno u sebi, nešto što može u potpunosti da preokrene rezultate. Na primer, zamislite da morate da izaberete između dve bolnice zbog operacije starijeg rođaka. Od poslednjih 1000 pacijenata iz svake bolnice, u bolnici A je preživelo 900, dok je u bolnici B preživelo svega 800. Pa se čini da je bolnica A bolji izbor. No, pre nego što se odlučite, zapamtite da svi pacijenti ne stižu u bolnicu istog zdravstvenog stanja. A ako podelimo poslednjih 1000 pacijenata iz svake bolnice na one koji su stigli dobrog zdravlja i one koji su stigli lošeg zdravlja, slika počinje da izgleda veoma drugačije. Bolnica A je imala samo 100 pacijenata koji su stigli lošeg zdravlja, od kojih je 30 preživelo. Međutim, bolnica B je imala 400 takvih i uspeli su da spase 210. Pa je bolnica B bolji izbor za pacijente koji stižu u bolnicu lošeg zdravlja, sa stopom preživelih od 52,5%. A šta ako je zdravlje vašeg rođaka dobro kad stigne u bolnicu? Zvuči čudno, ali bolnica B je i dalje bolji izbor, sa stopom preživelih preko 98%. Pa, kako može bolnica A da ima bolju ukupnu stopu preživelih, ako bolnica B ima bolje stope preživelih u obe grupe pacijenata? Ono na šta smo nabasali je slučaj Simpsonovog paradoksa, gde ista grupa podataka može da pokaže suprotne trendove, u zavisnosti od toga kako su grupisani. Ovo se često dešava kad skup podataka skriva uslovnu varijablu, koju ponekad zovu skrivenom varijablom, a to je skriveni dodatni faktor koji značajno utiče na rezultate. Ovde je skriveni faktor, relativna srazmera pacijenata koji stižu dobrog ili lošeg zdravlja. Simpsonov paradoks nije prosto hipotetičan scenario. S vremena na vreme se pojavljuje u stvarnom svetu, ponekad u bitnim kontekstima. Jedno istraživanje u Britaniji je pokazalo da pušači imaju veću stopu preživelih od nepušača tokom perioda od 20 godina. Sve dok učesnici u istraživanju nisu podeljeni po starosnim grupama, tada se pokazalo da su nepušači u proseku značajno stariji i stoga je bila veća verovatnoća da će da umru tokom istraživanja, baš zbog toga što su inače živeli duže. Ovde su starosne grupe skrivena varijabla i od suštinskog su značaja za pravilno tumačenje podataka. U drugom primeru, analiza slučajeva smrtne kazne u Floridi nije se činilo da otkriva rasnu nejednakost kod presuda između crnih i belih prestupnika osuđenih na smrt. Međutim, podela slučajeva prema rasi žrtve, govorila je nešto drugo. U oba slučaja, crni prestupnici su češće osuđivani na smrt. Sveukupno nešto veća stopa osuđenih belih prestupnika je bila posledica činjenice da slučajevi sa belim žrtvama češće uzrokuju smrtnu kaznu od slučajeva gde je žrtva crnac, a većina ubistava se dešavala među ljudima iste rase. Pa, kako da izbegnemo podleganje ovom paradoksu? Nažalost, ne postoji univerzalno rešenje. Podaci se mogu grupisati i podeliti na bezbroj načina, a sveukupne cifre mogu ponekad da daju tačniju sliku od podataka podeljenih u varljive ili proizvoljne kategorije. Sve što možemo da učinimo je da izučavamo stvarne situacije koje statistika opisuje i da pazimo na prisustvo skrivenih varijabli. U suprotnom, podložni smo uticaju onih koji će da iskoriste podatke kako bi manipulisali drugima i promovisali sopstvene ciljeve.