I write fiction sci-fi thrillers, so if I say "killer robots," you'd probably think something like this. But I'm actually not here to talk about fiction. I'm here to talk about very real killer robots, autonomous combat drones.
Ja pišem naučno fantastične trilere i ako kažem "roboti ubice" verovatno ćete pomisliti na ovako nešto. Ali nisam ovde da bih pričao o fikciji. Ovde sam da bih pričao o vrlo stvarnim robotima ubicama, autonomnim bespilotnim letelicama za borbu.
Now, I'm not referring to Predator and Reaper drones, which have a human making targeting decisions. I'm talking about fully autonomous robotic weapons that make lethal decisions about human beings all on their own. There's actually a technical term for this: lethal autonomy.
Ne mislim pritom na bespilotne letelice "Predator i "Riper" gde odluke o ciljanju donose ljudi. Govorim o potpuno nezavisnom robotskom oružju koje donosi odluke o smrti ljudskih bića potpuno samostalno. Postoji i tehnički izraz za to: smrtonosna autonomija.
Now, lethally autonomous killer robots would take many forms -- flying, driving, or just lying in wait. And actually, they're very quickly becoming a reality. These are two automatic sniper stations currently deployed in the DMZ between North and South Korea. Both of these machines are capable of automatically identifying a human target and firing on it, the one on the left at a distance of over a kilometer. Now, in both cases, there's still a human in the loop to make that lethal firing decision, but it's not a technological requirement. It's a choice. And it's that choice that I want to focus on, because as we migrate lethal decision-making from humans to software, we risk not only taking the humanity out of war, but also changing our social landscape entirely, far from the battlefield. That's because the way humans resolve conflict shapes our social landscape. And this has always been the case, throughout history.
Smrtonosno autonomni roboti ubice mogu imati više oblika - leteći, na točkovima ili samo u zasedi. U stvari, oni vrlo brzo postaju realnost. Ovo su dve automatske snajperske stanice trenutno postavljene u demilitarizovanoj zoni između Severne i Južne Koreje. Obe ove mašine su sposobne za automatsko identifikovanje ljudskih meta i pucanje na njih, ona s leve strane, na razdaljini od preko jednog kilometra. U oba slučaja još uvek je čovek uključen da donese odluku o smrtonosnoj paljbi, ali to nije zahtev tehnologije. To je izbor. I taj izbor je ono na šta želim da se fokusiram, jer kako prenosimo donošenje smrtonosne odluke sa ljudi na softver, rizikujemo ne samo to da izbacimo ljudskost iz rata, nego i da potpuno promenimo sliku našeg društva daleko od bojnog polja. To je zbog toga što način na koji ljudi razrešavaju sukob oblikuje sliku našeg društva. Ovo je uvek bio slučaj tokom istorije.
For example, these were state-of-the-art weapons systems in 1400 A.D. Now they were both very expensive to build and maintain, but with these you could dominate the populace, and the distribution of political power in feudal society reflected that. Power was focused at the very top. And what changed? Technological innovation. Gunpowder, cannon. And pretty soon, armor and castles were obsolete, and it mattered less who you brought to the battlefield versus how many people you brought to the battlefield. And as armies grew in size, the nation-state arose as a political and logistical requirement of defense. And as leaders had to rely on more of their populace, they began to share power. Representative government began to form.
Na primer, ovo su bili moderni sistemi naoružanja 1400. godine. Oba su bila skupa za izgradnju i održavanje, ali njima ste mogli da vladate narodom i to se ogledalo u raspodeli političke moći u feudalnom društvu. Moć je bila koncentrisana u samom vrhu. Šta se promenilo? Tehnološka inovacija. Barut, top. I uskoro, oklopi i tvrđave su postali beskorisni i bilo je manje važno koga ste doveli na bojno polje od toga koliko ste ljudi doveli na bojno polje. I kako su vojske rasle, nastajala je nacionalna država kao politički i logistički zahtev odbrane. I kako su lideri morali da se oslanjaju više na svoj narod, počeli su da dele moć. Počela je da se formira vlada.
So again, the tools we use to resolve conflict shape our social landscape. Autonomous robotic weapons are such a tool, except that, by requiring very few people to go to war, they risk re-centralizing power into very few hands, possibly reversing a five-century trend toward democracy.
I opet, sredstva koja koristimo da razrešimo sukob oblikuju sliku našeg društva. Autonomna robotička oružja su takvo sredstvo, koje osim što zahteva da vrlo malo ljudi pođe u rat, rizikuje ponovno centralizovanje moći u mali broj ruku što bi verovatno obrnulo petovekovni trend ka demokratiji.
Now, I think, knowing this, we can take decisive steps to preserve our democratic institutions, to do what humans do best, which is adapt. But time is a factor. Seventy nations are developing remotely-piloted combat drones of their own, and as you'll see, remotely-piloted combat drones are the precursors to autonomous robotic weapons. That's because once you've deployed remotely-piloted drones, there are three powerful factors pushing decision-making away from humans and on to the weapon platform itself.
Znajući ovo, mislim da možemo preduzeti odlučne korake da sačuvamo naše demokratske institucije, da uradimo ono što ljudi rade najbolje, a to je da se adaptiramo. Ali vreme je faktor. Sedamdeset nacija razvija sopstvene daljinski upravljane bespilotne letelice i kao što ćete videti, daljinski upravljane bespilotne letelice su preteče autonomnih robotskih letelica. To je zato što, kad ste razmestili daljinske bespilotne letelice, postoje tri moćna faktora koji podstiču premeštanje donošenja odluke sa ljudi na sama oružja.
The first of these is the deluge of video that drones produce. For example, in 2004, the U.S. drone fleet produced a grand total of 71 hours of video surveillance for analysis. By 2011, this had gone up to 300,000 hours, outstripping human ability to review it all, but even that number is about to go up drastically. The Pentagon's Gorgon Stare and Argus programs will put up to 65 independently operated camera eyes on each drone platform, and this would vastly outstrip human ability to review it. And that means visual intelligence software will need to scan it for items of interest. And that means very soon drones will tell humans what to look at, not the other way around.
Prvi od njih je poplava video snimaka koje prave bespilotne letelice. Na primer 2004., flota bespilotnih letelica SAD-a je proizvela ukupno 71 sat video nadzora za analizu. Do 2011., taj broj se popeo na 300.000 sati, prevazilazeći ljudsku sposobnost da sve to pregleda i taj broj će se drastično povećavati. Programi Pentagona "Gorgonin pogled" i "Argus" će postaviti do 65 nezavisnih objektiva kamere na svaku bespilotnu letelicu i ovo će uveliko prevazići ljudsku sposobnost da sve to pregleda. To znači da će vizuelno inteligentni softver morati da to skenira da bi izdvojio ono što je značajno. A to znači da će uskoro bespilotne letelice govoriti ljudima šta da pogledaju, a ne obratno.
But there's a second powerful incentive pushing decision-making away from humans and onto machines, and that's electromagnetic jamming, severing the connection between the drone and its operator. Now we saw an example of this in 2011 when an American RQ-170 Sentinel drone got a bit confused over Iran due to a GPS spoofing attack, but any remotely-piloted drone is susceptible to this type of attack, and that means drones will have to shoulder more decision-making. They'll know their mission objective, and they'll react to new circumstances without human guidance. They'll ignore external radio signals and send very few of their own.
Ali tu je i drugi moćni podsticaj za prenošenje donošenja odluka sa ljudi na mašine i to je elektromagnetno ometanje koje prekida vezu između bespilotne letelice i onoga ko njom upravlja. Videli smo primer ovoga 2011. kada se američka bespilotna letelica RQ-170 "Sentinel" malo zbunila iznad Irana usled napada pomoću GPS obmane, ali bilo koja daljinska bespilotna letelica bila bi podložna ovom tipu napada, a to znači da će bespilotne letelice morati u većoj meri da preuzmu donošenje odluke. One će znati cilj svoje misije i reagovaće na nove okolnosti bez ljudskog vođstva. One će ignorisati spoljne radio signale i slati vrlo malo sopstvenih.
Which brings us to, really, the third and most powerful incentive pushing decision-making away from humans and onto weapons: plausible deniability. Now we live in a global economy. High-tech manufacturing is occurring on most continents. Cyber espionage is spiriting away advanced designs to parts unknown, and in that environment, it is very likely that a successful drone design will be knocked off in contract factories, proliferate in the gray market. And in that situation, sifting through the wreckage of a suicide drone attack, it will be very difficult to say who sent that weapon.
I to nas dovodi do trećeg i najmoćnijeg podsticaja za prenošenje donošenja odluka sa ljudi na oružje: uverljivo negiranje. Živimo u globalnoj ekonomiji. Proizvodnja visoke tehnologije se dešava na uglavnom svim kontinentima. Sajber špijunaža krijumčari napredne dizajne na nepoznate destinacije i u takvom okruženju je vrlo verovatno da će uspešan dizajn bespilotne letelice stati u fabrikama i umnožiti se na crnom tržištu. I u toj situaciji, pretraživanjem olupine posle samoubilačkog napada bespilotne letelice, biće vrlo teško reći ko je poslao to oružje.
This raises the very real possibility of anonymous war. This could tilt the geopolitical balance on its head, make it very difficult for a nation to turn its firepower against an attacker, and that could shift the balance in the 21st century away from defense and toward offense. It could make military action a viable option not just for small nations, but criminal organizations, private enterprise, even powerful individuals. It could create a landscape of rival warlords undermining rule of law and civil society. Now if responsibility and transparency are two of the cornerstones of representative government, autonomous robotic weapons could undermine both.
Ovo predstavlja realnu mogućnost anonimnog rata. Ovo bi moglo okrenuti geopolitičku ravnotežu naglavce i vrlo otežati naciji da upotrebi svoju oružanu moć protiv napadača i to može da promeni ravnotežu u 21. veku sa odbrane na napad. Može da učini vojnu akciju izvodljivom opcijom ne samo za male narode, nego i za kriminalne organizacije, privatne poduhvate, pa čak i za moćne pojedince. Može da kreira sliku u kojoj rivalski gospodari rata podrivaju zakon prava i civilnog društva. Ako su odgovornost i transparentnost dva stuba vlade, autonomna robotska oružja bi mogla da ih potkopaju.
Now you might be thinking that citizens of high-tech nations would have the advantage in any robotic war, that citizens of those nations would be less vulnerable, particularly against developing nations. But I think the truth is the exact opposite. I think citizens of high-tech societies are more vulnerable to robotic weapons, and the reason can be summed up in one word: data. Data powers high-tech societies. Cell phone geolocation, telecom metadata, social media, email, text, financial transaction data, transportation data, it's a wealth of real-time data on the movements and social interactions of people. In short, we are more visible to machines than any people in history, and this perfectly suits the targeting needs of autonomous weapons.
Možda mislite da bi građani visoko tehnoloških nacija imali prednost u bilo kom robotskom ratu, da bi građani ovih nacija bili manje ranjivi, posebno u poređenju sa nacijama u razvoju. Ali mislim da je istina potpuno suprotna. Mislim da su građani visoko tehnoloških društava ranjiviji za robotska oružja i razlog za to može da se sumira u jednoj reči: podaci. Podaci vladaju visoko tehnološkim društvima. Geolokacija mobilnih telefona, telekomunikacioni metapodaci, društveni mediji, e-mail, SMS, podaci finansijskih transakcija saobraćajni podaci, to je bogatstvo podataka u realnom vremenu o potezima i društvenim interakcijama ljudi. Ukratko, vidljiviji smo mašinama od bilo kojih drugih ljudi u istoriji i ovo savršeno odgovara ciljanim potrebama autonomnog oružja.
What you're looking at here is a link analysis map of a social group. Lines indicate social connectedness between individuals. And these types of maps can be automatically generated based on the data trail modern people leave behind. Now it's typically used to market goods and services to targeted demographics, but it's a dual-use technology, because targeting is used in another context. Notice that certain individuals are highlighted. These are the hubs of social networks. These are organizers, opinion-makers, leaders, and these people also can be automatically identified from their communication patterns. Now, if you're a marketer, you might then target them with product samples, try to spread your brand through their social group. But if you're a repressive government searching for political enemies, you might instead remove them, eliminate them, disrupt their social group, and those who remain behind lose social cohesion and organization. Now in a world of cheap, proliferating robotic weapons, borders would offer very little protection to critics of distant governments or trans-national criminal organizations. Popular movements agitating for change could be detected early and their leaders eliminated before their ideas achieve critical mass. And ideas achieving critical mass is what political activism in popular government is all about. Anonymous lethal weapons could make lethal action an easy choice for all sorts of competing interests. And this would put a chill on free speech and popular political action, the very heart of democracy.
Ono što vidite ovde je mapa analize veza društvene grupe. Linije označavaju društvenu povezanost između osoba. Ova vrsta mapa može se generisati automatski na osnovu tragova podataka koje moderni ljudi ostavljaju za sobom. Obično se koristi u tržištu robe i usluga za ciljanu demografiju, ali to je tehnologija za obe strane jer je "ciljanje" upotrebljeno u drugom kontekstu. Vidite da su određene osobe istaknute. Reč je o centrima društvenih mreža. To su organizatori, oni utiču na formiranje mišljenja, vođe i ti ljudi takođe mogu biti automatski identifikovani pomoću svojih obrazaca komunikacije. Ako ste prodavac, onda ih možete naciljati pomoću uzoraka proizvoda i pokušati da proširite svoj brend u njihovoj društvenoj grupi. Ali, ako ste represivna vlada koja traži političke protivnike, mogli biste umesto toga da ih uklonite, eliminišete, poremetite njihovu društvenu grupu i one koji stoje u labavoj društvenoj koheziji i organizaciji. U svetu jeftinih robotskih oružja koja se sve više umnožavaju granice bi pružale vrlo malu zaštitu kritičarima udaljenih vlada ili međunarodnim kriminalnim organizacijama. Narodni pokreti koji agituju za promenu mogli bi biti otkriveni rano i njihovi lideri eliminisani pre neko što njihove ideje usvoji kritična masa. A ideje koje dostižu kritičnu masu su ono čime se politički aktivizam u narodnoj vladi bavi. Anonimna smrtonosna oružja mogu preduzeti smrtonosnu akciju, lak izbor za sve vrste takmičarskih interesa. I ovo bi stavilo tačku na slobodu govora i političku aktivnost naroda, samu srž demokratije.
And this is why we need an international treaty on robotic weapons, and in particular a global ban on the development and deployment of killer robots. Now we already have international treaties on nuclear and biological weapons, and, while imperfect, these have largely worked. But robotic weapons might be every bit as dangerous, because they will almost certainly be used, and they would also be corrosive to our democratic institutions.
To je ono zbog čega nam treba međunarodni sporazum o robotskom oružju i posebno globalna zabrana razvoja i razmeštanja robota ubica. Već imamo međunarodne sporazume o nuklearnom i biološkom oružju koji su uglavnom funkcionisali, iako nisu savršeni. Ali robotska oružja mogu biti isto tako opasna jer će ona skoro sigurno biti upotrebljena i bila bi isto tako nagrizajuća za naše demokratske institucije.
Now in November 2012 the U.S. Department of Defense issued a directive requiring a human being be present in all lethal decisions. This temporarily effectively banned autonomous weapons in the U.S. military, but that directive needs to be made permanent. And it could set the stage for global action. Because we need an international legal framework for robotic weapons. And we need it now, before there's a devastating attack or a terrorist incident that causes nations of the world to rush to adopt these weapons before thinking through the consequences. Autonomous robotic weapons concentrate too much power in too few hands, and they would imperil democracy itself.
U novembru 2012. Ministarstvo odbrane Sjedinjenih Država izdalo je direktivu koja zahteva da čovek bude prisutan kod svih smrtonosnih odluka. Ovim su u stvari zabranjena autonomna oružja u američkoj vojsci, ali ta direktiva bi trebalo da postane trajna. To bi moglo da pripremi pozornicu za globalnu akciju. Zato što nam treba međunarodni zakonski okvir za robotska oružja. I potreban nam je odmah, pre nego što se desi razoran napad ili teroristički incident koji uzrokuje da nacije sveta požure da usvoje ova oružja pre nego što razmisle o posledicama. Autonomna robotska oružja koncentrišu previše moći u suviše malom broju ruku i to bi ugrozilo samu demokratiju.
Now, don't get me wrong, I think there are tons of great uses for unarmed civilian drones: environmental monitoring, search and rescue, logistics. If we have an international treaty on robotic weapons, how do we gain the benefits of autonomous drones and vehicles while still protecting ourselves against illegal robotic weapons?
Nemojte me shvatiti pogrešno, mislim da ima mnogo sjajnih načina za upotrebu nenaoružanih civilnih bespilotnih letelica: nadgledanje okoline, potraga i spasavanje, logistika. Ako bismo imali međunarodni sporazum o robotskom oružju, kako bismo onda dobili koristi od autonomnih bespilotnih letelica i vozila, dok se u isto vreme čuvamo od ilegalnog robotskog oružja?
I think the secret will be transparency. No robot should have an expectation of privacy in a public place.
Mislim da će tajna biti u transparentnosti. Nijedan robot ne bi trebalo da očekuje da ima privatnost u javnom prostoru.
(Applause)
(Aplauz)
Each robot and drone should have a cryptographically signed I.D. burned in at the factory that can be used to track its movement through public spaces. We have license plates on cars, tail numbers on aircraft. This is no different. And every citizen should be able to download an app that shows the population of drones and autonomous vehicles moving through public spaces around them, both right now and historically. And civic leaders should deploy sensors and civic drones to detect rogue drones, and instead of sending killer drones of their own up to shoot them down, they should notify humans to their presence. And in certain very high-security areas, perhaps civic drones would snare them and drag them off to a bomb disposal facility.
Svaki robot i bespilotna letelica bi trebalo da ima kriptografsku identifikaciju utisnutu u fabrici koja može da se iskoristi za praćenje njegovog kretanja kroz javne prostore. Imamo tablice na kolima, brojeve na repovima aviona. Ovo nije ništa drugačije. Svaki građanin bi trebalo da može da skine aplikaciju koja prikazuje populaciju bespilotnih letelica i autonomnih vozila koja se kreće u javnim prostorima oko njih i u sadašnjem trenutku i u prošlosti. I gradske vođe bi trebalo da postave senzore i gradske bespilotne letelice da otkrivaju nezakonite letelice i umesto da šalju same letelice ubice da ih obore, trebalo bi da obaveste ljude o njihovom prisustvu. I u određenim oblastima visoke bezbednosti možda bi gradske bespilotne letelice mogle da ih uhvate i odvuku ih do mesta za odlaganje bombi.
But notice, this is more an immune system than a weapons system. It would allow us to avail ourselves of the use of autonomous vehicles and drones while still preserving our open, civil society.
Ali vidite, ovo je otporniji sistem od sistema oružja. To bi nam omogućilo da iskoristimo upotrebu autonomnih vozila i bespilotnih letelica dok u isto vreme čuvamo naše otvoreno, građansko društvo.
We must ban the deployment and development of killer robots. Let's not succumb to the temptation to automate war. Autocratic governments and criminal organizations undoubtedly will, but let's not join them. Autonomous robotic weapons would concentrate too much power in too few unseen hands, and that would be corrosive to representative government. Let's make sure, for democracies at least, killer robots remain fiction.
Moramo zabraniti razmeštanje i razvoj robota ubica. Nemojmo da podlegnemo iskušenju da automatizujemo rat. Apsolutističke vlade i kriminalne organizacije bez sumnje hoće, ali nemojmo da im se pridružimo. Autonomna robotska oružja bi koncentrisala suviše moći u suviše mali broj nevidljivih ruku i to bi bilo nagrizajuće za predstavnike vlasti. Osigurajmo da, barem za demokratije, roboti ubice ostanu fikcija.
Thank you.
Hvala vam.
(Applause) Thank you. (Applause)
(Aplauz) Hvala vam. (Aplauz)