I write fiction sci-fi thrillers, so if I say "killer robots," you'd probably think something like this. But I'm actually not here to talk about fiction. I'm here to talk about very real killer robots, autonomous combat drones.
我是写科幻惊悚小说的, 所以如果我说"机器人杀手" 你也许会这么想。 但我其实不打算在这里谈论小说。 我在这里要谈论非常真实的杀手机器人, 自主作战无人机。
Now, I'm not referring to Predator and Reaper drones, which have a human making targeting decisions. I'm talking about fully autonomous robotic weapons that make lethal decisions about human beings all on their own. There's actually a technical term for this: lethal autonomy.
现在,我不是指“捕食者”和“死神无人机”, 他们是由人类决定攻击目标的。 我说的是完全自主的机器人武器 它们自己作出攻击 人类的决定。 这实际上有一个技术术语: 致命的自主权。
Now, lethally autonomous killer robots would take many forms -- flying, driving, or just lying in wait. And actually, they're very quickly becoming a reality. These are two automatic sniper stations currently deployed in the DMZ between North and South Korea. Both of these machines are capable of automatically identifying a human target and firing on it, the one on the left at a distance of over a kilometer. Now, in both cases, there's still a human in the loop to make that lethal firing decision, but it's not a technological requirement. It's a choice. And it's that choice that I want to focus on, because as we migrate lethal decision-making from humans to software, we risk not only taking the humanity out of war, but also changing our social landscape entirely, far from the battlefield. That's because the way humans resolve conflict shapes our social landscape. And this has always been the case, throughout history.
现在,致命的自主机器人杀手 有许多形式——飞行、 驾驶、 或只是埋伏。 实际上,他们正在很快成为现实。 这些都是两个自动的狙击地点 目前部署在朝鲜和韩国之间非军事区。 这些机器都能够自动 识别人类目标并向其射击, 左边的那个可以在一公里外(就发现目标)。 现在,这两种情况下,仍有一个人类在控制 来下这一致命射击的决定, 但它不是一项技术要求。它是一种选择。 我重点关注的正是这一选择, 因为随着我们致命决策权的迁移 从人到软件, 我们甘冒风险不只让人们远离战争 而且也在改变我们的整个社会格局, 使其远离战场。 这是因为人类解决冲突的方式 决定了我们社会的格局。 在整个历史长河中一直是如此。
For example, these were state-of-the-art weapons systems in 1400 A.D. Now they were both very expensive to build and maintain, but with these you could dominate the populace, and the distribution of political power in feudal society reflected that. Power was focused at the very top. And what changed? Technological innovation. Gunpowder, cannon. And pretty soon, armor and castles were obsolete, and it mattered less who you brought to the battlefield versus how many people you brought to the battlefield. And as armies grew in size, the nation-state arose as a political and logistical requirement of defense. And as leaders had to rely on more of their populace, they began to share power. Representative government began to form.
例如,这些都是最先进的武器系统 在公元 1400 年 现在建造和维护它们都很昂贵 但有了这些,你就可以主宰民众, 同时也反映了封建的社会中政治权力的分配。 权力集中在最顶端。 什么改变了?技术创新。 火药,加农炮。 很快,装甲和城堡都已过时, 你带谁上战场已经不那么重要 重要的是你带了多少人上战场。 并且随着军队的组织的扩大,为了满足国防上的 政治和后勤要求,民族国家出现了。 作为领导人不得不依靠更多的民众, 他们开始分享权力。 代表政府开始形成。
So again, the tools we use to resolve conflict shape our social landscape. Autonomous robotic weapons are such a tool, except that, by requiring very few people to go to war, they risk re-centralizing power into very few hands, possibly reversing a five-century trend toward democracy.
再次,我们用来解决冲突的工具 塑造了我们社会的格局。 自主机器人武器就是这样一个工具, 除此之外,通过只需要很少的人去打仗, 他们的风险在于权力可能重新回到少数人手里, 可能扭转五个世纪的民主潮流。
Now, I think, knowing this, we can take decisive steps to preserve our democratic institutions, to do what humans do best, which is adapt. But time is a factor. Seventy nations are developing remotely-piloted combat drones of their own, and as you'll see, remotely-piloted combat drones are the precursors to autonomous robotic weapons. That's because once you've deployed remotely-piloted drones, there are three powerful factors pushing decision-making away from humans and on to the weapon platform itself.
现在,我想,知道这个, 我们可以采取果断措施,维护我们的民主机构, 做人类擅长的事情,这是适应。 但时间是一个因素。 七十个国家正在自主发展遥控 无人作战机, 如你所见,遥控作战无人机是 自主机器人武器的前提。 这是因为一旦您已经部署了遥控无人作战机, 有三个强大的因素迫使人类 把决策权交给武器平台本身。
The first of these is the deluge of video that drones produce. For example, in 2004, the U.S. drone fleet produced a grand total of 71 hours of video surveillance for analysis. By 2011, this had gone up to 300,000 hours, outstripping human ability to review it all, but even that number is about to go up drastically. The Pentagon's Gorgon Stare and Argus programs will put up to 65 independently operated camera eyes on each drone platform, and this would vastly outstrip human ability to review it. And that means visual intelligence software will need to scan it for items of interest. And that means very soon drones will tell humans what to look at, not the other way around.
第一是无人机拍摄的大量视频。 例如,2004 年,美国无人机梯队拍摄了 总计 71 个小时的视频监控可供分析。 2011 年,这已经达 300,000 小时 超过了人类的审查能力, 但即使这一数字会猛烈地增长。 五角大楼的“戈尔贡凝视”(Gorgon Stare)和守卫程序 还将把 65个 独立的照相机眼睛 安装在每个无人机平台上, 这将远远超过人类的审查能力。 这就需要视觉情报软件 去扫描它感兴趣的目标。 这意味着很快 无人机将告诉人类要看什么 不是其他方式。
But there's a second powerful incentive pushing decision-making away from humans and onto machines, and that's electromagnetic jamming, severing the connection between the drone and its operator. Now we saw an example of this in 2011 when an American RQ-170 Sentinel drone got a bit confused over Iran due to a GPS spoofing attack, but any remotely-piloted drone is susceptible to this type of attack, and that means drones will have to shoulder more decision-making. They'll know their mission objective, and they'll react to new circumstances without human guidance. They'll ignore external radio signals and send very few of their own.
但还有第二个的强大动力迫使 人类把决策权交给机器。 这就是电磁干扰, 它会切断无人飞机和操控者 之间的联系。 现在我们看到的例子发生在 2011 年 当美国 RQ-170 哨兵无人机 由于GPS欺骗攻击,在伊朗上空而发生混乱, 但任何遥控无人飞机都极易受这种类型的攻击, 这就意味着无人机 将不得不承担更多的决策。 他们就会知道他们的使命目标 他们在无人操纵的情况下有自主应对新情况的能力。 他们会忽略外部的无线电信号 只发出极少量的无线信号。
Which brings us to, really, the third and most powerful incentive pushing decision-making away from humans and onto weapons: plausible deniability. Now we live in a global economy. High-tech manufacturing is occurring on most continents. Cyber espionage is spiriting away advanced designs to parts unknown, and in that environment, it is very likely that a successful drone design will be knocked off in contract factories, proliferate in the gray market. And in that situation, sifting through the wreckage of a suicide drone attack, it will be very difficult to say who sent that weapon.
这就引出了,第三个 最强大的诱因促使我们 将决策权交给武器自身: 貌似合理的推脱。 现在我们生活在一个全球化的经济大潮中。 高科技制造业在大多数大陆上发生。 网络间谍活动正在将先进的设计 带到未知世界, 在那种环境下,它是很有可能 一个成功的无人驾驶飞机设计在授权工厂被停工, 导致其在灰色市场量产。 在这种情况,即使仔细筛查 无人机自杀式攻击的残骸, 也很难说谁发射了这个武器。
This raises the very real possibility of anonymous war. This could tilt the geopolitical balance on its head, make it very difficult for a nation to turn its firepower against an attacker, and that could shift the balance in the 21st century away from defense and toward offense. It could make military action a viable option not just for small nations, but criminal organizations, private enterprise, even powerful individuals. It could create a landscape of rival warlords undermining rule of law and civil society. Now if responsibility and transparency are two of the cornerstones of representative government, autonomous robotic weapons could undermine both.
这就引起了非常现实的可能性 匿名的战争。 这将导致地缘政治失衡并倾向一头, 使一个国家很难将其火力 对准攻击者,这可能会令21 世纪的平衡 从防御走向进攻。 它能使军事行动成为可行的选择 不只是那些小国家 甚至犯罪组织、 私人企业、 甚至是强大的个人都可以这么做。 它可以引起军阀割据的混战, 破坏法治和文明社会。 现在如果责任和透明度 是代议制政府的两个基石 自主机器人武器可能同时摧毁它们。
Now you might be thinking that citizens of high-tech nations would have the advantage in any robotic war, that citizens of those nations would be less vulnerable, particularly against developing nations. But I think the truth is the exact opposite. I think citizens of high-tech societies are more vulnerable to robotic weapons, and the reason can be summed up in one word: data. Data powers high-tech societies. Cell phone geolocation, telecom metadata, social media, email, text, financial transaction data, transportation data, it's a wealth of real-time data on the movements and social interactions of people. In short, we are more visible to machines than any people in history, and this perfectly suits the targeting needs of autonomous weapons.
现在你可能会想, 高科技的国家的公民 在任何机器人的战争中占有优势 这些国家的公民将不再那么脆弱, 尤其是针对发展中国家。 但事实是正好相反。 我认为高科技社会的公民 面对机器人武器更加脆弱, 原因可以用一个词概括: 数据。 数据掌控着高科技社会。 手机定位,电信的元数据, 社会媒体、 电子邮件、 文本、 金融交易数据, 交通数据,那么多丰富的实时数据 关于个人的动向和社交关系。 简而言之,对于机器而言, 我们比历史上任何人更显而易见, 这完全符合了自主武器瞄准的需要。
What you're looking at here is a link analysis map of a social group. Lines indicate social connectedness between individuals. And these types of maps can be automatically generated based on the data trail modern people leave behind. Now it's typically used to market goods and services to targeted demographics, but it's a dual-use technology, because targeting is used in another context. Notice that certain individuals are highlighted. These are the hubs of social networks. These are organizers, opinion-makers, leaders, and these people also can be automatically identified from their communication patterns. Now, if you're a marketer, you might then target them with product samples, try to spread your brand through their social group. But if you're a repressive government searching for political enemies, you might instead remove them, eliminate them, disrupt their social group, and those who remain behind lose social cohesion and organization. Now in a world of cheap, proliferating robotic weapons, borders would offer very little protection to critics of distant governments or trans-national criminal organizations. Popular movements agitating for change could be detected early and their leaders eliminated before their ideas achieve critical mass. And ideas achieving critical mass is what political activism in popular government is all about. Anonymous lethal weapons could make lethal action an easy choice for all sorts of competing interests. And this would put a chill on free speech and popular political action, the very heart of democracy.
现在你正在看到的 是某一社会群体的关系分析图。 线表示人与人之间的社会联系。 这类型的地图,基于现代人留下的数据追踪, 可以自动生成。 现在它通常用于市场商品和服务 有针对性的人口统计数据,但这种技术有双重用途, 因为目标被用于另一种情境。 请注意那些突出显示的个人。 这些都是社交网络的枢纽。 这些都是组织者、 决策者、 领袖、 这些人也可以进行通过他们的通讯模式 被自动识别出来 现在,如果你是营销人员,您可能瞄准他们 用产品的样本,试图传播您的品牌 通过他们的社群。 但是如果你是一个专制的政府 寻找政治的敌人,你反而可能会想要拔掉这些眼中钉, 消灭他们,从而扰乱他们的社团 而那些留下的人将失去社会凝聚力 和组织。 在一个机器人武器便宜又泛滥的世界里, 国界的保护微乎其微, 无法防御远方政府批评者 或跨国犯罪组织。 鼓动改革的民众运动 可以在早期即被发现, 他们的领袖在向大众充分宣传他们的观点之前就会被消灭。 向大众充分宣传其观点 正是政治激进主义政府所追求的。 匿名致命武器能让致命行动 成为各种相互竞争利益的一种最简单的选择。 这将会令民主的核心——言论自由 和民众政治运动受到限制。
And this is why we need an international treaty on robotic weapons, and in particular a global ban on the development and deployment of killer robots. Now we already have international treaties on nuclear and biological weapons, and, while imperfect, these have largely worked. But robotic weapons might be every bit as dangerous, because they will almost certainly be used, and they would also be corrosive to our democratic institutions.
这就是为什么我们需要一项国际条约 关于机器人武器的禁令 禁止全球开发和部署机器人杀手。 现在我们已经有国际条约 关于核武器和生化武器,虽然不尽完善, 这些很大程度上起了作用。 但机器人武器可能一样危险 因为它们几乎可以肯定将会被使用, 他们也会侵蚀我们的民主机构。
Now in November 2012 the U.S. Department of Defense issued a directive requiring a human being be present in all lethal decisions. This temporarily effectively banned autonomous weapons in the U.S. military, but that directive needs to be made permanent. And it could set the stage for global action. Because we need an international legal framework for robotic weapons. And we need it now, before there's a devastating attack or a terrorist incident that causes nations of the world to rush to adopt these weapons before thinking through the consequences. Autonomous robotic weapons concentrate too much power in too few hands, and they would imperil democracy itself.
2012 年 11 月,美国国防部 直接要求 所有致命决定都必须有人类参与。 这暂时有效地阻止了自主武器在美国军队中的使用 但该指令需要永久有效。 它可以为全球立法行动铺路。 因为,我们需要 一个针对于机器人武器的国际法律框架。 我们现在就需要这样的法律,用以防止一个具有破坏性攻击 或一个恐怖袭击使得世界各国 在完全不顾及后果的前提下, 就急于采用这些武器。 自主机器人武器集中了太多的权利 集中在太少人手中,他们将会危及民主本身。
Now, don't get me wrong, I think there are tons of great uses for unarmed civilian drones: environmental monitoring, search and rescue, logistics. If we have an international treaty on robotic weapons, how do we gain the benefits of autonomous drones and vehicles while still protecting ourselves against illegal robotic weapons?
现在,别误会我,我觉得这些非武装的民用无人机 用途很多: 环境监测、 搜索和救援,物流。 如果我们有一项关于机器人的武器国际条约, 我们如何从自主无人机和自主车上获得好处 同时还保护我们自己 免受非法机器人武器的攻击?
I think the secret will be transparency. No robot should have an expectation of privacy in a public place.
我认为秘密会变得透明化。 任何机器人都不应该 在公众领域有隐私。
(Applause)
(掌声)
Each robot and drone should have a cryptographically signed I.D. burned in at the factory that can be used to track its movement through public spaces. We have license plates on cars, tail numbers on aircraft. This is no different. And every citizen should be able to download an app that shows the population of drones and autonomous vehicles moving through public spaces around them, both right now and historically. And civic leaders should deploy sensors and civic drones to detect rogue drones, and instead of sending killer drones of their own up to shoot them down, they should notify humans to their presence. And in certain very high-security areas, perhaps civic drones would snare them and drag them off to a bomb disposal facility.
每个机器人和无人驾驶飞机都应该 在工厂内被烙上一个加密的身份信息 可以用来对其在公共领域内动向的跟踪。 我们的汽车有车牌,飞机尾部有编号。 这没有什么不同。 每个公民应能够下载应用程序 能够显示通过他们周围公共空间的 无人机和自主车的数量 不论现在还是过去。 领导者应部署传感器和民用无人驾驶飞机 来检测流氓无人机, 而不是发送能够击落他们的杀手无人机, 应该通知人们这些无人机的存在。 在某些非常高安全等级的地区, 或许民用无人机可以将他们拖到 一个设有炸弹设施处理掉。
But notice, this is more an immune system than a weapons system. It would allow us to avail ourselves of the use of autonomous vehicles and drones while still preserving our open, civil society.
但请注意,这更象一个免疫系统, 而不是武器系统。 这样,它将使我们能够利用 自主车辆和无人驾驶飞机的好处 同时仍然保留我们开放的公民社会。
We must ban the deployment and development of killer robots. Let's not succumb to the temptation to automate war. Autocratic governments and criminal organizations undoubtedly will, but let's not join them. Autonomous robotic weapons would concentrate too much power in too few unseen hands, and that would be corrosive to representative government. Let's make sure, for democracies at least, killer robots remain fiction.
我们必须禁止部署与发展 机器人杀手。 我们不会屈服于自动化战争的诱惑。 独裁政府和犯罪组织 毫无疑问的会屈服,让我们不要加入他们。 自主机器人武器 集中了太多的权力 在太少看不见的手中, 那将是侵蚀我们的代议制政府。 让我们保证,至少为了民主的理由, 让机器人杀手只存在于小说里吧。
Thank you.
谢谢。
(Applause) Thank you. (Applause)
(掌声) 谢谢。(掌声)