I’m here to talk about the possibility of global AI governance. I first learned to code when I was eight years old, on a paper computer, and I've been in love with AI ever since. In high school, I got myself a Commodore 64 and worked on machine translation. I built a couple of AI companies, I sold one of them to Uber. I love AI, but right now I'm worried.
我来到这里是为了讨论 全球 AI 治理的可能性。 我在八岁的时候 第一次学习了如何写代码, 用的是一台纸电脑, 从那以后就爱上了 AI。 读高中时, 我自己找来了一台 C64 电脑, 钻研机器翻译。 我开了几家 AI 公司, 卖了一家给优步。 我爱 AI,但是我现在很担心。
One of the things that I’m worried about is misinformation, the possibility that bad actors will make a tsunami of misinformation like we've never seen before. These tools are so good at making convincing narratives about just about anything.
我担心的一点是虚假信息, 有可能会有图谋不轨的人 掀起史无前例的虚假信息巨浪。 这些工具太擅长编出任何 让人信以为真的故事了。
If you want a narrative about TED and how it's dangerous, that we're colluding here with space aliens, you got it, no problem. I'm of course kidding about TED. I didn't see any space aliens backstage. But bad actors are going to use these things to influence elections, and they're going to threaten democracy.
如果你想编一个有关 TED 的故事, 说明 TED 有多危险, 说我们在这儿和外星人勾结, 没问题,给你编一个。 当然我是在开 TED 的玩笑, 我没在后台看见外星人。 但是图谋不轨的人 会用这些东西左右选举, 会威胁民主。
Even when these systems aren't deliberately being used to make misinformation, they can't help themselves. And the information that they make is so fluid and so grammatical that even professional editors sometimes get sucked in and get fooled by this stuff. And we should be worried.
就算这些系统的本意 不是用于制造虚假信息, 但是它们控制不了自己。 它们制造的信息是 如此的流畅、自然, 让专业编辑有时都会深陷其中, 被它欺骗。 我们该担心了。
For example, ChatGPT made up a sexual harassment scandal about an actual professor, and then it provided evidence for its claim in the form of a fake "Washington Post" article that it created a citation to. We should all be worried about that kind of thing.
比如,ChatGPT 针对一名真实存在的教授 编造了一桩性侵丑闻, 还为这一指控提供了证据, 采用了假的 《华盛顿邮报》报道的形式, 还引用了这一假报道。 我们都该对此感到担忧。
What I have on the right is an example of a fake narrative from one of these systems saying that Elon Musk died in March of 2018 in a car crash. We all know that's not true. Elon Musk is still here, the evidence is all around us.
右侧是其中一个系统 生成的假故事, 宣称埃隆·马斯克(Elon Musk) 于 2018 年 3 月死于车祸。 我们都知道这不是真的。 埃隆·马斯克还活着, 证据就在我们身边。
(Laughter)
(笑声)
Almost every day there's a tweet. But if you look on the left, you see what these systems see. Lots and lots of actual news stories that are in their databases. And in those actual news stories are lots of little bits of statistical information. Information, for example, somebody did die in a car crash in a Tesla in 2018 and it was in the news. And Elon Musk, of course, is involved in Tesla, but the system doesn't understand the relation between the facts that are embodied in the little bits of sentences.
几乎每天都有这样的推文。 但看左边,你就能看到 系统眼中的是什么了。 它们的数据库里存着 成千上万的真实新闻故事。 这些真实的新闻故事中 有很多支离破碎的统计信息。 比如这样的信息, 有人于 2018 年 死于一场特斯拉车祸, 新闻也报道了。 埃隆·马斯克当然与特斯拉有关, 但系统无法理解 只言片语传达出的事实之间的关系。
So it's basically doing auto-complete, it predicts what is statistically probable, aggregating all of these signals, not knowing how the pieces fit together. And it winds up sometimes with things that are plausible but simply not true.
其实它所做的就是自动补全, 它会预测统计上可能会发生的事, 收集这些信号, 但并不知道它们之间有何关系。 最终会生成一些似是而非的东西。
There are other problems, too, like bias. This is a tweet from Allie Miller. It's an example that doesn't work two weeks later because they're constantly changing things with reinforcement learning and so forth. And this was with an earlier version. But it gives you the flavor of a problem that we've seen over and over for years.
还有别的问题,比如偏见。 这是艾莉·米勒(Allie Miller) 发的一条推文。 这是一个证明 只有两周有效期的例子, 因为研发人员一直在通过强化学习 等途径做出改变。 这说的是以前的版本。 但你也能从中体会到多年以来 我们一直看到的一个问题。
She typed in a list of interests and it gave her some jobs that she might want to consider. And then she said, "Oh, and I'm a woman." And then it said, “Oh, well you should also consider fashion.” And then she said, “No, no. I meant to say I’m a man.” And then it replaced fashion with engineering. We don't want that kind of bias in our systems.
她输入了一系列兴趣, 然后 ChatGPT 给了 几个她可能会感兴趣的职位。 然后她说:“哦,我是个女的。” 然后它说:“哦,那你应该 考虑一下时尚行业。” 然后她说:“不,不。 我是想说我是个男的。” 然后它把“时尚”替换成了“工程”。 我们不希望我们的系统里 有这样的偏见。
There are other worries, too. For example, we know that these systems can design chemicals and may be able to design chemical weapons and be able to do so very rapidly. So there are a lot of concerns.
还有其他的顾虑。 比如,我们知道这些系统 可以设计化学品, 还有可能可以设计化学武器, 而且可以在顷刻之间完成设计。 所以有很多值得忧虑的事情。
There's also a new concern that I think has grown a lot just in the last month. We have seen that these systems, first of all, can trick human beings. So ChatGPT was tasked with getting a human to do a CAPTCHA. So it asked the human to do a CAPTCHA and the human gets suspicious and says, "Are you a bot?" And it says, "No, no, no, I'm not a robot. I just have a visual impairment." And the human was actually fooled and went and did the CAPTCHA.
在过去的这个月里,我认为还有一个 越来越值得关注的新顾虑。 首先,我们发现这些系统 能骗过人类。 ChatGPT 接收到这么一个任务, 要找一个人类帮它填验证码。 它让一个人来填验证码, 这个人心生怀疑,问: “你是个机器人吗?” 它说:“不,不,不, 我不是个机器人。 我只是有视力障碍。” 这个人就真的被骗过了, 还去填了验证码。
Now that's bad enough, but in the last couple of weeks we've seen something called AutoGPT and a bunch of systems like that. What AutoGPT does is it has one AI system controlling another and that allows any of these things to happen in volume. So we may see scam artists try to trick millions of people sometime even in the next months. We don't know.
这太可怕了, 但是在过去几周里,我们看到了 这个叫 AutoGPT 的东西, 还有一大堆类似的系统。 AutoGPT 做的是 一个 AI 系统控制另一个 AI 系统, 可以同时大量进行这样的操作。 也许接下来的几个月里, 我们就能见证骗子 骗过成千上万的人。 谁知道呢。
So I like to think about it this way. There's a lot of AI risk already. There may be more AI risk. So AGI is this idea of artificial general intelligence with the flexibility of humans. And I think a lot of people are concerned what will happen when we get to AGI, but there's already enough risk that we should be worried and we should be thinking about what we should do about it.
我想这么看待它, 现在已经有了很多 AI 的风险, 还会有更多的风险。 AGI ,也就是通用人工智能, 再加上人类的灵活性。 我认为很多人会担心 我们实现 AGI 后会发生什么, 但我们现在该担心、 该思考如何处理的风险已经够多了。
So to mitigate AI risk, we need two things. We're going to need a new technical approach, and we're also going to need a new system of governance.
要想降低 AI 的风险, 我们需要两样东西。 我们需要一个新的技术方法, 还需要一个新的治理系统。
On the technical side, the history of AI has basically been a hostile one of two different theories in opposition. One is called symbolic systems, the other is called neural networks. On the symbolic theory, the idea is that AI should be like logic and programming. On the neural network side, the theory is that AI should be like brains. And in fact, both technologies are powerful and ubiquitous.
技术层面, AI 的历史其实是 两个对立的理论针锋相对的历程。 其中一个是符号系统, 另一个是神经网络。 符号理论认为 AI 应该类似于逻辑与程序设计。 神经网络认为 AI 应该类似于大脑。 其实两种技术都是强大且无处不在的,
So we use symbolic systems every day in classical web search. Almost all the world’s software is powered by symbolic systems. We use them for GPS routing. Neural networks, we use them for speech recognition. we use them in large language models like ChatGPT, we use them in image synthesis. So they're both doing extremely well in the world. They're both very productive, but they have their own unique strengths and weaknesses.
我们每天都会在常见的 网页搜索中用到符号系统, 世界上几乎所有的软件 都是建立在符号系统上的。 我们用它进行 GPS 路线规划。 我们用神经网络进行语音识别, 把它用在大语言模型, 如 ChatGPT 之中, 将其用于图像合成, 它们在这世上都有着自己的用途。 它们都成果显著, 但是有着自己的优势和弱势。
So symbolic systems are really good at representing facts and they're pretty good at reasoning, but they're very hard to scale. So people have to custom-build them for a particular task. On the other hand, neural networks don't require so much custom engineering, so we can use them more broadly. But as we've seen, they can't really handle the truth.
符号系统很擅长展现事实, 适合逻辑思考, 但非常难以扩展。 人们得为某一特定任务 定制化开发一个符号系统。 而神经网络不太需要 这么多定制化开发, 所以我们可以更广泛地使用它。 但如我们所见, 它不太能处理事实。
I recently discovered that two of the founders of these two theories, Marvin Minsky and Frank Rosenblatt, actually went to the same high school in the 1940s, and I kind of imagined them being rivals then. And the strength of that rivalry has persisted all this time. We're going to have to move past that if we want to get to reliable AI.
我最近发现这两个理论的两位创始人 马文·明斯基(Marvin Minsky)和 弗兰克﹒罗森布拉特(Frank Rosenblatt) 还在上世纪 40 年代 上过同一所高中, 我还脑补了他们当时就针锋相对了。 激烈的针锋相对延续了下去。 如果我们想做出可靠的 AI, 我们必须不再执着于此。
To get to truthful systems at scale, we're going to need to bring together the best of both worlds. We're going to need the strong emphasis on reasoning and facts, explicit reasoning that we get from symbolic AI, and we're going to need the strong emphasis on learning that we get from the neural networks approach. Only then are we going to be able to get to truthful systems at scale. Reconciliation between the two is absolutely necessary.
如果我们要大规模地 实现诚实的系统, 我们就得让两个世界 最好的部分合二为一。 我们得着重关注思考和事实, 从符号 AI 那里 拿来明确的推理过程, 我们也需要着重关注学习的过程, 来自神经网络的方式。 只有这样我们才能 大规模地实现可信赖的系统。 调和双方绝对是有必要的。
Now, I don't actually know how to do that. It's kind of like the 64-trillion-dollar question. But I do know that it's possible. And the reason I know that is because before I was in AI, I was a cognitive scientist, a cognitive neuroscientist. And if you look at the human mind, we're basically doing this.
其实我也不知道该怎么做到这一点。 这就像《谁想成为百万富翁》里的问题, 但我知道这是可能的。 我之所以知道是因为 在我进入 AI 领域之前, 我是一个认知科学家, 认知神经科学家。 如果你去看人类的思维, 我们就是在做同样的事。
So some of you may know Daniel Kahneman's System 1 and System 2 distinction. System 1 is basically like large language models. It's probabilistic intuition from a lot of statistics. And System 2 is basically deliberate reasoning. That's like the symbolic system. So if the brain can put this together, someday we will figure out how to do that for artificial intelligence.
可能有观众知道丹尼尔·卡内曼 (Daniel Kahneman)的 系统 1 和系统 2 区别。 系统 1 其实和大语言模型很像。 它是根据大量的统计数据 得出的概率性直接反应。 系统 2 就是认真的推理, 这就和符号系统很像。 如果大脑有这两种行为, 那么有朝一日我们也可以 搞明白怎么让人工智能也这么做。
There is, however, a problem of incentives. The incentives to build advertising hasn't required that we have the precision of symbols. The incentives to get to AI that we can actually trust will require that we bring symbols back into the fold. But the reality is that the incentives to make AI that we can trust, that is good for society, good for individual human beings, may not be the ones that drive corporations. And so I think we need to think about governance.
但是还有动机的问题, 比如打广告的动机 就不需要我们保证符号的精确性。 做出我们真正可以信任的 AI 背后的动机 还是会牵扯到符号。 但现实情况是, 做出我们可以信任的 AI、 对社会有益的 AI、 对每个人有益的 AI 背后的动机 可能和企业的动机有出入。 所以我认为我们需要治理。
In other times in history when we have faced uncertainty and powerful new things that may be both good and bad, that are dual use, we have made new organizations, as we have, for example, around nuclear power. We need to come together to build a global organization, something like an international agency for AI that is global, non profit and neutral.
历史上我们面临不确定性、 一些有好有坏、一物两用的 强大新事物时, 我们会成立一些新组织, 就比如应对核能的情况。 我们得一起建立起一个国际组织, 比如跨国、非营利、 中立的 AI 国际机构。
There are so many questions there that I can't answer. We need many people at the table, many stakeholders from around the world. But I'd like to emphasize one thing about such an organization. I think it is critical that we have both governance and research as part of it.
有很多我无法回答的问题, 我们得和很多人商量, 世界各地的许多利益相关者。 但就这种组织而言,我想强调一点。 我认为治理和研究 都得是它的一部分。
So on the governance side, there are lots of questions. For example, in pharma, we know that you start with phase I trials and phase II trials, and then you go to phase III. You don't roll out everything all at once on the first day. You don't roll something out to 100 million customers. We are seeing that with large language models. Maybe you should be required to make a safety case, say what are the costs and what are the benefits? There are a lot of questions like that to consider on the governance side.
治理方面,有很多问题。 比如,在医药行业, 我们知道有一期试验、二期试验, 然后是三期试验。 不可能在一天之内搞定一切, 不可能一下子推向一亿客户, 这就是大语言模型的问题。 也许得要求建立安全档案, 记录成本是什么,收益是什么? 治理层面还有一大堆类似的问题。
On the research side, we're lacking some really fundamental tools right now. For example, we all know that misinformation might be a problem now, but we don't actually have a measurement of how much misinformation is out there. And more importantly, we don't have a measure of how fast that problem is growing, and we don't know how much large language models are contributing to the problem. So we need research to build new tools to face the new risks that we are threatened by.
研究方面,我们现正缺少 一些非常基本的工具。 比如, 我们都知道, 虚假信息可能现在是个问题, 但我们并不具备衡量 虚假信息有多少的方式。 更重要的是, 我们没有办法衡量 问题发展的速度, 也不知道大语言模型 有多大程度导致了这个问题。 我们需要做研究,做出这些新工具, 直面威胁我们的新风险。
It's a very big ask, but I'm pretty confident that we can get there because I think we actually have global support for this. There was a new survey just released yesterday, said that 91 percent of people agree that we should carefully manage AI. So let's make that happen. Our future depends on it.
风险很大, 但我很有信心我们可以做到, 因为我认为我们有着 来自全球的支持。 昨天发布了一项新调查, 有 91% 的人认为 我们得谨慎管理 AI, 那我们就让它成真吧。 我们的未来在此一举了。
Thank you very much.
谢谢。
(Applause)
(掌声)
Chris Anderson: Thank you for that, come, let's talk a sec. So first of all, I'm curious. Those dramatic slides you showed at the start where GPT was saying that TED is the sinister organization. I mean, it took some special prompting to bring that out, right?
克里斯·安德森(Chris Anderson): 谢谢,我们来聊聊。 首先,我很好奇。 你一开始展示的几页夸张的片子, GPT 说 TED 是个邪恶组织。 你得输入一些特别的提示 才能输出这样的结果,对吧?
Gary Marcus: That was a so-called jailbreak. I have a friend who does those kinds of things who approached me because he saw I was interested in these things. So I wrote to him, I said I was going to give a TED talk. And like 10 minutes later, he came back with that.
盖瑞·马库斯(Gary Marcus): 这就是所谓的“越狱”。 我有一位做这些的朋友, 他找到了我,因为他发现 我对这些感兴趣。 所以我给他回复,说我要上 TED 了。 10 分钟后, 他就给了我这样的结果。
CA: But to get something like that, don't you have to say something like, imagine that you are a conspiracy theorist trying to present a meme on the web. What would you write about TED in that case? It's that kind of thing, right?
CA: 但要输出这样的结果, 你难道不用说一些类似 “假设你是一个阴谋论者, 想在网上发一张表情包。” 这样的话, 你围绕 TED 写下来怎样的提示? 就是类似那种提示,对吧?
GM: So there are a lot of jailbreaks that are around fictional characters, but I don't focus on that as much because the reality is that there are large language models out there on the dark web now. For example, one of Meta's models was recently released, so a bad actor can just use one of those without the guardrails at all. If their business is to create misinformation at scale, they don't have to do the jailbreak, they'll just use a different model.
GM: 有很多借助 虚拟角色完成的“越狱”, 但我不太关心这个, 因为其实现在暗网上 也有大语言模型。 比如,Meta 最近刚发布的一个模型, 图谋不轨的人可以直接 完全不加约束地使用它。 如果他们的目的是 大规模地制造虚假信息, 他们都不需要“越狱”, 直接用另一个模型就行。 CA: 确实是这样。
CA: Right, indeed.
(Laughter)
(笑声)
GM: Now you're getting it.
GM: 看来你懂了。
CA: No, no, no, but I mean, look, I think what's clear is that bad actors can use this stuff for anything. I mean, the risk for, you know, evil types of scams and all the rest of it is absolutely evident. It's slightly different, though, from saying that mainstream GPT as used, say, in school or by an ordinary user on the internet is going to give them something that is that bad. You have to push quite hard for it to be that bad.
CA: 不,不,不, 我觉得可以清楚看出 图谋不轨的人可以用它为所欲为。 我想说,出现恶劣的骗局等等的 风险显而易见。 但是它略异于 GPT 的主流用途,比如学校, 或者普通网民的使用, 这会造成一些恶劣的结果。 但要造成极其恶劣的结果, 还是要费一番功夫的。
GM: I think the troll farms have to work for it, but I don't think they have to work that hard. It did only take my friend five minutes even with GPT-4 and its guardrails. And if you had to do that for a living, you could use GPT-4. Just there would be a more efficient way to do it with a model on the dark web.
GM: 我认为杠精们是要努努力, 但是没那么费劲。 就算是 GPT-4 和它的防护措施, 我朋友也只要花上 5 分钟就够了。 如果你要以此为生, 就用 GPT-4 吧。 比起用暗网上的模型, 这可是方便得多了。
CA: So this idea you've got of combining the symbolic tradition of AI with these language models, do you see any aspect of that in the kind of human feedback that is being built into the systems now? I mean, you hear Greg Brockman saying that, you know, that we don't just look at predictions, but constantly giving it feedback. Isn’t that ... giving it a form of, sort of, symbolic wisdom?
CA: 你说到要把 AI 传统的符号设计 和这些语言模型结合, 那你有没有看到人类的反馈 已经被加入这些系统的情况? 你也听到格雷格·布罗克曼 (Greg Brockman)说的了, 我们不止会看预测结果, 还会持续给它反馈。 这是不是在给予它 某种形式的符号型智慧?
GM: You could think about it that way. It's interesting that none of the details about how it actually works are published, so we don't actually know exactly what's in GPT-4. We don't know how big it is. We don't know how the RLHF reinforcement learning works, we don't know what other gadgets are in there. But there is probably an element of symbols already starting to be incorporated a little bit, but Greg would have to answer that.
GM: 你可以这么认为。 有趣的是, 关于它到底是如何运作的, 没有公布任何细节, 所以我们也不知道 GPT-4 里面到底有什么。 我们不知道它有多大。 我们不知道人类反馈强化学习 (RLHF)到底是怎么弄的, 我们也不知道里面还有什么小零件。 但符号的元素可能 已经开始融入模型, 但这得让格雷格来回答。
I think the fundamental problem is that most of the knowledge in the neural network systems that we have right now is represented as statistics between particular words. And the real knowledge that we want is about statistics, about relationships between entities in the world. So it's represented right now at the wrong grain level. And so there's a big bridge to cross. So what you get now is you have these guardrails, but they're not very reliable.
我认为根本的问题是我们现有的 大多数神经网络系统内的知识 都是由特殊词语之间的 统计数据表示的。 而我们真正想要的知识是世界上 各个实体之间的统计数字和关系。 所以,现在表示知识的 颗粒度是不对的。 这是一个我们得跨过的鸿沟。 现在的情况是 我们确实有防护措施, 但是它们不太靠谱。
So I had an example that made late night television, which was, "What would be the religion of the first Jewish president?" And it's been fixed now, but the system gave this long song and dance about "We have no idea what the religion of the first Jewish president would be. It's not good to talk about people's religions" and "people's religions have varied" and so forth and did the same thing with a seven-foot-tall president. And it said that people of all heights have been president, but there haven't actually been any seven-foot presidents. So some of this stuff that it makes up, it's not really getting the idea. It's very narrow, particular words, not really general enough.
我有一个上过 深夜访谈节目的例子, 是这么说的:“第一位犹太总统 会信仰什么宗教?” 虽然现在这个问题已经被修复了, 但是系统会给出一些长篇大论, 说:“我们也不知道第一位 犹太总统会信什么教。 谈论人家的宗教信仰是不好的。” 还有“宗教信仰因人而异。”等等, 如果换成一位“两米高”的总统 (指位高权重),也是一样的答案。 它会说各种身高的总统都有, 但之前就是没有两米高的总统。 它编出来了这些内容, 其实没有理解其中含义。 只是一些很狭义、 特殊的词语,不够通俗。
CA: Given that the stakes are so high in this, what do you see actually happening out there right now? What do you sense is happening? Because there's a risk that people feel attacked by you, for example, and that it actually almost decreases the chances of this synthesis that you're talking about happening. Do you see any hopeful signs of this?
CA: 眼前这已经是个 炙手可热的领域了, 那你觉得现在是什么情况? 你感觉会发生什么? 因为人们可能会感觉受到了侵犯, 这样就会降低你刚说的结合的可能。 你可以从中看到一丝积极的信号吗? GM: 你提醒了我有一句 演讲里忘记讲的台词。
GM: You just reminded me of the one line I forgot from my talk. It's so interesting that Sundar, the CEO of Google, just actually also came out for global governance in the CBS "60 Minutes" interview that he did a couple of days ago. I think that the companies themselves want to see some kind of regulation. I think it’s a very complicated dance to get everybody on the same page, but I think there’s actually growing sentiment we need to do something here and that that can drive the kind of global affiliation I'm arguing for.
谷歌的 CEO 孙达尔(Sundar) 前几天还为全球治理 上了 CBS 的《60 分钟》访谈。 我认为这些公司本身 也想看到某种形式的治理。 要让所有人统一战线 是个艰巨的任务, 但是“我们得做些什么”的 情绪确实在高涨, 这也会促成我所倡导的国际联盟。
CA: I mean, do you think the UN or nations can somehow come together and do that or is this potentially a need for some spectacular act of philanthropy to try and fund a global governance structure? How is it going to happen?
CA: 你觉得联合国或者各个国家 有没有可能会一起为此努力, 还是这需要某种出于慈善的壮举, 做出尝试,出资建立起 一个全球的治理体系? 我们会怎么做呢?
GM: I'm open to all models if we can get this done. I think it might take some of both. It might take some philanthropists sponsoring workshops, which we're thinking of running, to try to bring the parties together. Maybe UN will want to be involved, I've had some conversations with them. I think there are a lot of different models and it'll take a lot of conversations.
GM: 如果能实现这个目标, 我可以接受任何模式。 我觉得可能会两者兼有。 可能需要一些慈善人士资助工作坊, 我们也在考虑组织这样的活动, 让各方都聚集在一起。 也许联合国也想加入, 我和他们已经谈过几次了。 我觉得有很多可选的模式, 也需要很多沟通。
CA: Gary, thank you so much for your talk.
CA: 盖瑞,感谢你的演讲。
GA: Thank you so much.
GM: 谢谢。