Competition. It's a fundamental part of human nature. I was a professional poker player for 10 years, so I've very much seen all the good, bad and ugly ways it can manifest. When it's done right, it can drive us to incredible feats in sports and innovation, like when car companies compete over who can build the safest cars or the most efficient solar panels. Those are all examples of healthy competition, because even though individual companies might come and go, in the long run, the game between them creates win-win outcomes where everyone benefits in the end.
竞争。 它是人性的基本组成部分。 我从事职业扑克牌手已有 10 年了, 所以我已经亲眼目睹了它所展现的 所有好、坏和丑恶一面。 如果用在正道, 它可以推动我们在体育和 创新方面取得丰功伟绩, 就像汽车公司 竞争制造最安全的汽车 或最高效的太阳能电池板。 这些都是良性竞争的例子, 因为尽管个别公司可能会是过客, 但从长远来看, 它们之间的博弈会创造双赢的结果, 最终让各方受益。
But sometimes competition is not so great and can create lose-lose outcomes where everyone's worse off than before. Take these AI beauty filters, for example. As you can see, they're a very impressive technology. They can salvage almost any picture. They can even make Angelina and Margot more beautiful. So they're very handy, especially for influencers who, now, at the click of a button, can transform into the most beautiful Hollywood versions of themselves. But handy doesn't always mean healthy. And I've personally noticed how quickly these things can train you to hate your natural face. And there's growing evidence that they're creating issues like body dysmorphia, especially in young people.
但有时候竞争并不是那么有益, 可能会造成双输的结果, 每个人的境况都比以前更糟。 以这些 AI 美颜滤镜为例。 如你所见, 这是一项非常厉害的技术。 它几乎可以拯救任何照片。 它甚至可以让安吉丽娜 和玛格特更加美丽。 它非常方便, 尤其是对网红来说,一键 就可以变成自己 最漂亮的好莱坞版本。 但是方便并不总是代表健康。 我亲身体验了这些东西 能有多快训练你 讨厌你的天然长相。 越来越多的证据表明, 它们正在造成 容貌焦虑等问题, 尤其是在年轻人中。
Nonetheless, these things are now endemic to social media because the nature of the game demands it. The platforms are incentivized to provide them because hotter pictures means more hijacked limbic systems, which means more scrolling and thus more ad revenue. And users are incentivized to use them because hotter pictures get you more followers.
尽管如此,这些东西 目前仅限于社交媒体中, 出于竞争的本质。 这些平台兴致勃勃地 提供这样的服务, 因为更火爆的图片意味着 更能抓住你的兴趣、 更多的滑动屏幕、 更多的广告收入。 用户兴致勃勃地使用它们, 因为更火爆的图片可以 让你获取更多的粉丝。
But this is a trap, because once you start using these things, it's really hard to go back. Plus, you don't even get a competitive advantage from them anymore because everyone else is already using them too. So influencers are stuck using these things with all the downsides and very little upside. A lose-lose game.
但这是一个陷阱, 因为你一旦开始使用这些东西, 就很难回头。 而且你甚至无法再从中 获得竞争优势, 因为其他人也在使用它们。 网红离不开这些滤镜, 全是负面影响, 几乎没有正面影响。 这是一场双输的游戏。
A similar kind of trap is playing out in our news media right now, but with much worse consequences. You'd think since the internet came along that the increased competition between news outlets would create a sort of positive spiral, like a race to the top of nuanced, impartial, accurate journalism. Instead, we're seeing a race to the bottom of clickbait and polarization, where even respectable papers are increasingly leaning into these kind of low-brow partisan tactics. Again, this is due to crappy incentives.
我们的新闻媒体 正在落入类似的陷阱, 但后果要恶劣得多。 你可能会认为 自互联网问世以来, 新闻媒体之间竞争的加剧 将形成一种积极的螺旋式上升, 比如争相向细致、公正、 准确的新闻报道的顶峰进发。 相反,我们看到的是争相 向标题党和“二极管”的低谷进发, 连颇有名望的报纸都逐渐 采取这种令人不齿的 带有倾向性的策略。 同样,这是因为激励措施不当。
Today, we no longer just read our news. We interact with it by sharing and commenting. And headlines that trigger emotions like fear or anger are far more likely to go viral than neutral or positive ones. So in many ways, news editors are in a similar kind of trap as the influencers, where, the more their competitors lean into clickbaity tactics, the more they have to as well. Otherwise, their stories just get lost in the noise. But this is terrible for everybody, because now the media get less trust from the public, but also it becomes harder and harder for anyone to discern truth from fiction, which is a really big problem for democracy.
我们现在不只是“看”新闻。 我们还会通过分享、评论与它互动。 引发恐惧或愤怒等情绪的头条新闻 比中立或正面的头条新闻 更有可能会被广泛传播。 某种程度新闻编辑陷入的陷阱 和网红差不多, 他们的竞争对手 越采取标题党策略, 他们也不得不这么做。 否则,他们的故事 就会被纷纷扰扰淹没。 但这对每个人来说都很糟糕, 因为现在公众对媒体的信任越来越少, 而且所有人都越来越难以分辨真伪, 这对民主来说是一个非常大的问题。
Now, this process of competition gone wrong is actually the driving force behind so many of our biggest issues. Plastic pollution, deforestation, antibiotic overuse in farming, arms races, greenhouse gas emissions. These are all a result of crappy incentives, of poorly designed games that push their players -- be them people, companies or governments -- into taking strategies and tactics that defer costs and harms to the future. And what's so ridiculous is that most of the time, these guys don't even want to be doing this. You know, it's not like packaging companies want to fill the oceans with plastic or farmers want to worsen antibiotic resistance. But they’re all stuck in the same dilemma of: "If I don't use this tactic, I’ll get outcompeted by all the others who do. So I have to do it, too.”
竞争过程出错 其实在背后推动着 我们的许多重大问题。 塑料污染、 森林砍伐、 农业过度使用抗生素、 军备竞赛、 温室气体排放。 这些都是激励措施不当的后果, 竞争游戏设计不当, 迫使玩家—— 个人、公司或政府, 采取策略和战术 把成本和伤害推迟到以后再还。 可笑的是,大多数时候, 连这些人自己都不想这样做。 并不是包装公司 想用塑料填满海洋, 也不是农民想加剧抗生素耐药性。 但是他们都陷入了同样的困境: “如果我不采取这种策略, 我就会被所有这么干的人击败。 所以我也必须这样做。”
This is the mechanism we need to fix as a civilization. And I know what you're probably all thinking, "So it's capitalism." No, it's not capitalism. Which, yes, can cause problems, but it can also solve them and has been fantastic in general. It's something much deeper. It's a force of misaligned incentives of game theory itself.
这是我们整个文明需要修复的机制。 我知道你们可能在想什么, “这就是资本主义。” 不,这不是资本主义。 是的,这可能会导致问题, 但它也可以解决问题, 总体来说很不错。 这是更深层次的东西。 这是博弈论本身的不当激励。
So a few years ago, I retired from poker, in part because I wanted to understand this mechanism better. Because it takes many different forms, and it goes by many different names. These are just some of those names. You can see they're a little bit abstract and clunky, right? They don't exactly roll off the tongue. And given how insidious and connected all of these problems are, it helps to have a more visceral way of recognizing them.
几年前,我从扑克届退役, 部分原因是我想更好地理解这种机制。 因为它有许多不同的形式, 还有许多不同的名称。 这只是其中的一些名字。 你可以看出它们 有点抽象、笨重,对吧? 它们并不完全是胡说八道。 由于这些问题的隐藏性和关联性, 用一种更直观的方式 理解它们会更好一些。
So this is probably the only time you're going to hear about the Bible at this conference. But I want to tell you a quick story from it, because allegedly, back in the Canaanite days, there was a cult who wanted money and power so badly, they were willing to sacrifice their literal children for it. And they did this by burning them alive in an effigy of a God that they believed would then reward them for this ultimate sacrifice. And the name of this god was Moloch. Bit of a bummer, as stories go. But you can see why it's an apt metaphor, because sometimes we get so lost in winning the game right in front of us, we lose sight of the bigger picture and sacrifice too much in our pursuit of victory. So just like these guys were sacrificing their children for power, those influencers are sacrificing their happiness for likes. Those news editors are sacrificing their integrity for clicks, and polluters are sacrificing the biosphere for profit.
这可能是 你在这场会议中 唯一一次听到圣经。 但我想简短地讲一个故事, 因为据称,在迦南时代, 有一个邪教组织非常渴望金钱和权力, 他们愿意为此牺牲自己的子女。 为此,他们在神像前 活活烧死自己的孩子, 他们相信神会奖励他们做出 这一终极牺牲。 这尊神的名字叫摩洛克(Moloch)。 故事讲下去就有点扫兴了。 但是你会明白 为什么这是一个恰当的比喻, 因为有时我们太过沉迷于 赢得眼前的比赛, 而在追求胜利时 忽视了大局,牺牲了太多。 就像这些人为了权力牺牲孩子一样, 那些网红为了点赞牺牲了快乐。 那些新闻编辑为了点击量 牺牲了自己的诚信, 污染者则为了利润 牺牲了生物圈。
In all these examples, the short-term incentives of the games themselves are pushing, they're tempting their players to sacrifice more and more of their future, trapping them in a death spiral where they all lose in the end. That's Moloch's trap. The mechanism of unhealthy competition. And the same is now happening in the AI industry.
在这些例子中, 游戏本身的短期激励措施是在鼓动, 引诱玩家牺牲越来越多的未来, 陷入死亡螺旋中,最终全盘皆输。 那是摩洛克的陷阱。 不良竞争的机制。 AI 行业中也正在上演。
We're all aware of the race that's heating up between companies right now over who can score the most compute, who can get the biggest funding round or get the top talent. Well, as more and more companies enter this race, the greater the pressure for everyone to go as fast as possible and sacrifice other important stuff like safety testing. This has all the hallmarks of a Moloch trap. Because, like, imagine you're a CEO who, you know, in your heart of hearts, believes that your team is the best to be able to safely build extremely powerful AI. Well, if you go too slowly, then you run the risk of other, much less cautious teams getting there first and deploying their systems before you can. So that in turn pushes you to be more reckless yourself. And given how many different experts and researchers, both within these companies but also completely independent ones, have been warning us about the extreme risks of rushed AI, this approach is absolutely mad. Plus, almost all AI companies are beholden to satisfying their investors, a short-term incentive which, over time, will inevitably start to conflict with any benevolent mission.
我们都知道公司间的竞争正热火朝天, 竞争最高算力、 最高融资、最优人才。 随着越来越多公司加入竞争, 每个人都面临着更大的压力, 迫使他们尽可能快地动起来, 牺牲安全测试等重要事项。 这就是摩洛克陷阱的所有特征。 因为想象一下你是一位 CEO, 你内心深处 相信你的团队是 能够安全地打造 极其强大的 AI 的最佳人选。 如果你动作太慢,那你就会面临其他 不那么谨慎的团队先行一步, 在你之前部署他们的系统。 因此,这反过来又会 促使你自己更冲动行事。 有多少专家和研究人员, 包括这些公司的职员, 还有完全独立的人士, 一直在警告我们警惕 仓促发展的 AI 带来的极端风险, 这种做法简直太疯狂了。 此外,几乎所有 AI 公司 都必须满足投资者的要求, 这种短期激励措施随着时间的推移, 将不可避免地开始 与任何“圣母”使命发生冲突。
And this wouldn't be a big deal if this was really just toasters we're talking about here. But AI, and especially AGI, is set to be a bigger paradigm shift than the agricultural or industrial revolutions. A moment in time so pivotal, it's deserving of reverence and reflection, not something to be reduced to a corporate rat race of who can score the most daily active users. I'm not saying I know what the right trade-off between acceleration and safety is, but I do know that we'll never find out what that right trade-off is if we let Moloch dictate it for us.
如果我们在这里谈论的 真的只是面包机, 那也没什么大不了的。 但是,AI,尤其是 AGI (通用人工智能), 必将带来比农业 或工业革命更大的范式转变。 这是个关键时刻, 需要敬意和反思, 不能简化成企业间你死我活的竞争, 比拼谁能获取更多的日活跃用户。 我并不是说我知道 加速发展和安全之间 权衡取舍的正确答案, 但我确实知道, 如果我们受制于摩洛克, 我们将永远找不到 正确的权衡取舍。
So what can we do? Well, the good news is we have managed to coordinate to escape some of Moloch's traps before. We managed to save the ozone layer from CFCs with the help of the Montreal Protocol. We managed to reduce the number of nuclear weapons on Earth by 80 percent, with the help of the Strategic Arms Reduction Treaty in 1991. So smart regulation may certainly help with AI too, but ultimately, it's the players within the game who have the most influence on it. So we need AI leaders to show us that they're not only aware of the risks their technologies pose, but also the destructive nature of the incentives that they're currently beholden to. As their technological capabilities reach towards the power of gods, they're going to need the godlike wisdom to know how to wield them.
那我们能做些什么? 好消息是我们之前合力成功 逃脱了一些摩洛克陷阱。 通过《蒙特利尔议定书》, 我们从氯氟碳化物中拯救了臭氧层。 通过 1991 年的《削减和 限制进攻性战略武器条约》 我们将地球上的核武器数量 减少了 80%。 明智的监管措施当然 也可能会对 AI 有所帮助, 但归根结底, 对 AI 影响最大的是 游戏中的玩家。 因此,我们需要 AI 领袖向我们表明 他们不仅意识到了 其技术构成的风险, 还有他们目前所面临的 激励措施的破坏性。 当他们的技术能力 匹敌神明的力量时, 他们需要神一般的智慧 学习如何驾驭它们。
So it doesn't fill me with encouragement when I see a CEO of a very major company saying something like, "I want people to know we made our competitor dance." That is not the type of mindset we need here. We need leaders who are willing to flip Moloch's playbook, who are willing to sacrifice their own individual chance of winning for the good of the whole. Now, fortunately, the three leading labs are showing some signs of doing this. Anthropic recently announced their responsible scaling policy, which pledges to only increase capabilities once certain security criteria have been met. OpenAI have recently pledged to dedicate 20 percent of their compute purely to alignment research. And DeepMind have shown a decade-long focus of science ahead of commerce, like their development of AlphaFold, which they gave away to the science community for free. These are all steps in the right direction, but they are still nowhere close to being enough. I mean, most of these are currently just words, they're not even proven actions.
因此,当我看到一家 大型公司的 CEO 说 “我想让人们知道对手对我们言听计从” 之类的话时,我并不觉得信心倍增。 这不是我们需要的那种心态。 我们需要愿意 推翻摩洛克故事的领导者, 他们愿意为了整体利益 牺牲自己的获胜机会。 所幸有三家头部实验室 都或多或少展现了这种迹象。 Anthropic 最近宣布了 负责任的扩展政策, 该政策承诺只有在满足 特定安全标准后才会提升性能。 OpenAI 最近承诺 将其 20% 的算力 完全用于“对齐”研究。 DeepMind 已经在过去十年中表现出 对科学的关注优先于商业, 例如他们开发的 AlphaFold, 将其免费赠与科学界。 这些都是朝着正确方向迈出的步伐, 但还远远不够。 我想说目前大都只是言语, 甚至不是经过验证的行动。
So we need a clear way to turn the AI race into a definitive race to the top.
我们需要一种明确的方法
Perhaps companies can start competing over who can be within these metrics, over who can develop the best security criteria. A race of who can dedicate the most compute to alignment. Now that would truly flip the middle finger to Moloch.
将 AI 竞赛转变为明确向上的竞争。 也许各公司可以开始 争夺谁可以达到这些指标, 谁可以制定出最佳的安全标准。 竞赛谁能将最多的算力投入到对齐上。 这确实是与摩洛克陷阱 背道而驰的做法。
Competition can be an amazing tool, provided we wield it wisely. And we're going to need to do that because the stakes we are playing for are astronomical. If we get AI, and especially AGI, wrong, it could lead to unimaginable catastrophe. But if we get it right, it could be our path out of many of these Moloch traps that I've mentioned today. And as things get crazier over the coming years, which they're probably going to, it's going to be more important than ever that we remember that it is the real enemy here, Moloch. Not any individual CEO or company, and certainly not one another.
如果我们明智地运用竞争, 它会是一个强大的工具。 我们之所以需要这样做, 是因为我们手上的赌注是天文数字。 如果我们让 AI, 尤其是 AGI,跑偏了, 可能会导致难以想象的灾难。 但如果我们做对了, 可能就会是我们摆脱多次提及的 摩洛克陷阱的出路。 接下来几年, 情况会越来越疯狂, 确实可能如此, 我们必须记住摩洛克 才是我们真正的敌人。 不是哪个 CEO,哪家公司, 当然也不是彼此。
So don't hate the players, change the game.
不要憎恨玩家, 去改变游戏吧。
(Applause)
(掌声)