I'm going to talk about a failure of intuition that many of us suffer from. It's really a failure to detect a certain kind of danger. I'm going to describe a scenario that I think is both terrifying and likely to occur, and that's not a good combination, as it turns out. And yet rather than be scared, most of you will feel that what I'm talking about is kind of cool.
我想講下 一個好多人都經歷過嘅感官錯覺 當呢個錯覺嚟嗰陣 我哋會唔識得留意危險 我亦都想講下一個我認為駭人聽聞 同時又好有可能會發生嘅情景 呢個情景發生嘅話,唔係一件好事嚟 你哋可能唔覺得我依家講緊嘅嘢恐怖 反而覺得好型
I'm going to describe how the gains we make in artificial intelligence could ultimately destroy us. And in fact, I think it's very difficult to see how they won't destroy us or inspire us to destroy ourselves. And yet if you're anything like me, you'll find that it's fun to think about these things. And that response is part of the problem. OK? That response should worry you. And if I were to convince you in this talk that we were likely to suffer a global famine, either because of climate change or some other catastrophe, and that your grandchildren, or their grandchildren, are very likely to live like this, you wouldn't think, "Interesting. I like this TED Talk."
所以我想講下 我哋人類喺人工智能方面取得嘅成就 最终會點樣摧毀我哋 而事實上,我認為好難會見到 佢哋唔會摧毀我哋 或者導致我哋自我毀滅 依家你哋或者同我一樣 覺得諗呢啲嘢好得意 正因為覺得得意 亦都成為咗問題嘅一部份 你哋應該擔心你哋嘅反應至真! 如果我喺呢場演講度話畀你哋聽 因為氣候變化或者大災難嘅原因 我哋會遭遇一場饑荒 而你嘅孫,或者佢哋嘅孫 會好似咁樣生活 你就唔會覺得 「好有趣,我鍾意呢場 TED 演講。」
Famine isn't fun. Death by science fiction, on the other hand, is fun, and one of the things that worries me most about the development of AI at this point is that we seem unable to marshal an appropriate emotional response to the dangers that lie ahead. I am unable to marshal this response, and I'm giving this talk.
饑荒一啲都唔有趣 但科幻小說描繪嘅死亡就好有趣 呢一刻,人工智能發展 最令我最困擾嘅係 我哋面對近在眼前嘅危險似乎無動於衷 雖然我喺你哋面前演講 但我同你哋一樣都係冇反應
It's as though we stand before two doors. Behind door number one, we stop making progress in building intelligent machines. Our computer hardware and software just stops getting better for some reason. Now take a moment to consider why this might happen. I mean, given how valuable intelligence and automation are, we will continue to improve our technology if we are at all able to. What could stop us from doing this? A full-scale nuclear war? A global pandemic? An asteroid impact? Justin Bieber becoming president of the United States?
成件事就好似我哋企喺兩道門前面 喺一號門後面,我哋唔再發展智能機器 因為某啲原因 我哋電腦嘅硬件同軟件都停滯不前 依家嚟諗一下點解呢種情況會發生 即係話,因為智能同自動化好重要 所以我哋會喺許可嘅情況之下 繼續改善科技 咁究竟係乜嘢會阻止我哋? 一個全面嘅核戰爭? 一個全球流行病? 一個小行星撞擊? Justin Bieber 做咗美國總統?
(Laughter)
(笑聲)
The point is, something would have to destroy civilization as we know it. You have to imagine how bad it would have to be
之但係,如我哋所知 有一啲嘢會摧毀文明
to prevent us from making improvements in our technology permanently, generation after generation. Almost by definition, this is the worst thing that's ever happened in human history.
你必須要想像 如果我哋一代又一代人 永遠改善唔到科技 情況會有幾嚴重 幾乎可以確定嘅係 呢個係人類史上最壞嘅事 所以唯一嘅選擇 就係二號門後嘅做法
So the only alternative, and this is what lies behind door number two, is that we continue to improve our intelligent machines year after year after year. At a certain point, we will build machines that are smarter than we are, and once we have machines that are smarter than we are, they will begin to improve themselves. And then we risk what the mathematician IJ Good called an "intelligence explosion," that the process could get away from us.
我哋繼續年復一年升級改造智能機器 到咗某個地步 我哋就會整出比我哋仲要聰明嘅機器 一旦我哋有咗比我哋自己 仲聰明嘅機器 佢哋就會自我改良 到時我哋就會面臨數學家 IJ Good 講嘅「智能爆炸」危機 即係話,改良過程唔再需要人類
Now, this is often caricatured, as I have here, as a fear that armies of malicious robots will attack us. But that isn't the most likely scenario. It's not that our machines will become spontaneously malevolent. The concern is really that we will build machines that are so much more competent than we are that the slightest divergence between their goals and our own could destroy us.
依家,經常會有人學呢張諷刺漫畫咁 描繪叛變嘅機器人會攻擊我哋 但係呢個唔係最有可能發生嘅情景 我哋嘅機器唔會自動變惡 所以問題在於我哋製造出 比我哋更加做到嘢嘅機器嘅時候 佢哋目標上同我哋嘅細微分歧 會置我哋於死地
Just think about how we relate to ants. We don't hate them. We don't go out of our way to harm them. In fact, sometimes we take pains not to harm them. We step over them on the sidewalk. But whenever their presence seriously conflicts with one of our goals, let's say when constructing a building like this one, we annihilate them without a qualm. The concern is that we will one day build machines that, whether they're conscious or not, could treat us with similar disregard.
就諗下我哋同螞蟻之間嘅關係︰ 我哋唔討厭佢哋 我哋唔會傷害佢哋 甚至我哋為咗唔傷害佢哋 而會受一啲苦 例如我哋會為咗唔踩到佢哋 而跨過佢哋 但係一旦佢哋嘅存在 同我哋嘅其中一個目標有嚴重衝突 譬如話要起一棟咁樣嘅樓 我哋諗都唔諗就殺死佢哋 問題係,我哋終有一日整出嘅機器—— 無論佢哋自己有冇意識都好 同樣會冷漠咁對待我哋
Now, I suspect this seems far-fetched to many of you. I bet there are those of you who doubt that superintelligent AI is possible, much less inevitable. But then you must find something wrong with one of the following assumptions. And there are only three of them.
依家,我估對於你哋大部份人嚟講 呢件情景都係遙不可及嘅 我賭你哋當中有人質疑 超級智能嘅可能性 更加唔好講 人類要避免超級智能 但係你哋肯定會喺下面嘅假設當中 搵到一啲謬誤 呢度一共有三個假設
Intelligence is a matter of information processing in physical systems. Actually, this is a little bit more than an assumption. We have already built narrow intelligence into our machines, and many of these machines perform at a level of superhuman intelligence already. And we know that mere matter can give rise to what is called "general intelligence," an ability to think flexibly across multiple domains, because our brains have managed it. Right? I mean, there's just atoms in here, and as long as we continue to build systems of atoms that display more and more intelligent behavior, we will eventually, unless we are interrupted, we will eventually build general intelligence into our machines.
喺物理系統裏面,智能等如訊息處理 但係,呢個超過咗假設 因為我哋已經喺我哋嘅機器裏面 植入咗弱人工智能 而且呢啲機器好多 已經處於一個超人類智能水平 同時我哋知道僅僅係物質 就可以產生所謂嘅「一般智能」 一種可以喺唔同領域之間 靈活思考嘅能力 咁係因為我哋嘅大腦 已經可以做到,係唔係? 我嘅意思係,大腦凈係得原子 只要我哋繼續加設原子系統 機器就可以有更加多智能行為 除非進度有咩停頓 否則我哋最終會喺機器裏面 建構出一般智能 明白進度嘅快慢並唔影響係好重要
It's crucial to realize that the rate of progress doesn't matter, because any progress is enough to get us into the end zone. We don't need Moore's law to continue. We don't need exponential progress. We just need to keep going.
因為任何過程都足以令我哋返唔到轉頭 我哋唔需要按照摩爾定律進行 我哋唔需要指數式增長 我哋只需要繼續做
The second assumption is that we will keep going. We will continue to improve our intelligent machines. And given the value of intelligence -- I mean, intelligence is either the source of everything we value or we need it to safeguard everything we value. It is our most valuable resource. So we want to do this. We have problems that we desperately need to solve. We want to cure diseases like Alzheimer's and cancer. We want to understand economic systems. We want to improve our climate science. So we will do this, if we can. The train is already out of the station, and there's no brake to pull.
第二個假設就係我哋會繼續做 我哋會繼續改造我哋嘅智能機器 而考慮到智能嘅價值… 我係話,因為有智能 我哋至會珍重事物 或者我哋需要智能 去保護我哋珍重嘅一切 智能係我哋最有寶貴嘅資源 所以我哋想繼續發展智能 我哋有極需解決嘅問題 例如我哋想治療類似阿茲海默症 同癌症嘅疾病 我哋想認識經濟系統 我哋想改善我哋嘅氣候科學 所以如果可以做到嘅話 我哋會繼續發展智能 件事亦都可以比喻為︰ 列車已經開出,但冇刹車掣可以踩
Finally, we don't stand on a peak of intelligence, or anywhere near it, likely. And this really is the crucial insight. This is what makes our situation so precarious, and this is what makes our intuitions about risk so unreliable.
最終,我哋唔會去到 智能嘅頂峰或者高智能水平 而呢個就係非常重要嘅觀察結果 就係呢個結果 將我哋置於岌岌可危嘅境地 亦令到我哋對於危險嘅觸覺唔可靠
Now, just consider the smartest person who has ever lived. On almost everyone's shortlist here is John von Neumann. I mean, the impression that von Neumann made on the people around him, and this included the greatest mathematicians and physicists of his time, is fairly well-documented. If only half the stories about him are half true, there's no question he's one of the smartest people who has ever lived. So consider the spectrum of intelligence. Here we have John von Neumann. And then we have you and me. And then we have a chicken.
依家,就諗下史上最聰明嘅人 幾乎喺每個人嘅名單上面 都會有 John von Neumann 我嘅意思係 John von Neumann 畀佢周圍嘅人嘅印象 包括佢畀嗰個時代最犀利嘅數學家 同物理學家嘅印象 都係有紀錄低嘅 如果一半關於佢嘅故事有一半係真嘅 咁毫無疑問 佢係有史以來其中一個最聰明嘅人 所以當我哋畫一幅比較智力嘅圖 喺右邊高分嘅位置 我哋有 John von Neumann 喺中間有你同我 去到最左邊,我哋有雞仔
(Laughter)
(笑聲)
Sorry, a chicken.
係吖,就係一隻雞仔
(Laughter)
(笑聲)
There's no reason for me to make this talk more depressing than it needs to be.
我冇理由將呢個演講搞到咁灰㗎
(Laughter)
(笑聲)
It seems overwhelmingly likely, however, that the spectrum of intelligence extends much further than we currently conceive, and if we build machines that are more intelligent than we are, they will very likely explore this spectrum in ways that we can't imagine, and exceed us in ways that we can't imagine.
但好有可能智力分佈 遠比我哋目前認知嘅廣 如果我哋建造出 比我哋擁有更高智慧嘅機器 佢哋嘅智力好有可能會 超越我哋認知嘅最高智力 同埋以無法想像嘅方式超越我哋
And it's important to recognize that this is true by virtue of speed alone. Right? So imagine if we just built a superintelligent AI that was no smarter than your average team of researchers at Stanford or MIT. Well, electronic circuits function about a million times faster than biochemical ones, so this machine should think about a million times faster than the minds that built it. So you set it running for a week, and it will perform 20,000 years of human-level intellectual work, week after week after week. How could we even understand, much less constrain, a mind making this sort of progress?
同樣重要嘅係 單憑運算速度就可以超越我哋 啱唔啱?諗下如果我哋整咗一個 冇哈佛或者麻省理工研究人員 咁聰明嘅超級人工智能 但電路運行速度大概 比生化電路快一百萬倍 所以呢個機器嘅思考速度應該會 比佢嘅創造者快大概一百萬倍 所以如果佢運行一個星期 佢就可以完成人類要兩萬年 先至完成得到嘅工作 而我哋又點會明白 人工智能係點樣完成咁龐大嘅運算呢?
The other thing that's worrying, frankly, is that, imagine the best case scenario. So imagine we hit upon a design of superintelligent AI that has no safety concerns. We have the perfect design the first time around. It's as though we've been handed an oracle that behaves exactly as intended. Well, this machine would be the perfect labor-saving device. It can design the machine that can build the machine that can do any physical work, powered by sunlight, more or less for the cost of raw materials. So we're talking about the end of human drudgery. We're also talking about the end of most intellectual work.
另一個令人擔憂嘅事,老實講 就係…不如想像一下最好嘅情形 想像一下我哋設計咗一個 冇安全問題嘅超級人工智能 我哋第一次擁有完美嘅設計 就好似我哋摞住 一個按照預期發展嘅神諭 呢個機器仲會變成完美嘅慳力設備 事關機器可以生產另一款機器出嚟 做任何體力勞動 兼由太陽能驅動 成本仲同買原材料差唔多 所以,我哋唔單止講緊咕哩勞力嘅終結 我哋同時講緊大部份用腦工作嘅終結
So what would apes like ourselves do in this circumstance? Well, we'd be free to play Frisbee and give each other massages. Add some LSD and some questionable wardrobe choices, and the whole world could be like Burning Man.
咁我哋人類面對工作削減 應該何去何從? 我哋會好自由咁去掟飛盤 、同人按摩 服食一啲 LSD 精神藥 同埋著上怪異服飾 於是成個世界都會變成火人節嘅人咁
(Laughter)
(笑聲)
Now, that might sound pretty good, but ask yourself what would happen under our current economic and political order? It seems likely that we would witness a level of wealth inequality and unemployment that we have never seen before. Absent a willingness to immediately put this new wealth to the service of all humanity, a few trillionaires could grace the covers of our business magazines while the rest of the world would be free to starve.
頭先講到嘅嘢聽起上嚟好似好好咁 但係撫心自問 面對目前嘅經濟政治秩序 乜嘢會發生呢? 似乎我哋會目睹 我哋從未見過咁嚴重嘅 貧富懸殊同失業率 如果呢筆新財富唔即時用嚟服務全人類 就算一啲億萬富翁使好多錢 㨘靚商業雜誌嘅封面 世界上其他人都要挨餓
And what would the Russians or the Chinese do if they heard that some company in Silicon Valley was about to deploy a superintelligent AI? This machine would be capable of waging war, whether terrestrial or cyber, with unprecedented power. This is a winner-take-all scenario. To be six months ahead of the competition here is to be 500,000 years ahead, at a minimum. So it seems that even mere rumors of this kind of breakthrough could cause our species to go berserk.
咁如果俄羅斯人或者中國人 聽到矽谷嘅一啲公司 打算使用一個超級人工智能 佢哋會點諗? 呢個機器有能力用未見過嘅力度 發動地面或者網絡戰爭 呢個係「勝者全取」嘅情況 喺呢場人工智能較量中有六個月嘅優勢 就係至少要做多人類五十萬年做到嘅嘢 甚至只係關於人工智能突破嘅謠言 就可以令到人類亂起上嚟
Now, one of the most frightening things, in my view, at this moment, are the kinds of things that AI researchers say when they want to be reassuring. And the most common reason we're told not to worry is time. This is all a long way off, don't you know. This is probably 50 or 100 years away. One researcher has said, "Worrying about AI safety is like worrying about overpopulation on Mars." This is the Silicon Valley version of "don't worry your pretty little head about it."
依家最驚人嘅一件事,我覺得 就係人工智能研究人員 安定人心時講嘅說話 佢哋成日話,因為我哋有時間 所以我哋唔需要擔心 「乜你唔知有排咩? 仲有五十年或者一百年先到。」 一位研究人員曾經咁講︰ 「擔心人工智能嘅安全就好似 擔心火星人口爆棚一樣。」 呢句嘢等如矽谷同你講︰ 「你十八廿二就杞人憂天!」
(Laughter)
(笑聲)
No one seems to notice that referencing the time horizon is a total non sequitur. If intelligence is just a matter of information processing, and we continue to improve our machines, we will produce some form of superintelligence. And we have no idea how long it will take us to create the conditions to do that safely. Let me say that again. We have no idea how long it will take us to create the conditions to do that safely.
冇人意識到 攞時間嚟到講完全係無稽之談 如果智能凈係同處理訊息有關 同埋我哋繼續改良我哋嘅機器嘅話 我哋最終會生產到超級智能 但我哋唔知道要用幾長時間 先可以生產安全嘅超級智能 等我再講多一次 我哋唔知道要用幾長時間 先可以生產安全嘅超級智能
And if you haven't noticed, 50 years is not what it used to be. This is 50 years in months. This is how long we've had the iPhone. This is how long "The Simpsons" has been on television. Fifty years is not that much time to meet one of the greatest challenges our species will ever face. Once again, we seem to be failing to have an appropriate emotional response to what we have every reason to believe is coming.
如果你仲未意識到 五十年嘅概念已經唔同咗喇 呢幅圖顯示咗以月份計嘅五十年 先係 iPhone 面世至今嘅時間 再係阿森一族出現係電視上嘅時間 五十年不足以畀人類應對最大挑戰 再一次,我哋對於有理由發生嘅事 未有採取適當嘅情緒反應
The computer scientist Stuart Russell has a nice analogy here. He said, imagine that we received a message from an alien civilization, which read: "People of Earth, we will arrive on your planet in 50 years. Get ready." And now we're just counting down the months until the mothership lands? We would feel a little more urgency than we do.
對此,電腦科學家 Stuart Russell 有一個好嘅比喻 佢話︰想像一下我哋收到 一個來自外星文明嘅信息 上面寫住: 「地球上嘅人類, 我哋五十年之後會到達你哋嘅星球。 請準備好。」 咁我哋依家凈係會倒數外星人來臨? 我哋應該更加緊張至係
Another reason we're told not to worry is that these machines can't help but share our values because they will be literally extensions of ourselves. They'll be grafted onto our brains, and we'll essentially become their limbic systems. Now take a moment to consider that the safest and only prudent path forward, recommended, is to implant this technology directly into our brains. Now, this may in fact be the safest and only prudent path forward, but usually one's safety concerns about a technology have to be pretty much worked out before you stick it inside your head.
另一個我哋被告知唔使擔心嘅原因係 呢啲機器只會識得 將我哋嘅價值觀傳開 因為佢哋係我哋人類嘅附屬嘅一部分 但同時佢哋會被植入我哋嘅大腦 所以我哋會成為佢哋嘅邊緣系統 依家使啲時間諗下 最安全同唯一審慎嘅做法 而推薦嘅做法就係 直接將呢種科技植入我哋嘅大腦 呢種做法可能係最安全同唯一審慎嘅 但係喺你將佢植入你個腦之前 科技嘅安全問題需要解決
(Laughter)
(笑聲)
The deeper problem is that building superintelligent AI on its own seems likely to be easier than building superintelligent AI and having the completed neuroscience that allows us to seamlessly integrate our minds with it. And given that the companies and governments doing this work are likely to perceive themselves as being in a race against all others, given that to win this race is to win the world, provided you don't destroy it in the next moment, then it seems likely that whatever is easier to do will get done first.
更深一層嘅問題係 人工智能自己整超級人工智能 似乎比整一個可以喺神經科學上 同我哋腦部無縫接合嘅 超級人工智能簡單 考慮到從事研發人工智能嘅公司 同政府好可能會互相競爭 考慮到要贏呢場比賽就要贏成個世界 同埋先假設如果你下一刻 唔會糟塌人工智能嘅成果 咁樣,似乎更加簡單嘅事會完成咗先
Now, unfortunately, I don't have a solution to this problem, apart from recommending that more of us think about it. I think we need something like a Manhattan Project on the topic of artificial intelligence. Not to build it, because I think we'll inevitably do that, but to understand how to avoid an arms race and to build it in a way that is aligned with our interests. When you're talking about superintelligent AI that can make changes to itself, it seems that we only have one chance to get the initial conditions right, and even then we will need to absorb the economic and political consequences of getting them right.
但唔好彩嘅係 我除咗叫大家反思呢個問題 我就再冇辦法解決呢個問題 我覺得我哋喺人工智能方面 需要好似「曼哈頓計劃」咁嘅計劃 唔係講點樣整人工智能 因為我認為人工智能終有一日會整到 而係搞清楚點樣避免一場軍備競賽 同埋往符合我哋利益嘅方向 發展人工智能 當你講緊可以自我改造嘅超級人工智能 我哋似乎只有一個機會 令到人工智能發展得安全 就算發展得安全 我哋都要接受 人工智能對經濟同政治產生嘅結果
But the moment we admit that information processing is the source of intelligence, that some appropriate computational system is what the basis of intelligence is, and we admit that we will improve these systems continuously, and we admit that the horizon of cognition very likely far exceeds what we currently know, then we have to admit that we are in the process of building some sort of god. Now would be a good time to make sure it's a god we can live with.
但係當我哋同意 訊息處理係智能嘅起步點 同意一啲適當嘅計算系統係智能嘅基礎 同意我哋會不斷完善人工智能 同意將來有好多嘢超越我哋認知嘅 咁我哋就必須要承認 我哋正喺度創造緊某種神明 依家會係一個好時機 確保佢係可以同我哋共存嘅神明
Thank you very much.
好多謝你哋
(Applause)
(掌聲)