I work with a bunch of mathematicians, philosophers and computer scientists, and we sit around and think about the future of machine intelligence, among other things. Some people think that some of these things are sort of science fiction-y, far out there, crazy. But I like to say, okay, let's look at the modern human condition. (Laughter) This is the normal way for things to be.
我和一群數學家、 哲學家、 及電腦科學家一起工作。 我們坐在一起 思考機器智慧的未來。 以及其他問題。 有些人可能認為 這是科幻小說的範疇, 離我們很遙遠,很瘋狂。 但是我要說, 好,我們來看看 現代人類的狀況.... (觀眾笑聲) 這是人類的常態。
But if we think about it, we are actually recently arrived guests on this planet, the human species. Think about if Earth was created one year ago, the human species, then, would be 10 minutes old. The industrial era started two seconds ago. Another way to look at this is to think of world GDP over the last 10,000 years, I've actually taken the trouble to plot this for you in a graph. It looks like this. (Laughter) It's a curious shape for a normal condition. I sure wouldn't want to sit on it. (Laughter)
但如果我們仔細想想, 其實人類是剛剛才抵達地球的訪客 假設地球在一年前誕生, 人類這個物種則僅存在了10分鐘。 工業革命在2秒鐘前開始。 另外一個角度是 看看這一萬年來的GDP增長 我花了時間作了張圖表, 它長這個樣子 (觀眾笑聲) 對一個正常的狀態來說 這是個很有趣的形狀。 我可不想要坐在上面。 (觀眾笑聲)
Let's ask ourselves, what is the cause of this current anomaly? Some people would say it's technology. Now it's true, technology has accumulated through human history, and right now, technology advances extremely rapidly -- that is the proximate cause, that's why we are currently so very productive. But I like to think back further to the ultimate cause.
我們不禁問自己: 「是什麼造成了這種異態呢?」 有些人會說是科技 這是對的, 科技在人類歷史上不斷累積, 而現在科技正以飛快的速度進步。 這個是近因, 這也是為什麼 我們現在的生產力很高。 但是我想要進一步 回想到最終的原因
Look at these two highly distinguished gentlemen: We have Kanzi -- he's mastered 200 lexical tokens, an incredible feat. And Ed Witten unleashed the second superstring revolution. If we look under the hood, this is what we find: basically the same thing. One is a little larger, it maybe also has a few tricks in the exact way it's wired. These invisible differences cannot be too complicated, however, because there have only been 250,000 generations since our last common ancestor. We know that complicated mechanisms take a long time to evolve. So a bunch of relatively minor changes take us from Kanzi to Witten, from broken-off tree branches to intercontinental ballistic missiles.
看看這兩位非常傑出的紳士: 這位是坎茲先生 他掌握了200個詞彙, 這是一個難以置信的壯舉。 以及 愛德 維騰, 他掀起了第二次超弦革命。 如果我們往腦袋瓜裡面看, 這是我們看到的: 基本上是一樣的東西。 一個稍微大一點, 它可能有一些特別的連結方法。 但是這些無形的差異不會太複雜, 因為從我們共同的祖先以來, 只經過了25萬代。 我們知道複雜的機制 需要很長的時間演化。 因此 一些相對微小的變化 將我們從坎茲先生變成了維騰, 從撿起掉落的樹枝當武器 到發射洲際彈道飛彈
So this then seems pretty obvious that everything we've achieved, and everything we care about, depends crucially on some relatively minor changes that made the human mind. And the corollary, of course, is that any further changes that could significantly change the substrate of thinking could have potentially enormous consequences.
因此,顯而易見的是 至今我們所實現的所有事 以及我們關心的所有事物, 都取決於人腦中相對微小的改變。 由此而來的推論就是:在未來, 任何能顯著地改變思想基體的變化 都有可能會帶來巨大的後果。
Some of my colleagues think we're on the verge of something that could cause a profound change in that substrate, and that is machine superintelligence. Artificial intelligence used to be about putting commands in a box. You would have human programmers that would painstakingly handcraft knowledge items. You build up these expert systems, and they were kind of useful for some purposes, but they were very brittle, you couldn't scale them. Basically, you got out only what you put in. But since then, a paradigm shift has taken place in the field of artificial intelligence.
我的一些同事覺得我們即將發現 足以深刻的改變思想基體的科技 那就是超級機器智慧 以前的人工智慧是 將指令輸入到一個箱子裡。 你需要程式設計師 精心的將知識設計成指令。 你建立這些專門系統, 這些系統在某些特定的領域中有點用, 但是它們很生硬, 你無法延展這些系統。 基本上這些系統所輸出的東西 僅限於你事先輸入的範圍。 但是從那時起, 人工智慧的領域裡發生了模式的轉變。
Today, the action is really around machine learning. So rather than handcrafting knowledge representations and features, we create algorithms that learn, often from raw perceptual data. Basically the same thing that the human infant does. The result is A.I. that is not limited to one domain -- the same system can learn to translate between any pairs of languages, or learn to play any computer game on the Atari console. Now of course, A.I. is still nowhere near having the same powerful, cross-domain ability to learn and plan as a human being has. The cortex still has some algorithmic tricks that we don't yet know how to match in machines.
現在主要的課題是機器的學習。 因此,與其設計知識的表現及特點, 我們寫出具有學習原始感官數據 的能力的程式碼。 基本上和嬰兒所做的是一樣的。 結果就是不侷限於 某個領域的人工智慧 -- 同一個系統可以學習 在任何兩種語言之間翻譯 或者學著玩雅達利系統上 的任何一款遊戲。 現在當然 人工智慧到現在 還未能達到像人類一樣 具有強大的跨領域的學習能力。 人類大腦還具有一些運算技巧 我們不知道如何將這些技巧 複製到機器中。
So the question is, how far are we from being able to match those tricks? A couple of years ago, we did a survey of some of the world's leading A.I. experts, to see what they think, and one of the questions we asked was, "By which year do you think there is a 50 percent probability that we will have achieved human-level machine intelligence?" We defined human-level here as the ability to perform almost any job at least as well as an adult human, so real human-level, not just within some limited domain. And the median answer was 2040 or 2050, depending on precisely which group of experts we asked. Now, it could happen much, much later, or sooner, the truth is nobody really knows.
所以現在需要問的是: 我們還要多久才能 在機器裡面複製這些技巧? 在幾年前, 我們對世界頂尖的人工智慧專家 做了一次問卷調查, 想要看看他們的想法, 其中的一個題目是: "到哪一年你覺得 人類會有50%的機率 能夠達成人類級的人工智慧?" 我們把人類級的人工智慧 定義為有能力 將任何任務至少執行得 像一名成年人一樣好, 所以是真正的人類級別, 而不是僅限於某些領域。 而答案的中位數是2040或2050年 取決於我們問的專家屬於什麼群體。 當然,這個有可能過很久才實現, 也有可能提早實現 沒有人知道確切的時間。
What we do know is that the ultimate limit to information processing in a machine substrate lies far outside the limits in biological tissue. This comes down to physics. A biological neuron fires, maybe, at 200 hertz, 200 times a second. But even a present-day transistor operates at the Gigahertz. Neurons propagate slowly in axons, 100 meters per second, tops. But in computers, signals can travel at the speed of light. There are also size limitations, like a human brain has to fit inside a cranium, but a computer can be the size of a warehouse or larger. So the potential for superintelligence lies dormant in matter, much like the power of the atom lay dormant throughout human history, patiently waiting there until 1945. In this century, scientists may learn to awaken the power of artificial intelligence. And I think we might then see an intelligence explosion.
我們知道的是, 機器基體處理資訊能力的最終界限 比生物組織的界限要大的多。 這取決於物理原理。 一個生物神經元發出脈衝的頻率 可能在200赫茲,每秒200次。 但就算是現在的電晶體 都以千兆赫(GHz)的頻率運轉。 神經元在軸突中傳輸的速度 比較慢,頂多是每秒100公尺。 但在電腦裡面,信號是以光速傳播的。 另外還有尺寸的限制 就像人類的大腦 必需要放得進顱骨內。 但是一部電腦可以 跟倉庫一樣大,甚至更大。 因此超級智慧的潛能 現在正潛伏在物質裡面, 就像是原子能 在人類的歷史中一直潛伏著, 耐心的等著,一直到1945年。 在這個世紀內, 科學家有可能會 將人工智慧的力量喚醒。 屆時我覺得我們會 見證到智慧的大爆發。
Now most people, when they think about what is smart and what is dumb, I think have in mind a picture roughly like this. So at one end we have the village idiot, and then far over at the other side we have Ed Witten, or Albert Einstein, or whoever your favorite guru is. But I think that from the point of view of artificial intelligence, the true picture is actually probably more like this: AI starts out at this point here, at zero intelligence, and then, after many, many years of really hard work, maybe eventually we get to mouse-level artificial intelligence, something that can navigate cluttered environments as well as a mouse can. And then, after many, many more years of really hard work, lots of investment, maybe eventually we get to chimpanzee-level artificial intelligence. And then, after even more years of really, really hard work, we get to village idiot artificial intelligence. And a few moments later, we are beyond Ed Witten. The train doesn't stop at Humanville Station. It's likely, rather, to swoosh right by.
大部分的人,當他們在想 什麼是聰明什麼是愚笨的時候, 我想他們腦中浮現出的畫面 會是這樣的: 在一邊是村裡的傻子, 然後在另外一邊 是 愛德 維騰 或愛因斯坦, 或你喜歡的某位大師。 但是我覺得從人工智慧的觀點來看, 真正的畫面應該比較像這樣子: 人工智慧從這一點開始,零智慧 然後,在許多許多年 辛苦的研究以後, 我們可能可以達到 老鼠級的人工智慧, 它可以在凌亂的環境中找到路 就像一隻老鼠一樣。 然後,在更多年的辛苦研究 及投資了很多資源之後, 我們可能可以達到 黑猩猩級的人工智慧。 然後,在更加多年的辛苦研究之後, 我們達到村莊傻子級別的人工智慧。 然後過一小會兒後, 我們就超越了愛德維騰。 這列火車並不會 在人類村這一站就停車。 它比較可能會直接呼嘯而過。
Now this has profound implications, particularly when it comes to questions of power. For example, chimpanzees are strong -- pound for pound, a chimpanzee is about twice as strong as a fit human male. And yet, the fate of Kanzi and his pals depends a lot more on what we humans do than on what the chimpanzees do themselves. Once there is superintelligence, the fate of humanity may depend on what the superintelligence does. Think about it: Machine intelligence is the last invention that humanity will ever need to make. Machines will then be better at inventing than we are, and they'll be doing so on digital timescales. What this means is basically a telescoping of the future. Think of all the crazy technologies that you could have imagined maybe humans could have developed in the fullness of time: cures for aging, space colonization, self-replicating nanobots or uploading of minds into computers, all kinds of science fiction-y stuff that's nevertheless consistent with the laws of physics. All of this superintelligence could develop, and possibly quite rapidly.
這個具有深遠的寓意, 特別是在談到權力的問題。 舉例來說,黑猩猩很強壯 -- 以體重比例來說, 一隻黑猩猩 比一個健康的男性人類要強壯兩倍。 然而,坎茲和他朋友們的命運 則很大的部分取決於 人類的作為, 而非黑猩猩們自己的作為。 當超級智慧出現後, 人類的命運可能會取決於 超級智慧的作為。 想想看: 機器智慧將會是人類所需要作出的 最後一個發明。 從那之後機器將會比人類更會發明, 他們也將會在"數位時間"裡 做出這些事。 這意味著未來到來的時間將被縮短。 想想那些我們曾經想像過的瘋狂科技 人類可能在有足夠的時間下 可以發明出來: 防止衰老、殖民太空、 自行複製的奈米機器人, 或將我們的頭腦上載到電腦裡, 這一些僅存在科幻小說範疇, 但同時還是符合物理法則的東西 超級智慧有辦法開發出這些東西, 而且速度可能很快。
Now, a superintelligence with such technological maturity would be extremely powerful, and at least in some scenarios, it would be able to get what it wants. We would then have a future that would be shaped by the preferences of this A.I. Now a good question is, what are those preferences? Here it gets trickier. To make any headway with this, we must first of all avoid anthropomorphizing. And this is ironic because every newspaper article about the future of A.I. has a picture of this: So I think what we need to do is to conceive of the issue more abstractly, not in terms of vivid Hollywood scenarios.
這麼成熟的超級智慧 將會非常的強大, 最少在某些場景它將有辦法 得到它想要的東西。 這樣以來我們的未來就將會 被這個超級智慧的偏好所影響。 現在出現了一個好問題, 這些偏好是什麼呢? 這個問題更棘手。 要在這個領域往前走, 我們必須避免 將機器智慧擬人化(人格化)。 這一點很諷刺因為 每一篇關於未來的人工智慧 的報導都會有這張照片: 所以我覺得我們必須要 更抽象的來想像這個議題, 而非以好萊塢的鮮明場景來想像。
We need to think of intelligence as an optimization process, a process that steers the future into a particular set of configurations. A superintelligence is a really strong optimization process. It's extremely good at using available means to achieve a state in which its goal is realized. This means that there is no necessary connection between being highly intelligent in this sense, and having an objective that we humans would find worthwhile or meaningful.
我們需要把智慧看做是 一個優化的過程, 一個將未來指引到 特定的組態的過程。 一個超級智慧 是一個很強大的優化過程。 它將很會利用現有資源 去達到達成目標的狀態。 這意味著有著高智慧以及 擁有一個對人類來說 是有意義的目標之間 並沒有必然的聯繫。
Suppose we give an A.I. the goal to make humans smile. When the A.I. is weak, it performs useful or amusing actions that cause its user to smile. When the A.I. becomes superintelligent, it realizes that there is a more effective way to achieve this goal: take control of the world and stick electrodes into the facial muscles of humans to cause constant, beaming grins. Another example, suppose we give A.I. the goal to solve a difficult mathematical problem. When the A.I. becomes superintelligent, it realizes that the most effective way to get the solution to this problem is by transforming the planet into a giant computer, so as to increase its thinking capacity. And notice that this gives the A.I.s an instrumental reason to do things to us that we might not approve of. Human beings in this model are threats, we could prevent the mathematical problem from being solved.
假設我們給予人工智慧的目標 是讓人類笑。 當人工智慧比較弱時, 它會做出有用的或是好笑的動作 以讓使用者笑出來。 當人工智慧演化成超級智慧的時後, 它會體認到有更有效的方法 可以達到這個目標: 控制這個世界 然後在人類的臉部肌肉上連接電級 以使這個人不斷的微笑。 另外一個例子, 假設我們給人工智慧的目標是 解出一個非常困難的數學問題。 當人工智慧變成超級智慧時, 它會體認到最有效的方法是 把整個地球轉化成 一部超大號的電腦, 進而增加它自己的運算能力。 注意到這個模式 會給人工智慧理由去做 我們可能不認可的事情。 在這個模型裡面人類是威脅, 我們可能會在解開數學問題 的過程中成為阻礙。
Of course, perceivably things won't go wrong in these particular ways; these are cartoon examples. But the general point here is important: if you create a really powerful optimization process to maximize for objective x, you better make sure that your definition of x incorporates everything you care about. This is a lesson that's also taught in many a myth. King Midas wishes that everything he touches be turned into gold. He touches his daughter, she turns into gold. He touches his food, it turns into gold. This could become practically relevant, not just as a metaphor for greed, but as an illustration of what happens if you create a powerful optimization process and give it misconceived or poorly specified goals.
當然,在我們可預見的範圍內, 事情不會以這種方式出錯; 這些是誇大的例子。 但是它指出的概念很重要: 如果你創造了一個 非常強大的優化流程 要最大化目標X, 你最好確保你對目標X的定義 包含了所有你所在意的事情。 這也是在很多神話故事中 教導的寓意。 希臘神話中的米達斯國王希望 他碰到的所有東西都可以變成金子。 他碰到了他的女兒, 她變成了黃金。 他碰到了他的食物, 他的食物也變成了黃金。 這實際上跟我們的題目有關, 不僅僅是對貪婪的隱喻, 但也指出了如果你創造了 一個強大的優化流程 但同時給了它 不正確或不精確的目標後 會發生什麼事。
Now you might say, if a computer starts sticking electrodes into people's faces, we'd just shut it off. A, this is not necessarily so easy to do if we've grown dependent on the system -- like, where is the off switch to the Internet? B, why haven't the chimpanzees flicked the off switch to humanity, or the Neanderthals? They certainly had reasons. We have an off switch, for example, right here. (Choking) The reason is that we are an intelligent adversary; we can anticipate threats and plan around them. But so could a superintelligent agent, and it would be much better at that than we are. The point is, we should not be confident that we have this under control here.
你可能會說,如果電腦系統 開始在人臉上安裝電極, 我們可以直接把他關掉就好了。 一、這並不一定容易做到,如果我們 已經對這個系統產生依賴性 —— 比如:你知道網際網路的開關在哪裡嗎? 二、為什麼黑猩猩當初 沒有把人類的開關關掉? 或是尼安德特人? 他們有很明顯的理由要這麼做, 而我們的開關就在這裡: (窒息聲) 原因是人類是很聰明的敵人; 我們可以預見威脅 並為其做出準備。 但一個超級智慧也會, 而且它的能力將比我們強大的多。 我想要說的一點是,我們不應該 覺得一切都在我們的掌握之中。
And we could try to make our job a little bit easier by, say, putting the A.I. in a box, like a secure software environment, a virtual reality simulation from which it cannot escape. But how confident can we be that the A.I. couldn't find a bug. Given that merely human hackers find bugs all the time, I'd say, probably not very confident. So we disconnect the ethernet cable to create an air gap, but again, like merely human hackers routinely transgress air gaps using social engineering. Right now, as I speak, I'm sure there is some employee out there somewhere who has been talked into handing out her account details by somebody claiming to be from the I.T. department.
我們可能可以藉由 把AI放到一個盒子裡面 來給我們更多的掌握, 就像是一個獨立的軟體環境, 一個AI無法逃脫的虛擬實境。 但是我們有多大的信心 這個AI不會找到漏洞? 就算只是人類駭客, 他們還經常找出漏洞。 我想我們不是很有信心。 那所以我們把網路線拔掉, 製造一個物理間隙, 但同樣的,就算只是人類駭客 也經常可以利用社交工程陷阱 來突破物理間隙。 現在,在我在台上說話的同時 我確定在世界的某一個角落裡 有一名公司職員 才剛剛被自稱來自IT部門 的人士說服(詐騙) 並交出了她的帳戶信息。
More creative scenarios are also possible, like if you're the A.I., you can imagine wiggling electrodes around in your internal circuitry to create radio waves that you can use to communicate. Or maybe you could pretend to malfunction, and then when the programmers open you up to see what went wrong with you, they look at the source code -- Bam! -- the manipulation can take place. Or it could output the blueprint to a really nifty technology, and when we implement it, it has some surreptitious side effect that the A.I. had planned. The point here is that we should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever. Sooner or later, it will out.
更天馬行空的狀況也可能會發生, 就像是如果你是AI, 你可以想像藉由擺動你體內的電路 然後創造出無線電波, 用以與外界溝通。 或這你可以假裝有故障, 然後當程式設計師 把你打開檢查哪裡出錯時, 他們找出了原始碼 --梆-- 你可以在此做出操控。 或這它可以做出一個 很巧妙的科技藍圖, 當我們實施這個藍圖後, 它會產生一些AI計劃好的 秘密副作用。 寓意是我們不能 對我們控制人工智慧的能力 具有太大的信心 它終究會逃脫出來, 只是時間問題而已。
I believe that the answer here is to figure out how to create superintelligent A.I. such that even if -- when -- it escapes, it is still safe because it is fundamentally on our side because it shares our values. I see no way around this difficult problem.
我覺得解方是我們需要弄清楚 如何創造出一個超級智慧, 哪怕是它逃出來了, 它還是安全的, 因為它是站在我們這一邊的 因為它擁有了我們的價值觀。 我們沒有辦法避免這個艱難的問題。
Now, I'm actually fairly optimistic that this problem can be solved. We wouldn't have to write down a long list of everything we care about, or worse yet, spell it out in some computer language like C++ or Python, that would be a task beyond hopeless. Instead, we would create an A.I. that uses its intelligence to learn what we value, and its motivation system is constructed in such a way that it is motivated to pursue our values or to perform actions that it predicts we would approve of. We would thus leverage its intelligence as much as possible to solve the problem of value-loading.
但是我覺得我們可以解決這個問題。 我們並不需要把 我們在乎的所有事物寫下來, 或更麻煩的把這些事物 寫成電腦程式語言 像是 C++或 Python, 這是個不可能完成的任務。 與其,我們可以創造出 一個人工智慧,它用它自己的智慧 來學習我們的價值觀, 它的激勵機制要設計成 會讓它想要 來追求我們的價值觀或者 去做它認為我們會贊成的事情。 藉此我們可以最大化地 利用到它們的智慧 來解決這個價值觀的問題。
This can happen, and the outcome could be very good for humanity. But it doesn't happen automatically. The initial conditions for the intelligence explosion might need to be set up in just the right way if we are to have a controlled detonation. The values that the A.I. has need to match ours, not just in the familiar context, like where we can easily check how the A.I. behaves, but also in all novel contexts that the A.I. might encounter in the indefinite future.
這個是有可能的, 而且這個的結果 可對人類是非常有益的。 但是它不會自動發生。 如果我們需要控制 這個智慧的大爆炸, 那智慧大爆炸的初始條件 需要被正確的建立起來。 人工智慧的價值觀 要和我們的一致, 並不只是在常見的狀況下, 比如我們可以 很簡單低檢查它的行為, 但也要在未來所有人工智慧 可能會遇到的情況下 保持價值觀的一致。
And there are also some esoteric issues that would need to be solved, sorted out: the exact details of its decision theory, how to deal with logical uncertainty and so forth. So the technical problems that need to be solved to make this work look quite difficult -- not as difficult as making a superintelligent A.I., but fairly difficult. Here is the worry: Making superintelligent A.I. is a really hard challenge. Making superintelligent A.I. that is safe involves some additional challenge on top of that. The risk is that if somebody figures out how to crack the first challenge without also having cracked the additional challenge of ensuring perfect safety.
還有很多深奧的問題需要被解決: 它們決策概念的所有細節, 它如何面對解決 邏輯不確定性的情況等問題。 所以技術上待解決的問題 讓這個任務看起來蠻難的 -- 還沒有像做出一個超級智慧 那樣的難, 但還是挺難的。 我們所擔心的是: 創造出一個超級智慧 是一個很難的挑戰。 創造出一個安全的超級智慧 是一個更大的挑戰。 最大的風險在於 有人想出了如何解決第一個難題 但是沒有解決第二個問題 來確保安全性萬無一失。
So I think that we should work out a solution to the control problem in advance, so that we have it available by the time it is needed. Now it might be that we cannot solve the entire control problem in advance because maybe some elements can only be put in place once you know the details of the architecture where it will be implemented. But the more of the control problem that we solve in advance, the better the odds that the transition to the machine intelligence era will go well.
所以我覺得我們應該先想出 如何"控制"的方法。 這樣當我們需要的時候 我們可以用的到它。 現在也許我們無法 完全解決「控制」的問題 因為有時候你要了解 你所想要控制的架構後 你才能知道如何實施。 但是如果我們可以 事先解決更多的難題 我們順利的進入到 機器智能時代的機率 就會更高。
This to me looks like a thing that is well worth doing and I can imagine that if things turn out okay, that people a million years from now look back at this century and it might well be that they say that the one thing we did that really mattered was to get this thing right.
這對我來說是一個值得挑戰的事情 而且我能想像到如果一切順利的話, 我們的後代,幾百萬年以後的人類 回顧我們這個時代的時候 他們可能會說我們 所做的最重要的事就是 把這個事情弄對了。
Thank you.
謝謝
(Applause)
(觀眾掌聲)