How often are you frustrated by the time it takes to accurately get things in your mind into a computer? It is even worse for people like me, whose first language is not based on letters. I live and work in Australia, but I am originally from Taiwan. I moved to Sydney eight years ago and now run a university research center there.
Most of us use keyboards every day to get things in our minds into the computer. We have to learn to type. The fact that you have to learn to do some things shows how unnatural it is. The finger-driven touch screen has been around for 60 years. It's convenient, but it is also slow. There are other ways to control computers -- joystick or gestures -- but they are not very useful in capturing the words in your mind. And it is words -- they are critical to communication for human beings.
The problem is about to be over, because of AI. Today, I will show you how AI can turn the speech in your mind into words on screen.
Getting from the brain to the computer efficiently is a real bottleneck for any computer application. It has been my passion for 25 years. Many of you, or most of you, have heard of "brain-computer interface," BCI. I have been working on BCI, for the direct communication between the brain and machine, since 2004. I developed a series of EEG headsets that do this.
But they are not new. What is new is an interface that works in a natural way, based on how our brain is working naturally. Imagine reading words when someone is thinking, translating the brain signals into words. Today, you will see this in action, and with no implants.
We are using AI to decode the brain signals on the top of your head and identify the biomarkers of speaking. That means that you can send the words in your mind into the computer with wearable technology. It's exciting, and I believe it will open up the bottleneck of how we engage with computers.
We are making exciting progress in decoding EEG to test this. It's natural. We have had very promising results in decoding EEG when someone is speaking aloud. The frontier we are working on now is to decode EEG when the speech is not spoken aloud, the words that flow in your mind when you are listening to others or when you are talking to yourself or thinking. We are well on the way to make it a reality. Who would like to see this in action?
(Cheers and applause)
Great, we are ready to demonstrate it to you. I am going to invite two of my team, Charles and Daniel, to show it to us again. This is the first world premiere for us, so I hope you can be patient with us. We are getting around 50 percent accuracy ...
(Laughter)
in decoding the brain signals into words when someone is speaking silently. Here shows how it will work. We have a collection of words that we have trained our technology with. They are combined into sentences. Charles will select one sentence, and Daniel will read the sentence word by word, silently, and produce the brain signals that will be picked up by our sensors. Our technology will decode the brain signals into words. Charles, Daniel, are you ready to go ahead?
This is the sentence that Daniel is going to read silently.
(Applause)
Sorry, please keep silent. (Laughter) Here are the -- decoded words. They are likely the intended words. You can see the probability ranking of the decoded words by our technology. The pattern shows our predicted sentence ... is not so correct.
(Laughter)
Sorry, you see the other 50 percent, the system doesn't work. But actually, you still can see some keywords where we got it.
Let's invite Charles and Daniel to do it again, but please keep silent when he's reading silently. Here is the sentence, another sentence that Daniel will read word by word, silently. (Laughs) Again, here are the decoded words. They are likely the intended words. The pattern shows our predictive sentence is very close to the ground-truth sentence this time.
(Cheers and applause)
Thank you, thank you.
How does it work? We pick up the brain signals with sensors and amplify and filter them to reduce the noise and get the right biomarkers. We use AI for the task. We use deep learning to decode the brain signals into the intended words. And then we use the large language model to make the match of the decoded words and make up for the mistakes in EEG decoding. All of this is going on in the AI, but for the user, the interaction is natural, through thoughts and natural language. We are very excited about the advances that we are making in understanding words and sentences.
Another thing that is very natural to people is looking at something that has their attention. Imagine if you could select an item just by looking at it, not by picking it off the shelf or punching a code into the vending machine.
Two years ago, in a project about hands-free control of robots, we were very excited about robot control via visual identification of the fingers. We are now beyond that. We need not any fingers. The AI is making it natural.
There are four objects on the table. Toy car, toy animal, plastic flower and some food, which is also plastic, not left over from the breakfast this morning. You can also see the four objects' photo on the screen. Daniel is going to look at the photos, and select an item in his mind. If it is working as it should, you will see the selected item pop up on screen. We use photos for this because they are very controllable. To show that this is not all just built into my presentation, Charles will pick up one item for Daniel to select in mind. Please, Charles. It's a car. So, Daniel, select ... the car in his mind.
(Laughter)
Hamburger.
(Laughter)
It's incorrect.
(Laughter)
It's unlucky that the 30-percent error rate came with us again. Let's invite Charles and Daniel to show it again. It's a duck, a lovely duck.
(Laughter)
OK. Good.
(Cheers and applause)
Thank you. Thank you.
Daniel did this for his PhD. It's very impressive, isn't it? When Daniel selects an item in his mind, his brain recognizes and identifies the object and triggers his EEGs. Our technology decodes the triggers.
We are working on our way through the technical challenges. We will work on overcoming the interference issue. That's why I asked for the phones to be turned off. Different people have different neural signatures, which are important to decoding accuracy. One reason I brought Daniel along here is because he can give off great neural signatures.
(Laughter)
(Applause)
Yeah, he can give us a great neural signature, as far as our technology is concerned. They are still cables here as well. It is not yet very portable. Probably one biggest barrier to people using this will be: “How do I turn it off?” Any one of you will have had times when you are happy that the people you are with don't know what you are really thinking.
(Laughter)
There are serious privacy and ethical issues that will have to be dealt with.
I am very passionate about how important this technology can be. One exciting point is linking the brain-computer interface to wearable computers. You already have a computer on your head. The brain will be a natural interface. It is not only about controlling a computer. The natural BCI also provides another way for people to communicate with people. For example, it allows people who are not able to speak to communicate with others, or such as when privacy or silence are required.
If your idea of nature is a lovely forest, you could wonder how natural this could be. My answer is, it's natural language, it's the natural thought process that you are using. There are no unnatural implants in your body. I am challenging you to think about what you regard as natural communication. Turning the speech in your mind into words.
There is a standard way to finish up when talking with people -- you say: “Just think about it.” I hope you are as excited as we are for the prospect of a future in which, when you just think about something, the words in your mind appear on screen.
Thank you.
(Applause)
