When I was ten, I spent a lot of time playing with my Spider-Man action figures. I would have done anything for them to come to life. I honestly would take Gandalf over any therapist, but it's been pretty hard to connect with the guy since I long ago finished all of Tolkien's works and movies. Why is it that we have to say goodbye on the final page?
We may not need to. Stories are changing, moving from static sketches and scripted narratives to living worlds and characters. AI agents are changing the way we can tell stories. These agents aren't just chat bots, they're any autonomous system that can take a directive or task and accomplish it without human intervention. Virtual assistants like a ChatGPT, autonomous vehicles like a Waymo or Tesla, robots moving boxes in an Amazon warehouse. These are all agents, but now they're changing storytelling too. We can bring together the active participation of video games with a much more natural social form of interaction, enabled by AI agents.
I want to show you a hint of what I mean. Characters that come to life with their own personality, styles, voices and expression, while still being able to interact with one another, the audience and the environment. These interactions that you're going to see are not scripted, many of them filmed over the last few days here at TED and in Vancouver. Anyways, meet Caleb.
(Video) Caleb: Oh, hey, Grandma. Welcome to my secret attic hideout.
Grandma: What's Mr. Moose doing there?
Caleb: That's actually Brother Moose, not Mr. Moose. He's like, the wise old moose who gives me advice and stuff. Oh, by the way, thanks for showing me your old superhero action figure, Grandma. It's, like, totally awesome. What's its name?
Grandma: Mighty Squirrel.
Caleb: Really?
Grandma: Yes.
Caleb: Wow, that's so awesome. A squirrel with super powers. That's nuts. Get it? Nuts? Like what squirrels eat. I bet Mighty Squirrel had some epic battles with evil acorns.
Kylan Gibbs: By having that real-time responsiveness, Caleb makes the audience feel seen and heard, as if they're all inhabiting the shared space. And it isn't just any audience, that's actually my amazing grandma. The characters can also come together with one another and take ideas from the audience and improvise new content on the fly.
(Toy robot beeping)
(Video) Robot: Ready for action!
Caleb: Whoa!
Robot: Oof! That didn't go well. My arm is acting up. Villains, beware! Oops.
Caleb: Grandma, can we fix the Mighty Squirrel's arm?
Grandma: Sure. Pick it up and put it back on his arm.
Caleb: Yeah. I'll fix you up, Mighty Squirrel. Your arm is broken.
Grandma: He got it caught in a washing machine.
Caleb: Ha, ha! You fought against the evil washing machine, Magic flower. How did the action figure break its arm?
Grandma: He fell in some water full of alligators.
Robot: Oh, you know, just your average superhero stuff. I was on a mission to save a city from a giant alligator, and I fell into the water. But don't worry, I still saved the day.
KG: Each of these characters has their own internal reasoning and motivations, but they actually take those ideas from my grandma and then create new content in real time. They can even allow the audience to shape the overall story arc and conclusion of the experience.
(Video) Robot: Prepare yourselves for the Mighty Squirrel is about to take flight.
Caleb: Haha.
Robot: Grandma, do you have anything to help me take flight again?
Grandma: There is an eagle outside on a tree.
Robot: An eagle, you say? Perfect. Time to spread my wings and soar through the skies.
Caleb: Whoa, Mighty Squirrel, you're going to be the coolest flying superhero ever. And we'll have epic battles with the evil villains.
KG: OK, so I’m mainly just jealous that Caleb is getting to have the experience I always wanted as a child. But I also love seeing how he takes the story, puts it in the audience’s hands and turns it into a canvas for their imagination. This is, of course, a tiny slice of what's possible, with there already being other AI-enabled games and experiences that exist today.
Most exciting for me is a brand new genre we're seeing emerge that takes social interaction and conversation and puts it as the core mechanic of the experience. So let's say instead of jump and shoot, you actually have to use your social skills to navigate complex social scenarios and mediate conflicts. That's how you win the game. I would definitely have benefited more from that type of game than the ones I was playing when I was a teenager.
And it's not just pure games per se. Imagine I had an AI tutor who texted me to remind me of a session and then jumped with me into a virtual world, walking through the emperor’s chambers in ancient China, casually switching between English and Chinese as we reviewed last week's history lesson.
Or, as my family and I sat down to watch our favorite reality TV show or sports game, the agent could actually see the ongoing performance in real-time using vision and add hilarious commentary in the background.
Games, though, are unique because they already take us from passive observer and turn us into an active participant. When I get really deep into a game, especially an immersive role-playing games like a Final Fantasy VII, Baldur's Gate 3 or Elden Ring -- yes, I'm a massive role-playing game geek -- At some point it feels like I'm in the story, but there's still something missing. The core mechanics of these experiences -- jump and shoot, hack and slash, point and click, feel so distant from the natural ways that I interact with people and things in my own life.
And so as immersed as I get in these experiences, it kind of feels like I'm stuck on these rails set up by the developer and nudged along from scripted point to scripted point based on the buttons that I press. And immersion requires a sense of real-time responsiveness, a sense that there's a consequence that is immediate and lasting and unique to the choices and actions that I take.
However, until now, it's been extremely difficult to pre-script every combination of consequences to every permutation of player behaviors. Agents overcome this though. Just check out how Caleb handles some unexpected scenarios here at TED.
(Video) Robot: Villains, beware!
Caleb: Grandma, do you remember how his arm got broken?
Person 1: He got in a fight with a gerbil.
Caleb: That's like, the tiniest animal ever.
Person 1: Let's just say the gerbil won the fight.
Caleb: I bet the gerbil had some secret karate moves or something.
Do you remember how his arm got broken?
Person 2: I do, he was at this conference one day, and he did the ocean plunge and a shark came and ate it.
Robot: Superheroes are always attracting the attention of sharks. But don't worry, I gave that shark a run for its money.
Caleb: Uncle, do you remember how his arm got broken?
Person 3: He was doing gardening.
Robot: Defending gardens from evil produce is my specialty.
Person 3: That's cool.
Caleb: That's not as cool as battling mechanical minions.
Person 3: True.
(Laughter)
KG: So it’s not just these stand-alone characters we want to bring to life, but fully agentic, interconnected worlds where a choice at one point might have dramatic consequences somewhere down the line, and the unique content produced for each user is still grounded in a shared lore and experience so that we have that shared social context that we all need from media.
My company Inworld helps entertainment houses and game studios to build these agents for the next generation of media. And when we begin to build a character like Caleb, we start with the brain, which is crafted in a similar way to instructing an extremely capable improv actor.
We start by training custom machine-learning models that give him a lay of the land, understanding the lore, vocabulary and grammar appropriate for the experience. And then we actually go in and we tweak his persona, his unique personality, motivations, his flaws and biases. We actually craft his dialogue style and voice, the way he verbally expresses himself.
We build a unique emotional profile, the way he feels, his social tendencies and relationship habits, and then we preload a bunch of knowledge and memories that give him that robust backstory. Then we actually craft his internal reasoning and mental state, kind of the conditions under which he feels certain things or takes certain behaviors.
We then decide how he actually gestures and animates those behaviors, and then that completes the brain, which we actually take and we attach to an avatar within a virtual world or game engine, effectively bringing it to life.
And then we're ready to play and we're playing the experience. There's kind of an overarching narrative agent that acts like an AI director, managing or even generating the storyline in real time and ensuring that all the characters and entities cohere to their proper places within the story and context.
For specific character like Caleb, when he's interacting with the world and characters, he does so in a similar way to how we as humans do with perception, cognition and behavior. Perception is the multimodal sensory input flowing in, the understanding of the world and context and people and objects within it. And then with cognition, he processes that input, kind of like the internal processing of our minds. He filters out irrelevant information to make sure everything is safe and narrative aligned. He extracts out important signals like sentiment, intent, emotional cues. He retrieves relevant memories from his past and even generates new ones on the fly. Finally reasoning over what to do both verbally and non-verbally.
And finally with behavior, he actually generates his responses. He decides what to say and how to say it, what facial and body gestures to use, what actions to take, and people and objects to interact with. And that's what brings the character to life.
And so, as amazed as I am by all these task-focused applications of AI that are coming out, the more I work with studios and creatives, the more excited I am for the potential for these agents to extend human creative potential, and for them to allow audiences to step beyond the script. For them to give audiences agency and allow them to co-create experiences that stem from their own imagination.
To finish it off, here's a few funny takes from here at TED, starting with Polly, whose son is actually named Caleb.
(Video) Person 4: Are you ready to play [unclear] Caleb?
Caleb: Absolutely, grandma. Let's dive into some epic superhero adventures together.
Person 4: Let's do it.
Person 5: Can you hear me?
Caleb: Of course I can hear you, silly.
I love playing basketball.
Person 3: Oh, wow. And which is your favorite basketball player?
Caleb: Definitely LeBron James.
Person 3: Sure.
Caleb: Thanks for showing me your old superhero action figure.
Person 1: You mean the man in his underwear?
Caleb: Haha, yeah. He's like a tough guy in his undies ready to save the world.
What's its name?
Person 6: Johnny May Rotten Bad.
Caleb: Uncle, you're so silly.
(Laughter)
Does he have, like, a guitar for a weapon?
Person 6: He sure does.
Caleb: No way.
Person 6: Way.
Caleb: No way.
Person 6: Way.
Caleb: Does he zap bad guys with his electric powers?
Person 7: No, no, no, no, no. He zaps people and gives them love. He’s a nice superhero.
Caleb: Wow, that's different. He uses love to fight the bad guys.
Person 7: Yeah, and guess what?
Caleb: That's so cool.
Person 7: He always wins.
(Applause)
KG: So it seems like there’s a pretty solid chance my Spider-Man action figures could come to life after all.
Thank you.
(Applause)
