Katherine is a Sensorium avatar, a virtual being driven by fluxCortex AI.

Image Credit: Sensorium

Sensorium Galaxy is an upcoming virtual world that will blend human and AI characters so that people can have unscripted conversations with virtual beings. And it’s showing off some of those AI characters now.

I had a conversation with one of the AI characters, Katherine. She’s an aspiring makeup artist studying in Amsterdam. She wasn’t real. But it was fun to pretend she was, as she was able to hold a conversation better than me. I was having trouble coming up with interesting questions to ask, as a weird kind of stage fright took over me as we talked.

Sensorium AI-driven avatars are the first in a new generation of virtual beings. They are capable of supporting complex and unscripted  conversations through the prism of their unique personality, built upon their “biography” and interactions with users. Overall, they mark a new milestone in the advancement of сonversational AI, raising the bar from chat-bots or traditional non-player characters to ever-evolving human-like conversations.

Katherine gets her AI brain from fluxCortex, a hybrid Evolving AI technology based on genetic algorithms and reinforcement learning. It is paired with several large language models, including GPT-3 by OpenAI. Her programming is a result of continual learning, and a personality vector that defines a short bio on which the AI acts.

She recognizes speech, processes the query, and comes up with a response. Her programming is simply a set of words that can be put together as a kind of bio. Everything she says in response to questions is unscripted and unpredictable.

And that’s the whole point, as Sensorium wants to enable AI characters that you can talk to for hours inside a virtual world. Each avatar has a unique biography to draw from, which creates natural and evolving interactions. Avatars possess both long-term and short-term memory, enabling them to hold a long conversation, without losing context, said Sensorium Galaxy product development lead Ivan Nikitin (who helped me get through the demo).

Katherine is extroverted, energetic, and sometimes a bit overbearing. Her dream is to land an apprenticeship with a famous band or Cirque du Soleil. She doesn’t like alcohol and prefers to get her buzz from other things, but good wine is an exception. Or so her bio says.

I also spoke with Thomas Elon, a former professional boxer. He left the sport after serious trauma when he was 40. He is 45 now. He trains young boys in martial arts and keeps an eye on their well-being.

Both Katherine and Thomas had some delays, but I was speaking over a Zoom connection, and so the conditions weren’t ideal for voice recognition and faster processing.

Sensorium’s backstory

Sensorium was founded in 2018 by multibillionaire Mikhail Prokhorov and a former owner of the Brooklyn Nets. Sensorium makes money through avatar customization as well as making money from ticket sales to virtual concerts and premium subscriptions. Users can invest their own money in clothes or other features for their avatars, and people can buy digital tickets to go see events. You can pay more for VR or watch a video livestream for free.

Sensorium is creating other worlds, like Motion, with different settings and ways to show off photorealistic avatars, built with photogrammetry techniques, Nikitin said. Ultimately, he wants to create worlds where you can have free-flowing conversations with the AI characters, dubbed NPCs, or non-player characters.

Los Angeles-based Sensorium is known for providing simulated virtual experiences of concert venues. The company creates photorealistic avatars, or virtual characters, that look like real people.

Above: Sensorium’s AI virtual DJs can host concerts in VR.

Image Credit: Sensorium

Sensorium has raised more than $100 million in pursuit of this mission, and it will produce a diverse series of photorealistic virtual artists featuring unique identities and musical preferences. Genetic algorithms behind each of these all-digital characters will drive their behavior and allow them to absorb data from the environment to constantly nurture their knowledge and evolve their music styles. And yes, this is one more attempt to create the metaverse, the universe of virtual worlds that are all interconnected, like in novels such as Snow Crash and Ready Player One.

Sensorium Galaxy is a multi-user social VR platform aimed at remaking the way people interact with each other and experience the arts. The Galaxy is being built in partnership with artists, producers, and entertainment companies. Among them are Yann Pissenem, creator of nightlife hubs Ushuaïa Ibiza and Hï Ibiza, and music and entertainment streaming service Tidal, which is owned by artists such as JAY-Z, Lil Wayne, Rihanna, Daft Punk, and others.

Sensorium also creates virtual DJs who autonomously create a constant flow of ever-changing music that adapts to all kinds of moods and environments. It’s another attempt to stave off human loneliness by filling our lives with — artificial people. The company recently teamed up with Mubert to develop AI-driven DJs that can create music at virtual concerts in virtual reality. It’s the latest in an effort to create artificial people, or virtual beings.