Blurring the Lines Between Real and Virtual Spaces

Most of us have day jobs with set routines, responsibilities, and deliverables. Often, the mind wanders off, and as humans, we try to gauge alternative possibilities. What happens if I take the burden from one team member and add another team member to a project, suggest something to the executive leadership team, what should we cook for dinner, restaurant for the weekend, etc.? They are our banal thought processes. We are constantly juggling cause and effect and choosing the best of outcomes. Humans, in general, can do all this reasoning in the blink of an eye, which is the prophesized unique selling proposition of Artificial General Intelligence. Such decisions and their repercussions show us a parallel future that may exist — enter Metaverse, an environment that is a virtual space with augmented and virtual realities embedded therein. Think of people, cities, countries that exist digitally instead of flesh and bone or mortar and steel. In simple terms, Metaverse brings together the power of simulation technologies that we have championed over the last century and builds virtual worlds (almost like a video game) where people can have an immersive experience.

Immersive Experience

Little did Neal Stephenson (author of Snow Crash (1992) and the inventor of the term metaverse) know that this term will be foundational in building billion-dollar industries 30 years later. Metaverse has much excitement, and many companies are creating a digital presence. Nike has announced that it will create Nikeland on the metaverse platform Roblox. It will closely resemble the real physical Nike’s headquarters, and users can dress their avatars using Nike commodities, buy products from the virtual store, etc. Another metaverse platform Decentraland is hosting a first-of-its-kind Metaverse Fashion Week with lots of brands displaying their new collection and letting catwalk avatars walk the ramp. Some brands will be offering digital couture for the avatar where buyers can choose to sell the item and redeem the NFT (virtual currency — Non-fungible token) to buy a physical complement. Retailers like Walmart anticipate venturing into the Metaverse, creating their cryptocurrency and collection of NFTs. Instead of adding your milk and bread into a cart, we might be able to move around a store replica and buy our groceries (obviously delivered to our abode — we still cannot eat virtual bread and butter). Jokes apart, many exciting business models have been evolving in this space.

Advertising is yet another use case, as people embedded in the Metaverse can buy commodities for their digital self. This buying behavior will undoubtedly percolate into their real life. Instead of typing amazon.com on your web browser, we walk to the promised virtual land and buy the crockery we need for our dinner party by sieving through various advertised products in this virtual space (instead of your 2D computer screen). It indeed has taken 20+ years to get a seamless user interface for the 2D e-shopping world; I imagine, given the speed of innovation, one can get it right in half the time. Other use cases range from purchasing virtual avatars, buying homes to immersive learning in virtual classrooms. So, much scope for next-generation advertisement companies.

What’s in for the technologist? Metaverse can be a melting pot of machine learning technologies like computer vision (CV), Natural Language Processing (NLP) and reinforcement learning (RL). We currently have a myopic view — use NLP to create voice agents, CV to figure missed items on a shelf or RL for scheduling and planning. Only some people have seen what happens when we bring voice and vision to make decisions. Google Deepmind, FAIR, OpenAI, and many other labs have been championing the decision-making aspect of artificial intelligence. No limits of technological advancements in voice and vision alone can take us close to Artificial General Intelligence unless we can use these sensations in a concerted way to take decisions. Thus, Metaverse may become a virtual experimental ground — a massive multiplayer game (RL folks, please think more about multi-agent RL and your favorite recipes for decentralized partially observable Markov Decision Processes).

Build it, and They Would Come Flocking…

Significant technology corporations like Microsoft and Meta (formerly Facebook) and financial services firms like Morgan Stanley have been flocking to integrate some form of metaverse offering in their product line. Virtual reality headset companies (Oculus, Pico, Varjo, Vive, etc.) that enable us to enter Metaverse and hardware manufacturers from MediaTek and TSMC to Luxshare have invested in building the hardware for the foundational Metaverse. Despite decades of development, these “entry tickets to metaverse” are bulky to wear and expensive to own. The whole point of creating a future parallel to reality (the Metaverse) is to have an immersive experience. Anything less from a user-experience point of view, we hit the first snag from a hardware point of view.

Additionally, shouldn’t we start thinking about people who cannot see, hear or have an anxiety disorder (this was reignited as I watched Netflix’s As We See It). Or are we again going to create a judgmental metaverse? There are other issues concerning data and security, payment systems, legal jurisdiction in Metaverse that spans countries, ownership rights, etc. — more on these in a subsequent post.

Being a machine learning scientist in academia and a technology executive in the industry, I have learned to see ideas such as Metaverse (and its earlier avatar Second Life and a few others) at the intersection of technology and product, but with a lens of ‘value’. Even crucially, will this have customer adoption? Such a thought process quickly becomes labored. Much as the promise is Metaverse becoming a virtual space where people can “feel” physically and emotionally present — are we there yet? Not yet. Will we buy such a system whenever it is available? Maybe. The next question we will ask is what for? Remember, all in all, we will have to devote physical (not virtual) time, money, and energy to interact with Metaverse. At the same time, I can see it being parallel to playing a videogame building community with your virtual neighbors but do we have the cognitive capacity to live a parallel life. Don’t we already have an information overload? Yes, tons of companies would be minting money showing VCs and users the promised metaverse land, but it would be crucial to ask what the product is? Build it, and the customer will come flocking mantra may not work.

So, what can work? In my opinion, the first step towards the Metaverse is unleashed when you can use it to (a) make decisions and (b) use it as a synthetic world for machine learning (ML) data generation. One needs to build the product-market fit incrementally. Otherwise, the customer adoption argument becomes quite strained if industries start opening “shops” in virtual spaces and using digital currencies or Non-fungible Tokens for buying/selling digital assets. The baby step towards Metaverse is Digital Twins, again an age-old concept. Take a small piece of the natural world, say a retail store, and use a dumb-down metaverse (the digital twin) to enable real-time visibility of all assets (commodities, store associates, supply chain flows, etc.). Then use technologies like computer vision to measure real-time supply-demand here-and-now at the store. Natural language processing sieves through thousands of correspondences and tells you what tasks need to be done. Finally, under the constraints of the digital twin, reinforcement learning makes decisions unfolding futures. This will help store managers have a real-time view of the store operations and take the nascent field of Digital Twins to make usable decisions. Technologically, it allows us to combine various voice and vision attributes and take optimal decisions. Don’t we all want to make informed decisions? I bet we do.

The other baby step towards the Metaverse surrounds the digital twin again, but this time with a view of generating synthetic data. Our friends at Unity Technologies are already championing this line of thought, along with many other startups and technology companies. The central concept behind all of this is domain randomization. Digital twin, a subset of the Metaverse idea, enables us to create synthetic worlds and various subsets of the same world. What happens if my bedroom is painted red instead of white, the roads that I walk on are pebbled instead of concrete, change the visibility statistics (standard computer graphics tricks), etc. For example, most deep learning-based computer vision algorithms take a ton of training data. The digital twin (if constructed with rigor to reduce covariance shift from a real environment) can supply us with annotated synthetic data. Be it millions of kilometers of driving data for self-driving cars or hundreds of permutations for objects under different view statistics. Similarly, RL algorithms have a hard time generalizing when the domain is randomized, i.e., random changes in material (texture, color), light direction, light conditions, and placement of objects. The metaverse concept can help us alleviate some of the data efficiency problems.

In summary, if channeled from an end-product discussion, the excitement for Metaverse can shape and most probably lessen the impedance mismatch between the product and the market needs. Unchecked false assumptions can kill companies, no matter how technically great the solution is.

First Published here