The research team at the University of Waterloo is developing a new way to provide different viewing experiences on platforms such as twitter and Youtube This technology converts 2D live video into 3D interactive experience, and the audience can watch the action from any angle
Many multiplayer shooters have an audience mode that allows users to see what's happening from different angles. This is a great way to observe what strategies other players use, or enjoy watching an outstanding player destroy a competitor. However, to do this, you must be in the game.
Jeremy Hartmann and Daniel Vogel of the Cheriton School of computer science at the University of Waterloo published their research at the end of April. This paper, entitled "enhancing live video games by reconstructing interactive 3D game views for viewers", explains a way to achieve effects similar to audience mode, but without installing games.
This technology adopts 2D video streaming, and adds the depth buffer, camera pose and projection matrix of the game to create a 3D interactive environment, which can be explored by the audience when the streaming media player plays the game. Viewers can move their cameras freely in 3D and VR.
At first, it looks like a strange angle where the actions on the screen are superimposed on the gray background. However, as players move and look around, simple texturless models will begin to form because the system builds the environment in real time.
This is different from the spectator mode in the game. As mentioned earlier, everything has no texture except the player's perspective. In addition, anything outside the streaming media perspective is without a picture. For example, in the above video, you can see how the player's model is fully rendered, but slides in the environment, just like on a wheel.
The lack of textures and animation does weaken the viewing experience to some extent, but remember that this method of rendering live video game streams in 3D is still in its infancy. The advances in technology and machine learning described by Hartmann and Vogel may lead to a better and more enjoyable interactive live experience, allowing viewers to enter the game directly.