Macroblock

This is an excerpt of an article published by the Hangjianet on March 29, 2021
Visit https://mp.weixin.qq.com/s/5J6MIaeKpKIze03X2T7_eg for the original article.

Abstract
Studios, up-and-coming directors and production teams are trying out virtual visualization as China introduced the production technique made popular by The Mandalorian. Through examination and experiments, the industry can see the power and advantages of the technology, which uses LED display to create a virtual background that actors perform against, cutting production cost significantly. As the innovative production method can replace the traditional green screen directly and add value to the production, it is attracting the attention of TV and movie as well as LED display industries.

Thanks to the explosive popularity of glasses-free 3D LED displays last year (Details), the industry continues to have high hope for application this year. As China has curbed the pandemic effectively, the depressed demand for film and TV rebounded strongly, which was highlighted during the Chinese Lunar New Year. The attention to indoor 3D LED studio is also surging. The public is very curious about the differences in software and hardware production between indoor 3D LED studio and regular LED display as well as the popular outdoor 3D LED displays. Therefore we asked leading LED display makers including Leyard, Unilumin, LianTronics, Ledman, Cedar Electronics, Aoto, Macroblock, NationStar and Novastar for their insights, which we shared with our readers in this article.

Question 1. Definitions.
Why indoor 3D LED studio? What are the criteria for judging indoor 3D LED studios?
Unilumin (ROE Creative Display): LED displays that replace green screen enable everyone at the site to immerse in the shooting. Camera crew works like they are shooting the real thing. Actors don’t have to face markers representing imaginary objects. The virtual scene is right against them. Most importantly, the creative decisions are left to the production crew at the site. They can discuss the shooting location, lighting and composition to make decisions during the shoot. Simply put, it offers a new tool at the service of movie production.     
Leyard (Virtual Point): Technical criterial for indoor LED studios include: 1) Hardware that emphasizes LED display and infrared motion capture camera. The key criterial for LED displays include:
                                                
As to infrared motion capture cameras, precision is the key. They need to be highly precise with minimum error and latency. A case in point is the OptiTrack infrared motion cameras. With millimeter-level errors, the cameras track the subject for highly precise display of images on LED displays with reasonable capture volume. Images are shot from an immersive camera angles to put the actors in the environment that feels real; and 2) Background with regard to shooting technique, especially the prevention of goofs with virtual visualization: LED displays can act as large-scale electronic background to create an immersive shooting environment with motional capture technology. However, the prop crew needs to address and cover where the LED displays and the floor meet reasonably well. Sand, rugs and plants can be used over the seams based on the set as the arrangement enforces realism of the set to prevent goofs and create a more realistic performance environment, which helps actors emotionally. The choice of camera lens and depth of field: To prevent camera from capturing the moiré pattern on LED displays, the camera crew has to choose the right lens based on the pitch of the LED display and create enough depth of field to minimize the pattern for buttery smooth videos.
Ledman: Display is still the most important element of 3D LED studio. While the set can be very realistic and immersive for the actors, it needs a very precise system to control color reproduction, grayscale and brightness.
Aoto: Comparing to traditional green-screen studio, 3D LED studio has many technical advantages:

- It creates an immersive set in which actors can interact directly with virtual yet visible elements. More actor-friendly than the traditional empty green-screen set, it helps them to get into their characters sooner.
- The crew can freely switch among sets created in 3D LED studio. Desert and forest can be created in the 3D LED studio without location shooting to improve production efficiency significantly.
- The technology and reproduction technology saves a lot of post-production time for the VFX department. Production cost goes down significantly as a result. Shine, reflection and rebound off glossy costume is more accurate, which cannot be achieved with traditional shooting techniques on set.
- These are the benefits of 3D LED studio. Nevertheless, there is room for improvement as to the results of 3D LED studios. The post-production cost remains high also. Key factors affecting the results include: color shift and mask angle; moiré pattern; color reproduction; transmission latency and frame synchronization; and reflection and noise.

​LianTronics: The criteria for judging a 3D LED studio include LED display performance, protection and stability as well as motion tracking system and image processing engine algorithm.

- 3D LED studio is more critical of LED display, including display performance that must meet UHD, ultimate display performance, high chroma and color temperature adaptability, studio shooting and protection requirements to ensure professional cameras capture the LED virtual background cleanly and avoid goofed-up details while retaining high grayscale of shadow details and overall consistency and uniformity of images.
- 3D LED studio combines virtual-real integrity, template-based animation and motion capture. Recording preparation comprises 3D virtual background design, image tracking and collection, image rendering, virtual placement, motion capture system collection and processing among others. Camera tracking system, virtual engine and image rendering engine need superior algorithms and processing power to guarantee smooth hardware and software operation and stable input and output.

​​Macroblock: 3D LED studio replaces blue/green screen with LED displays for virtual background. Benefits include cutting post-production time and cost, immersive virtual environment and light source to improve acting and no set creation and location shooting. Therefore, criterial for judging a 3D LED studio include the ability to create virtual background materials, camera tracking and virtual shooting techniques and LED display performance. 
NationStar: Traditional movie production shoots many scenes with actors against a green screen followed by post production. That puts a lot of restrictions on the creative team in terms of performance and rendering. A little VFX mishap may upend the result. Audience will enjoy a visual feast as LED display is combined with VR technology. Camera can shoot the LED display showing preprogrammed VR background that is presented and adjusted in real time. The LED display may also act as a virtual light source for the shoot. The visual effects can be very realistic as light and shadows are combined. As a light source, it needs to meet automotive reliability standards with sub-10PPM defect rate. The objects need to have matte surfaces to reduce reflection. The surfaces have to be uniform to display uniform images. In other words, it needs to be at least as good as REESTAR RS1515, our flagship indoor offering.
Novastar: Blue screen composition has always been the first choice for putting fantastical environment and actors in the same frame in movies like Avatar and Life of Pi. Nevertheless, actors wearing highly reflective costume will result in reflection off the green screen, blue screen and the environment, leading to expensive production issues. It’s challenging to the director as they have to design the shots and communicate with actors who have their own artistic challenges. An immersive environment built with LED displays will be precise to pixels, facilitating accurate rendering with camera positioning. Actors, props and sets shot against the LED displays showing pre-created contents will ultimately realize in-camera visual effects. The video will be so close to the end result that only some technical touch-up is needed. The production cost will be reduced significantly. Actors will have better interaction with the virtual environment too.

Question 3. Ecosystem Challenges
Does the 3D LED studio pose new challenges to display, light source and driver IC providers? What are the actual challenges?
Leyard (Virtual Point): These hardcore criteria are strongly associated with the core processing IC and driver for the display. We not only strive to improve color processing depth and color space but also ensure stable refresh. There is still room for improvement when it comes to image uniformity and color consistency as this requires enhanced manufacturing technology.
LianTronics: The differentiation of large format LED displays centers on upstream LED supply as they have very stringent requirements for the beam angle, wavelength, color space and color temperature among other physical parameters. The upstream suppliers have to improve the technology and quality control to ensure the consistency of physical and electrical parameters for LEDs in the same batch, thus reducing the technical difficulties for the ecosystem, driving technical progress and cutting manufacturing cost. 
Aoto: There are major differences in product criteria and parameters between LED displays for shooting TV and movies and the general-purpose displays. The criteria and differentiation mostly manifest in the core control system of LED displays and the design of LED display offerings. The key challenge is to meet the display result and performance requirements specific to LED displays for shooting TV and movies. Aoto has the core LED display control system and has developed professional control systems and LED display offerings specifically for shooting TV and movies.
Ledman: They have a lot to do with the display. As LED display hardware shows the image directly, its performance is essential to how well the scene is shot. Color reproduction, refresh, grayscale control and color space all play pivotal roles to the shooting. XR-related elements including LEDs and driver ICs need to use high-end, high-bandwidth materials. Technical speaking, the chip wave length prevents us from covering 100% of the color space. The mid-stream technology continues to improve though.        
Macroblock: LED driver ICs for 3D LED studio applications have to support high refresh rate, high grayscale, high frame rate and excellent low-brightness performance. High refresh rate is needed to prevent capturing scan lines on the LED displays; high grayscale improves color depth, contrast ratio and color reproduction for HDR video; high frame (>120 FPS) supports images with high-speed movements; excellent low-brightness performance ensures image consistency and uniformity and prevent high contrast interference that hurt the images. Macroblock MBI5264 and MBI5850 support 120FPS/ 16-bit@3,840Hz/ with a maximum brightness of 2,000 nits, making them ideal for 3D LED studio applications.
                                                                             
Novastar: Given the camera exposure time and display frame rate, complete, flicker-free LED images require the right mix of camera angle and frame rate. For the best shooting results, LED displays need to support repeat output of frame-rate images for frame-rate expansion. Furthermore, most available cameras cannot do phase shift control, resulting in phased-locked camera and controller producing moving dark lines. XR demands high-quality LED displays to ensure ghost-free smooth images while shooting high-speed subjects. The controller needs to support high frame-rate input and super-low latency. It is also highly demanding of display grayscale, bright and shadow details as well as color accuracy.