Color-Aware Deep Temporal Backdrop Duplex Matting System

authored by

Hendrik Hachmann, Bodo Rosenhahn

Abstract

Deep learning-based alpha matting showed tremendous improvements in recent years, yet, feature film production studios still rely on classical chroma keying including costly post-production steps. This perceived discrepancy can be explained by some missing links necessary for production which are currently not adequately addressed in the alpha matting community, in particular foreground color estimation or color spill compensation. We propose a neural network-based temporal multi-backdrop production system that combines beneficial features from chroma keying and alpha matting. Given two consecutive frames with different background colors, our one-encoder-dual-decoder network predicts foreground colors and alpha values using a patch-based overlap-blend approach. The system is able to handle imprecise backdrops, dynamic cameras, and dynamic foregrounds and has no restrictions on foreground colors. We compare our method to state-of-The-Art algorithms using benchmark datasets and a video sequence captured by a demonstrator setup. We verify that a dual backdrop input is superior to the usually applied trimap-based approach. In addition, the proposed studio set is actor friendly, and produces high-quality, temporal consistent alpha and color estimations that include a superior color spill compensation.

Organisation(s)

Institute of Information Processing
L3S Research Centre

Type

Conference contribution

Pages

205-216

No. of pages

12

Publication date

08.06.2023

Publication status

Published

Peer reviewed

Yes

ASJC Scopus subject areas

Computer Graphics and Computer-Aided Design, Human-Computer Interaction, Software

Sustainable Development Goals

SDG 12 - Responsible Consumption and Production

Electronic version(s)

https://doi.org/10.48550/arXiv.2306.02954 (Access: Open)
https://doi.org/10.1145/3587819.3590973 (Access: Open)