Efficient Deep Networks for Image Matting

Abstract

Image matting is a fundamental technology serving downstream image editing tasks such as composition and harmonization. Given an image, its goal is to predict an accu- rate alpha matte with minimum manual e orts. Since matting applications are usually on PC or mobile devices, a high standard for e cient computation and storage is set. Thus, lightweight and e cient models are in demand. However, it is non-trivial to bal- ance the computation and the performance. We therefore investigate e cient model designs for image matting. We rst look into the common encoder-decoder architecture with a lightweight backbone and explore the skipped information and downsampling- upsampling operations, from which we notice the importance of indices kept in the encoder and recovered in the decoder. Based on the observations, we design data- dependant downsampling and upsampling operators conditioned on features from the encoder, which learn to index and show signi cant improvement against the baseline model while promising a lightweight structure. Then, considering a nity is widely used in both traditional and deep matting methods, we propose upsampling operators conditioned on the second-order a nity information, termed a nity-aware upsampling. Instead of modeling a nity in an additional module, we include it in the unavoidable upsampling stages for a compact architecture. Through implementing the operator by a low-rank bilinear model, we achieve signi cantly better results with only neglectable parameter increases. Further, we explore the robustness of matting algorithms and raise a more generalizable method. It includes designing a new framework assembling mul- tilevel context information and studying strong data augmentation strategies targeting matting. This method shows signi cantly higher robustness to various benchmarks, real-world images, and coarse-to- ne trimap precision compared with other methods while using less computation. Besides studying trimap-based image matting, we extend our lightweight matting architecture to portrait matting. Targeting portrait images, we propose a multi-task parameter sharing framework, where trimap generation and matting are treated as parallel tasks and help optimize each other. Compared with the conventional cascaded architecture, this design not only reduces the model capacity to a large margin but also presents more precise predictions.