Physically-Based Editing of Indoor Scene Lighting from a Single Image

ECCV 2022 (Oral presentation)

  • 1UC San Diego
  • 2Adobe
  • 3CMU

Overview

overview

We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based light representations that allow for intuitive editing, and infer both visible and invisible light sources. Our neural rendering framework combines physically-based direct illumination and shadow rendering with deep networks to approximate global illumination. It can capture challenging lighting effects, such as soft shadows, directional lighting, specular materials, and interreflections. Previous single image inverse rendering methods usually entangle lighting and geometry and only support applications like object insertion. Instead, by combining parametric 3D lighting estimation with neural scene rendering, we demonstrate the first automatic method for full scene relighting from a single image, including light source insertion, removal, and replacement.

Highlights

  • Novel editable, physically-based representations for common indoor light sources.
  • An inverse rendering method to accurately recover editable physically-based representations of visible and invisible light sources, as well as reflectance.
  • A novel neural differentiable renderer that allows physically-based training, rendering and optimization.
  • Single-image editing of individual light sources for full scene relighting with global light transport.
  • All code, trained models, data are to be publicly available.

Pipeline

overview

Editing Applications: Move Virtual Lamps

Editing Applications: Open Virtual Windows

Editing Applications: Turn off Light Sources

Editing Applications: Change Wall Colors

Acknowledgments

We acknowledge support from NSF, ONR, CHASE-CI, Adobe, Google and Qualcomm. The website template was borrowed from Michaël Gharbi.