Only Truth Matters
0%

Google Computational Photography

Posted on Edited on In , , ,

1. Alpha Matting

Goal: accurate matting for fine details

Application: background replacement, synthetic bokeh

Idea: not binary, but estimate a transparency value

How: Train a CNN

  • input: RGB image and coarse alpha matte
  • MobileNetV3 and shallow encoder-decoder
  • A studio to make high-quality dataset

2. Omnimattes

Goal: produce mask not only for objects but also their effects (shadow, smoke, reflection..)

Application: image and video editing

Idea: create different layer for each object and also the background

How: Train a CNN

  • learn the strong correlation first

3. Denoise and Sharpen

Goal: denoise for any image without knowing the parameter of the camera. (Computational effective)

Application: image post-processing

Idea: assess the noise level and blur type in patch level

How:

  • Pull-push denoising
  • first pull(down-sample), in different level(pyramid), use this to find local structure
  • Then push(up-scale), pixels fused with coarser level by doing weighted average
  • Polyblur: deblurring caused by a blur kernel
  • observation: the maximal value of the image gradient across all dir in a sharp image follows a particular distribution. So we can estimate the blur in given direction
  • Assumption: mild blur
  • How: reblur it serveral time

4. HDR+

Goal: reduce noise, handle large dynamic range scene

Application: mobile phone photograph

Idea: burst of under-exposure

How:

  • for noise, burst, align, average
  • for HDR, bracketing(same exposure time), burst, align, combine

5. HDR+ with bracketing

HDR+ works for maoderate dynamic range, but fails for High dyanmic range

HDR+’s problem: noise

  • shot noise, if this is the only noise, HDR+ works great
  • read noise, depends on the number of frames taken, make HDR+, can not be canceled

Goal: to capture real high dynamic range

Idea: before the shutter, use burst; after the shutter, use bracket; merge

How:

  • new spatial merge algorithm
  • Deghosting algorithm
  • merge raw earlier

6. Night Sight

Defien the dark level, lux.

  • Samrt phone start to struggle at 30lux
  • HDR+ make it possible at 3lux
  • Goal: between 3lux and 0.3lux with a smartphone and single shutter, no flash

Idea: capture more photons.

  • handheld for longer exposure
  • motion blur: use optical flow to minimize the blur

Problem:

  • alignmnet, HDR+, Super Res Zoom (require faster processor)
  • AWB in low light: ill-posed problem, develop a learning based algorithm
  • Tone mapping: well exposured image just like the daytime. Using the S-curve

7. Super Res Zoom

Goal: better digital zoom, merge many frames to a single high-res image

Idea: sensor shift that can be used to get the detail in Bayer

  • take the hand-motion as pixel-shift, OIS to compensate the large motion

How:

  1. take burst
  2. select 1 image as base
  3. interpolate between two images

Problem:

  • single image is noisy, align along apparent edges (a trade off, balance)
  • object has motion too, deal with imperfect alignment (only merge the confident deature)
  • the image data is not uniformly distributed

8. Motion Still

Goal: stablize the image

Idea: motion estimation and crop

How:

  • Pure software-based: analyze and tracking the feature, then do a affine transformation
  • Classify fore and back ground, estimate a optimal path by the metadata

9. Video Stablization

Goal: EIS+OIS

Problem: camera shake, motion blur, rolling shutter, focus breathing,

img

10. Image Quality Assessment

Ref

Multi-scale Transformer, take image with different aspect ratio

image-20221108142717028