The ultimate camera

What does it do?

The ultimate camera

Infinite resolution

Infinite zoom control

Desired object(s) are in focus

No noise

No motion blur

Infinite dynamic range (can see dark and bright things)

...

Creating the ultimate camera

The “analog” camera has changed very little in >100 yrs

we’re unlikely to get there following this path

More promising is to combine “analog” optics with computational techniques

“Computational cameras” or “Computational photography”

This lecture will survey techniques for producing higher quality images by combining optics and computation

Common themes:

take multiple photos

modify the camera

Noise reduction

Take several images and average them

Why does this work?

Basic statistics:

variance of the mean decreases with n:

Field of view

We can artificially increase the field of view by compositing several photos together (project 2).

Improving resolution: Gigapixel images

A few other notable examples:

Obama inauguration (gigapan.org)

HDView (Microsoft Research)

Improving resolution: super resolution

What if you don’t have a zoom lens?

Intuition (slides from Yossi Rubner & Miki Elad)

Slide 10

Slide 11

Slide 12

Slide 13

Intuition

Slide 15

Handling more general 2D motions

Super-resolution

Basic idea:

define a destination (dst) image of desired resolution

assume mapping from dst to each input image is known

usually a combination of a 2D motion/warp and an average (point-spread function)

can be expressed as a set of linear constraints

sometimes the mapping is solved for as well

add some form of regularization (e.g., “smoothness assumption”)

can also be expressed using linear constraints

but L1, other nonlinear methods work better

How does this work? [Baker & Kanade, 2002]

Limits of super-resolution [Baker & Kanade, 2002]

Performance degrades significantly beyond 4x or so

Doesn’t matter how many new images you add

space of possible (ambiguous) solutions explodes quickly

Major cause

quantizing pixels to 8-bit gray values

Possible solutions:

nonlinear techniques (e.g., L1)

better priors (e.g., using domain knowledge)

Baker & Kanade “Hallucination”, 2002

Freeman et al. “Example-based super-resolution”

Dynamic Range

HDR images — merge multiple inputs

HDR images — merged

Camera is not a photometer!

Limited dynamic range

8 bits captures only 2 orders of magnitude of light intensity

We can see ~10 orders of magnitude of light intensity

Unknown, nonlinear response

pixel intensity ¹ amount of light (# photons, or “radiance”)

Solution:

Recover response curve from multiple exposures, then reconstruct the radiance map

Slide 24

Calculating response function

Debevec & Malik [SIGGRAPH 1997]

The Math

Let g(z) be the discrete inverse response function

For each pixel site i in each image j, want:

Solve the over-determined linear system:

Capture and composite several photos

Same trick works for

field of view

resolution

signal to noise

dynamic range

Focus

But sometimes you can do better by modifying the camera…

Focus

Suppose we want to produce images where the desired object is guaranteed to be in focus?

Or suppose we want everything to be in focus?

Light field camera [Ng et al., 2005]

Conventional vs. light field camera

Prototype camera

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Slide 33

Simulating depth of field

stopping down aperture = summing only the central portion of each microlens

Digital refocusing

refocusing = summing windows extracted from several microlenses

Example of digital refocusing

All-in-focus

If you only want to produce an all-focus image, there are simpler alternatives

E.g.,

Wavefront coding [Dowsky 1995]

Coded aperture [Levin SIGGRAPH 2007], [Raskar SIGGRAPH 2007]

can also produce change in focus (ala Ng’s light field camera)

Slide 38

Slide 39

Slide 40

Slide 41

Close-up

Motion blur removal

Instead of coding the aperture, code the...

Slide 44

Slide 45

Many more possibilities

Seeing through/behind objects

Using a camera array (“synthetic aperture”)

Levoy et al., SIGGRAPH 2004

Removing interreflections

Nayar et al., SIGGRAPH 2006

Family portraits where everyone’s smiling

Photomontage (Agarwala at al., SIGGRAPH 2004)

…

More on computational photography

SIGGRAPH course notes and video

Other courses

MIT course

CMU course

Stanford course

Columbia course

Wikipedia page

Symposium on Computational Photography

ICCP 2009 (conference)


	Infinite resolution

	Infinite zoom control

	Desired object(s) are in focus

	No noise

	No motion blur

	Infinite dynamic range (can see dark and bright things)

	...


	The “analog” camera has changed very little in >100 yrs
		we’re unlikely to get there following this path

	More promising is to combine “analog” optics with computational techniques
		“Computational cameras” or “Computational photography”

	This lecture will survey techniques for producing higher quality images by combining optics and computation

	Common themes:
		take multiple photos
		modify the camera


	Take several images and average them

	Why does this work?

	Basic statistics:
		variance of the mean decreases with n:


	We can artificially increase the field of view by compositing several photos together (project 2).


	A few other notable examples:
		Obama inauguration (gigapan.org)
		HDView (Microsoft Research)


Basic idea:
	define a destination (dst) image of desired resolution
	assume mapping from dst to each input image is known
		usually a combination of a 2D motion/warp and an average (point-spread function)
		can be expressed as a set of linear constraints
		sometimes the mapping is solved for as well
	add some form of regularization (e.g., “smoothness assumption”)
		can also be expressed using linear constraints
		but L1, other nonlinear methods work better


Performance degrades significantly beyond 4x or so
Doesn’t matter how many new images you add
	space of possible (ambiguous) solutions explodes quickly
Major cause
	quantizing pixels to 8-bit gray values

Possible solutions:
	nonlinear techniques (e.g., L1)
	better priors (e.g., using domain knowledge)
		Baker & Kanade “Hallucination”, 2002
		Freeman et al. “Example-based super-resolution”


	Limited dynamic range
		8 bits captures only 2 orders of magnitude of light intensity
		We can see ~10 orders of magnitude of light intensity

	Unknown, nonlinear response
		pixel intensity ¹ amount of light (# photons, or “radiance”)

	Solution:
		Recover response curve from multiple exposures, then reconstruct the radiance map


	Let g(z) be the discrete inverse response function
	For each pixel site i in each image j, want:
	Solve the over-determined linear system:


	Same trick works for
		field of view
		resolution
		signal to noise
		dynamic range
		Focus

	But sometimes you can do better by modifying the camera…


	Suppose we want to produce images where the desired object is guaranteed to be in focus?





	Or suppose we want everything to be in focus?


	4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens


	stopping down aperture = summing only the central portion of each microlens


	refocusing = summing windows extracted from several microlenses


If you only want to produce an all-focus image, there are simpler alternatives

E.g.,
	Wavefront coding [Dowsky 1995]
	Coded aperture [Levin SIGGRAPH 2007], [Raskar SIGGRAPH 2007]
		can also produce change in focus (ala Ng’s light field camera)


	Seeing through/behind objects
		Using a camera array (“synthetic aperture”)
		Levoy et al., SIGGRAPH 2004

	Removing interreflections
		Nayar et al., SIGGRAPH 2006

	Family portraits where everyone’s smiling
		Photomontage (Agarwala at al., SIGGRAPH 2004)

	…


	SIGGRAPH course notes and video
	Other courses
		MIT course
		CMU course
		Stanford course
		Columbia course
	Wikipedia page
	Symposium on Computational Photography
	ICCP 2009 (conference)