A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent...

168
KATHOLIEKE UNIVERSITEIT LEUVEN FACULTEIT TOEGEPASTE WETENSCHAPPEN DEPARTEMENT COMPUTERWETENSCHAPPEN AFDELING INFORMATICA Celestijnenlaan 200A – B-3001 Leuven – Belgi¨ e A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE-BASED RELIGHTING Promotor: Prof. Dr. ir. Yves D. Willems Prof. Dr. ir. Philip Dutr´ e Proefschrift voorgedragen tot het behalen van het doctoraat in de toegepaste wetenschappen door Vincent MASSELUS December 2004

Transcript of A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent...

Page 1: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

KATHOLIEKE UNIVERSITEIT LEUVENFACULTEIT TOEGEPASTE WETENSCHAPPENDEPARTEMENT COMPUTERWETENSCHAPPENAFDELING INFORMATICACelestijnenlaan 200A – B-3001 Leuven – Belgie

A PRACTICAL FRAMEWORK FOR FIXEDVIEWPOINT IMAGE-BASED RELIGHTING

Promotor:Prof. Dr. ir. Yves D. WillemsProf. Dr. ir. Philip Dutre

Proefschrift voorgedragen tothet behalen van het doctoraatin de toegepaste wetenschappen

door

Vincent MASSELUS

December 2004

Page 2: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 3: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

KATHOLIEKE UNIVERSITEIT LEUVENFACULTEIT TOEGEPASTE WETENSCHAPPENDEPARTEMENT COMPUTERWETENSCHAPPENAFDELING INFORMATICACelestijnenlaan 200A – B-3001 Leuven – Belgie

A PRACTICAL FRAMEWORK FOR FIXEDVIEWPOINT IMAGE-BASED RELIGHTING

Committee:Prof. Dr. ir. Paul Van Houtte, chairmanProf. Dr. ir. Yves D. Willems, advisorProf. Dr. ir. Philip Dutre, advisorProf. Dr. ir. Marc Van BarelProf. Dr. ir. Luc Van GoolProf. Dr. Philippe Bekaert

(Limburgs Universitair Centrum)Prof. Dr. Hans-Peter Seidel

(Max-Planck-Institut fur Informatik)Prof. Dr. Paul E. Debevec

(University of Southern California)

Proefschrift voorgedragen tothet behalen van het doctoraatin de toegepaste wetenschappen

door

Vincent MASSELUS

U.D.C. 681.3*I3

December 2004

Page 4: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

c© Katholieke Universiteit Leuven — Faculteit Toegepaste WetenschappenArenbergkasteel, B-3001 Heverlee, Belgium

Alle rechten voorbehouden. Niets uit deze uitgave mag worden vermenigvuldigd en/ofopenbaar gemaakt worden door middel van druk, fotocopie, microfilm, elektronischof op welke andere wijze ook zonder voorafgaande schriftelijke toestemming van deuitgever.

All rights reserved. No part of the publication may be reproduced in any form byprint, photoprint, microfilm or any other means without written permission from thepublisher.

D/D/2004/7515/85

ISBN 90-5682-547-X

Page 5: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

A Practical Framework for FixedViewpoint Image-Based Relighting

Vincent Masselus

Department of Computer Science,Katholieke Universiteit Leuven

ABSTRACT

Computer graphics allow to create increasingly more fantastic yet still believableimages and the range of applications continues to grow in many areas such as themovie industry and computer games. In order to maintain the creative freedom ofvirtual reality and augment the realism of the resulting images, the visualized sceneshave become a mixture of existing real objects and virtually created objects. Thisdissertation deals with the visualization of real objects with novel incident, possiblyvirtual illumination, which is also called relighting.

An image-based relighting technique requires only to take photographs of the ob-ject under certain conditions, and no other information, such as a geometric descrip-tion or reflectance properties of the materials of the object, is required. In this work,we choose for an image-based relighting technique and out of practical considerations,the user is restricted to a fixed point of view to the object.

In the first part of this work, we design a framework for fixed viewpoint image-based relighting techniques. In this framework a function defining the transport ofincident light through the object space, is captured. This function is called the re-flectance field of the object. To acquire this reflectance field, a set of illuminationconditions needs to be chosen and the object is photographed from the fixed view-point while illuminated with each of these illumination conditions. From the resultingset of images, the reflectance field of the object is constructed. Once the reflectancefield is obtained, relighting the object boils down to evaluating the reflectance fieldwith the desired incident illumination.

In the second part, we demonstrate that previously published relighting techniquesfit in this framework, which allows to fairly compare these techniques. Some newlydeveloped relighting techniques are presented in this framework as well. A first set oftechniques deals with relighting real objects with restricted 2D illumination. For thiskind of relighting, a very practical data acquisition method is presented, which onlyrequires a hand-held light source, a camera and 4 white diffuse spheres. Additionally,more intelligent data processing techniques of the resulting set of images are presentedand compared, allowing to construct a better reflectance field of the object. Secondly,this work also provides the first presented technique for relighting a real object withnon-restricted 4D illumination, which enables to create spotlight effects or visualizethe object partly in shadow. The improvements presented for relighting with restricted2D illumination are transferred to relighting with 4D illumination as well.

i

Page 6: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

The research that led to this dissertation has been supported by a grant from the Fund for Scientific Research – Flanders

(F.W.O. grant #G.0354.00).

Page 7: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

AcknowledgementsArriving at the end of my Ph.D., I have not only learned a lot from books, papers and websites,but I also have had the opportunity to meet a lot of people who helped in all kinds of ways.Although they deserve much more than a few words, it is still my great privilege to thank themhere.

First, I would like to thank my two advisors, Prof. Willems and Prof. Dutre, for helping meto get started on this journey and for guiding me in the right directions. They were always thereto assist me and help me through all my troubles, each in their own specific way.

Also Prof. Van Houtte for accepting to be the chairman of my evaluation committee and toProf. Van Barel and Prof. Van Gool to be part of my reading committee and taking the time tomake insightfull comments on the text.

I want to thank Prof. Seidel for accepting to be in my evaluation committee. Also manythanks to Philippe Bekaert, for being in my jury. He is a man I humbly look up to ever sinceI was his master student, and with whom I even had the pleasure of working with in the earlybeginning of my research. As everybody who is dear to me knows, I’m extremely pleased thatPaul Debevec agreed to be in my jury. I consider this to be a real honor and a privilege.

I would like to thank all of my current and former colleagues in the graphics group, forcreating an inspiring research environment and a fun place to work. Especially many thanks toPieter Peers who has not only been a fun conference companion and inspiring discussant, butalso a wonderful office mate. A very special thanks goes to one of my former colleagues, FrankSuykens, who never really left the graphics group and often astonished me with his funny butinnovative look at my problems. I am not only in debt with the people of the graphics researchgroup but with the whole department of computer science, for all administrative issues, therelaxing coffee breaks and the fun chats.

This is also the time and place to thank a person that has made a true impression on me:Fabien Decruyenaere. He was one of the first to help me exploit my capabilities, I consider hisinspiration one of the first steps towards this work and I thank him for that.

I can not thank enough my family and all of my friends, for the great support and encour-agement during all this time. Especially my mom and dad, who understood how important thiswas for me and for letting me pursue this dream. Thank you so much. And also my brotherFrederik, for I have yet to meet another man who believes more in me than he does.

Finally, I’d like to thank my girlfriend Isabel, who unconditionally supported and loved meand made a true difference.

Vincent Masselus,Heverlee – December 2004.

Aan Isabel

Page 8: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 9: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Contents

Abstract i

Acknowledgements iii

Table of Contents v

1 Introduction 11.1 Mixed Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Image-based Relighting . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Overview of the Dissertation . . . . . . . . . . . . . . . . . . . . . . 7

2 General Image-based Relighting 92.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Light . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.1 Radiometry . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.2 Linearity of Light Transport . . . . . . . . . . . . . . . . . . 112.2.3 Rendering Equation . . . . . . . . . . . . . . . . . . . . . . 122.2.4 Plenoptic Function . . . . . . . . . . . . . . . . . . . . . . . 142.2.5 Light Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 Mathematical prerequisites . . . . . . . . . . . . . . . . . . . . . . . 162.3.1 Inner Product and Orthogonality . . . . . . . . . . . . . . . . 162.3.2 Orthogonal Projection of a Function on a Subspace . . . . . . 172.3.3 Calculating the Inner Product of Two functions . . . . . . . . 17

2.4 Image-based Relighting Framework . . . . . . . . . . . . . . . . . . 182.4.1 Scene Subdivision . . . . . . . . . . . . . . . . . . . . . . . 192.4.2 Relighting Equation . . . . . . . . . . . . . . . . . . . . . . 202.4.3 Reflectance Field . . . . . . . . . . . . . . . . . . . . . . . . 212.4.4 General Image-based Relighting Method . . . . . . . . . . . 232.4.5 Fast Relighting . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.5 Practical Image-based Relighting . . . . . . . . . . . . . . . . . . . . 252.5.1 Parameterization Schemes . . . . . . . . . . . . . . . . . . . 262.5.2 Capturing Incident Illumination . . . . . . . . . . . . . . . . 302.5.3 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . 35

v

Page 10: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

vi CONTENTS

2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3 Relighting with 2D Illumination 413.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.1.1 Relighting Equation Revisited . . . . . . . . . . . . . . . . . 423.1.2 General Image-based Relighting Method Revisited . . . . . . 43

3.2 Pulse Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.2.1 Light Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.2.2 Free-form Light Stage . . . . . . . . . . . . . . . . . . . . . 52

3.3 Structured Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 633.3.1 Steerable Functions . . . . . . . . . . . . . . . . . . . . . . . 653.3.2 Basis Functions as Illumination Conditions . . . . . . . . . . 66

3.4 Relighting with Spatially Dependent Illumination . . . . . . . . . . . 673.4.1 The General Framework Revisited, again . . . . . . . . . . . 673.4.2 Pulse Approach . . . . . . . . . . . . . . . . . . . . . . . . . 683.4.3 Structured Approach . . . . . . . . . . . . . . . . . . . . . . 68

3.5 Some Notes on Relighting with 2D Illumination . . . . . . . . . . . . 733.5.1 Restriction to 2D Illumination . . . . . . . . . . . . . . . . . 733.5.2 Consequences for Relighting with 2D Illumination . . . . . . 743.5.3 Environment Maps Expressing 2D illumination . . . . . . . . 75

3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4 Relighting with 4D Illumination 774.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.2 Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.2.1 Illumination Conditions . . . . . . . . . . . . . . . . . . . . 794.2.2 Data Acquisition Setup . . . . . . . . . . . . . . . . . . . . . 81

4.3 Reflectance Function Construction . . . . . . . . . . . . . . . . . . . 844.4 Relighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 864.5 Data Acquisition Speedup . . . . . . . . . . . . . . . . . . . . . . . 89

4.5.1 Basic Speedup Technique . . . . . . . . . . . . . . . . . . . 904.5.2 Influence Parameter . . . . . . . . . . . . . . . . . . . . . . 91

4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

5 2D Reflectance Functions 955.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.2 2D Reconstruction Techniques . . . . . . . . . . . . . . . . . . . . . 95

5.2.1 Zero-order Hold Reconstruction . . . . . . . . . . . . . . . . 965.2.2 Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . 985.2.3 Multilevel B-Splines . . . . . . . . . . . . . . . . . . . . . . 1015.2.4 Other Reconstruction Methods . . . . . . . . . . . . . . . . . 103

5.3 Comparison for 2D Reconstruction Techniques . . . . . . . . . . . . 1085.3.1 Numerical Comparison . . . . . . . . . . . . . . . . . . . . . 1085.3.2 Visual Comparison for a Single Image . . . . . . . . . . . . . 1095.3.3 Visual Comparison for a Series of Images . . . . . . . . . . . 111

Page 11: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

CONTENTS vii

5.4 Compact Representation of 2D Reflectance Functions . . . . . . . . . 1145.4.1 Box functions . . . . . . . . . . . . . . . . . . . . . . . . . . 1145.4.2 Spherical Harmonic Basis Functions . . . . . . . . . . . . . . 1175.4.3 Wavelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

6 4D Reflectance Functions 1256.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1256.2 Different Illumination Patterns . . . . . . . . . . . . . . . . . . . . . 126

6.2.1 General Solution . . . . . . . . . . . . . . . . . . . . . . . . 1266.2.2 Fast Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 133

6.3 4D Reconstruction Techniques . . . . . . . . . . . . . . . . . . . . . 1356.3.1 Rewriting the 4D Reconstruction Technique . . . . . . . . . . 1366.3.2 General 4D Reconstruction Technique . . . . . . . . . . . . . 1366.3.3 Instantiations of the General 4D Reconstruction Technique . . 137

6.4 Compact Representation of 4D Reflectance Functions . . . . . . . . . 1386.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

7 Conclusion 1437.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1437.2 Original Contributions . . . . . . . . . . . . . . . . . . . . . . . . . 1447.3 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

7.3.1 Better Reflectance Function Reconstruction . . . . . . . . . . 1467.3.2 Design new Data Acquisition Devices . . . . . . . . . . . . . 1467.3.3 Camera Movement . . . . . . . . . . . . . . . . . . . . . . . 1477.3.4 Other Avenues . . . . . . . . . . . . . . . . . . . . . . . . . 147

Publications 149

Bibliography 151

Page 12: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 13: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 1

Introduction

A couple of decades ago, only a few people could have imagined what computergraphics can do today. A computer gamer can be put into the most fantastic worldsand experience action, unthinkable in the real world. An architect can show all themarvelous design concepts to a potential buyer, without actually having to constructthem. A military pilot can undergo realistic training without facing any real danger.An audience actually cares for a character in a movie, although the character is com-pletely virtual. All these different aspects are possible thanks to the recent advancesin realistic image synthesis of virtual worlds.

The virtual environments created in these applications are often referred to as vir-tual reality, as opposed to the reality we live in. To make images even more realisticand believable, while maintaining the artistic and imaginative freedom of the creator,reality and virtual reality are often mixed. The environments consisting of both realand virtual objects are called mixed reality.

This thesis contributes to the research field of computer graphics, more specificallyto image synthesis for mixed reality.

1.1 Mixed RealityAn environment in mixed reality consists of both real and virtual objects. Merging avirtual object into a real environment is often referred to as augmented reality. An ex-ample of augmented reality is adding a virtual character into a real world environment.Augmenting virtual reality with a real object is called augmented virtuality [5]. Anexample of an application of this is allowing a real actor to walk around in a completevirtual environment.

An image of a single object whether virtual or real can be created with the ap-propriate tools. To display a virtual object, rendering algorithms are used to computethe image. These algorithms simulate nature and require all visual information of theobject, such as a geometric description of the object, the reflectance properties of thematerials of the object and all lighting conditions. On the other hand, an image of areal object can simply be created by recording a photograph of the object.

1

Page 14: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2 CHAPTER 1. INTRODUCTION

To generate an image of a mixed reality scene, one cannot merely combine theimages of virtual objects with the images of real objects. This would not result ina convincing image. An image will only be believable if it is consistent with theexpectation of how it should look like. As an example, we create an image of a virtualtea pot and tea cup on a real table (top of figure 1.1). Obviously, the tea pot and cup,and the table need to be viewed from the same relative viewpoint for the resultingimage to be convincing. Observers will also expect light interaction between the teapot and the table, such as a shadow of the tea pot cast onto the table or a reflection ofthe tea cup in the specular box. If the image is not consistent with the expectation, theobserver will doubt the image and assume the tea pot was not really on the table, butadded to the picture (bottom of figure 1.1).

Real tableVirtual thee pot and thee cup

Virtual thee pot and thee cup on a real table

Figure 1.1: A simple composition of an image of a virtual tea pot and tea cup in animage of a real table does not result in a convincing image, due to the lack of shadowsand inter-reflections.

To achieve photo-realistic convincing images of mixed reality scenes, all objectsin the scene have to be visualized together and we need to pay attention to the rela-tions between all objects, by creating shadows and inter-reflections. This can only beachieved by visualizing all objects of the scene at once and thus, by representing allobjects either into reality or into virtual reality.

If we choose to represent all objects into reality, an image of the real scene canbe obtained by simply taking a photograph of the real objects together with the con-structed virtual objects in reality. However, in this case all the virtual objects need tobe created in reality. In the example of figure 1.1, this would require to create a real

Page 15: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

1.1. MIXED REALITY 3

tea pot and a real tea cup which have exactly the same properties as their virtual coun-terparts. Creating virtual objects in reality is very difficult, even with a 3D printer, butcan also be impossible for some virtual objects, which makes this method impractical.

On the other hand, all real objects can be represented in the virtual reality. Thisis traditionally done by acquiring the geometric description of the objects and the re-flectance properties of all their materials. Using a 3D scanner, the geometry of realobjects can be captured. This allows to obtain a 3D model containing all fine geomet-ric details. However, acquiring the reflectance properties of the materials for a realobject is not straightforward. Reflectance properties describe the interaction of lightwith the material, which consists of reflection, absorption and subsurface scattering ofthe light. Capturing this interaction can be difficult and time consuming. Moreover,for each point on an object, the material may be different, which makes the acqui-sition of the reflectance properties of a real object even more difficult. A traditionalsolution to this problem is the use of analytically defined reflectance properties. Forreal objects, the real reflectance properties are sampled and approximated using theseanalytically defined material properties. In the example of the tea pot and tea cup onthe real table, the virtual counterparts of the table and objects on it can be obtainedby measuring the geometry and roughly approximating the reflectance properties (topleft of figure 1.2). This allows to visualize the virtual objects (tea pot and cup) onthe virtual table, revealing all the inter-reflections and shadows between the objects(top right figure 1.2). Based on these two images (on top), the virtual objects can befaithfully merged into the real photograph, using a technique proposed by Debevec etal. [11] (figure 1.2 bottom).

The sampling of the reflectance properties is required for all materials and theapproximations for these properties need to be very good for objects we often see,otherwise the observer will notice the object has been tampered with and the credibil-ity of the image will go down. That is why, although extremely realistic images canbe created, it is still difficult to visualize a virtual human being or any common object.

Another way to represent real objects in virtual reality is not by acquiring thegeometric description and reflectance properties of all the materials, but by capturingthe appearance of the object. If the image looks like the real object it will be perceivedas the real object and it will be considered as a real object. Using photographs of theobject recorded from different viewpoints, image-based rendering techniques allow tocreate images of the object viewed from arbitrary viewpoints. These techniques arebased on interpolation between recorded images and they do not require the difficulttask of measuring reflectance properties. However, with such a technique, the realobject can only be visualized with the lighting conditions present at the photo shootof the object.

As previously noted, the images of mixed reality scenes will only be convincingif light interaction between real and virtual objects can be noticed. Therefore, theappearance of a real object needs to be captured independently of the incident illu-mination at the time of the photo shoot. This will allow to visualize the real objectwith arbitrary incident illumination. Techniques capable of capturing the appearanceof a real object and visualize the object with arbitrary novel incident illumination arecalled image-based relighting techniques.

Page 16: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4 CHAPTER 1. INTRODUCTION

Virtual thee pot and thee cup on virtualtable with shadows and inter-reflections

Virtual model of the table andillumination around the table

Virtual thee pot and thee cup on a realtable with shadows and inter-reflections

Figure 1.2: By using a virtual version of the table and the surrounding illumination(top left), the interaction between the objects on the table and the virtual tea pot and teacup can be revealed (top right). These two images contain all the information requiredto merge the virtual objects into the real photograph resulting in a faithful image ofthe tea pot and tea cup on the table (bottom).

1.2 Image-based RelightingImage-based relighting techniques capture the appearance of real objects, indepen-dently of the incident illumination, and allow the visualization of these real objectswith novel incident illumination. In all of these techniques, a data acquisition methodis designed to obtain all necessary information to visualize the object with illuminationthat is not known at the time of the data acquisition. Furthermore, these techniquesalso define a data processing stage to enable to view the object with new incidentillumination. The image-based relighting methods generally consist of three stages:

1. The appearance of the object needs to be acquired, independently of the incidentillumination and without knowledge of the geometry or reflectance propertiesof the materials of the object. To acquire this appearance of the object, inde-pendently of the illumination, a set of illumination conditions is chosen. Anexample of such a set of illumination conditions is light originating from a set

Page 17: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

1.3. CONTRIBUTIONS 5

of known directions. Another example is a set of differently positioned anddifferently oriented spot lights.

The object is photographed while illuminated with each of these illuminationconditions separately. The resulting set of images represents the appearance ofthe object and is called the set of basis images.

2. After the photo shoot, the recorded photographs are processed, so that the dataare suitable to visualize the object with arbitrary new incident illumination. Thisprocessing procedure depends completely on the illumination conditions cho-sen.

3. To relight the object, the processed data is used along with the given incidentillumination. An image of the object illuminated with that novel illuminationcan then be created.

Note that only photographs have been taken and thus, no other information is re-quired of the object. The main distinction between different image-based relightingtechniques is often the choice of illumination conditions. The choice for some setof illumination conditions has a large impact on the data acquisition stage and thedata processing stage. Depending on this choice, the number of illumination condi-tions may be large or small. Moreover, the object needs to be illuminated with theseconditions, in order to record the basis images. Therefore, the data acquisition setupdepends on these illumination conditions as well. An even more important aspect ofthis choice is that the chosen kind of illumination conditions will limit the possibleillumination effects in the images of the relit object.

In our work, we will restrict the camera to a single viewpoint to the object. Thismeans the object will always be visualized from this fixed viewpoint while the inci-dent illumination can be altered. However, this does not reduce the applicability ofthe presented algorithms. All the techniques discussed in this work can be used forrelighting algorithms where the viewpoint can be changed. For such algorithms, thedata acquisition of our techniques could be repeated for multiple viewpoints. By com-bining this information, as proposed by Matusik et al. [37], the object can be viewedfrom an arbitrary viewpoint illuminated with arbitrary incident illumination.

1.3 ContributionsWe present a framework for general relighting of real objects with novel incident illu-mination. This framework is derived from the basic properties of light and light trans-port and is based on a single equation, which we call the relighting equation. Fromthis equation, we derive a general relighting method to capture the appearance of anobject independently of the incident illumination and to visualize the real object witharbitrary novel incident illumination. In order to be practical, we present a generalimage-based data acquisition approach for the appearance of the object, independentof the illumination.

To demonstrate the generality of the framework, we show throughout this disser-tation that previously published relighting techniques can be fitted in the framework.

Page 18: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6 CHAPTER 1. INTRODUCTION

This framework will be the backbone of our work and all our newly developed andpreviously published relighting techniques are treated as instantiations of the generalframework:

• Free-form Light Stage: This constitutes a very practical and inexpensive dataacquisition technique, with accompanying data processing method, for relight-ing real objects with angularly varying incident illumination.

The data acquisition requires a camera, a hand-held light source and a set of foursmall white diffuse spheres. The white spheres are placed around the object andthe camera takes photographs continuously, while the user moves the hand-heldlight source freely over and around the object. The data from the resultingset of photographs is processed and allows to relight the object with any noveldirectionally dependent incident illumination.

This technique was presented at the EuroGraphics Workshop on Rendering2002 as a full paper and as a technical sketch at the SIGGRAPH 2002 annualconference.

• Relighting with 4D Incident Light Fields: The first presented data acquisitiontechnique to relight real objects with angularly and positionally varying incidentillumination.

The object is placed in a specially designed gantry with a projector used as lightsource. By moving the projector to different positions and using the angularrange of the projector, a set of photographs of the object can be recorded withspecially chosen illumination for each photograph. In this way, all data neces-sary to visualize the object partially in shadow or with spot light effects can beobtained.

This technique was presented at the SIGGRAPH 2003 annual conference as afull paper.

• Smooth Reconstruction and Compact Representation of Reflectance Func-tions: Instead of using the acquired data captured during the photo shoot inits original form for the relighting, more intelligent data processing can be per-formed on the data.

We implemented and tested several techniques well known in the signal pro-cessing literature in the field of image-based relighting. These techniques, suchas linear interpolation, reconstructing a function with multilevel B-Splines andusing the wavelet transform, produce better images of the relit object. However,the data storage and the computation time may increase for these techniques. Tosolve this, the processed data may be approximated with box functions, spher-ical harmonic basis functions or a non-linear approximation using higher orderwavelets.

A part of this work was presented at the EuroGraphics Symposium on Render-ing 2004 as a full paper. Other new ideas in this field, especially extensions togeneral illumination relighting, are discussed in this dissertation but have notyet been published.

Page 19: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

1.4. OVERVIEW OF THE DISSERTATION 7

1.4 Overview of the DissertationThe remainder of this dissertation is organized as follows:

• Chapter 2 introduces the main properties of light along with basic light trans-port theory. Based on these properties, a general framework for image-basedrelighting is deduced. This framework comprises the general relighting equa-tion and a practical image-based relighting method based on this equation. Allthe relighting techniques discussed in our work will be fitted in this framework.

In order to relight the real objects, one can use the general relighting methodof this framework to acquire the data of the object, along with incident illumi-nation. Therefore, in this chapter, a survey of different incident illuminationacquisition techniques is given as well.

• In chapter 3, image-based relighting techniques using only angularly varying in-cident illumination or only spatially varying incident illumination are expressedin our framework. These techniques restrict the possible incident illuminationto a subset of general incident illumination. This restriction allows to reducethe relighting equation to a simpler form, resulting in a large variety of practicalrelighting techniques.

• In chapter 4, a basic relighting technique is discussed allowing to relight a realobject with all possible illumination effects. Since this requires using the re-lighting equation in its original form, this technique is not practical, due to thevery long data acquisition time. An improvement to the proposed data acquisi-tion method is presented to solve this.

• Chapter 5 surveys more intelligent techniques to process the acquired data torelight objects with angularly varying incident illumination. These techniquesuse the standard acquisition methods of chapter 3, but allow to obtain betterresults, especially for animated incident illumination. Because of the large datastorage and the long computation time of the resulting relighting techniques,compact representations of the processed data are discussed and compared aswell.

• Chapter 6 discusses qualitative improvements for relighting with general inci-dent illumination. These adjustments to the basic technique of chapter 4 arebased on the techniques developed in chapter 5. In order to improve the practi-cality of the resulting technique, quantitative improvements on the data storageare presented as well.

• Chapter 7 concludes the dissertation with a discussion of the contributions ofthe presented work along with some avenues for future research.

Page 20: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 21: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 2

General Image-basedRelighting

2.1 Introduction

The aim of our work is to enable the visualization of real objects under novel incidentillumination without acquiring the geometric description or reflectance properties forall the materials of the object. In order to achieve this, we introduce a practical frame-work for image-based relighting.

Since light will be one of the major actors in our work, we start with a surveyof definitions involving light and discuss some properties of light and light transport(section 2.2). Additionally, some mathematical tools useful in our work are reviewed(section 2.3).

Based on these properties and tools, a framework for image-based relighting isconstructed in which the relighting equation is introduced (section 2.4). This relight-ing equation states how the object is observed depending on the appearance of theobject under given incident illumination. In order to be able to use this equation, apractical method to capture the appearance of an object, independent of the incidentillumination, is needed. Such a method is derived in the framework as well, in theform of an image-based method. This framework will be used in the rest of our workto discuss different image-based relighting techniques, each with its own advantagesand drawbacks.

To make this framework of practical use, some general issues for image-basedrelighting are discussed (section 2.5). These issues include different methods to cap-ture incident illumination, which can later be used to relight the real objects. Cameracalibration for the image-based data acquisition is surveyed as well.

9

Page 22: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

10 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

2.2 LightIn this section we introduce several definitions involving light and survey some basicproperties of light, such as the light propagation in an environment, linearity of lighttransport and common ways to represent the light distribution in an environment.

2.2.1 Radiometry

Instead of dealing with light as an abstract entity, we view it from a physical point ofview. We define the units flux, irradiance and radiance as:

• Flux, denoted by Φ (= dQdt ), is power, or radiation energy Q per time unit and is

measured in Watt (W ).

• Irradiance, denoted by E (= dΦdA ), is incident flux on a differential area dA from

all directions above this area. Irradiance is expressed in Wm2 .

• Radiance L (= dΦd⊥ω dA ), is power per unit projected area per unit solid angle and

is measured in Wm2×sr .

Radiometric quantities also depend on the wavelength λ of the light. In our al-gorithms, we will not use every wavelength λ of the full spectrum of all light inde-pendently. As is common in computer graphics, we work in RGB space and expresslight with a triplet of values, representing three color channels red, green and blue.Each color channel is treated independently of the others and in the remainder of thistext, we always assume to be working with one of these channels, except when notedotherwise.

Radiance L along any ray in vacuum remains constant. In this work, we willassume all objects are in vacuum space and we can therefore exploit this property. Adirection can be expressed using the polar coordinate parameterization as (φ,θ), whereφ represents the azimuth angle and θ represents the tilt or elevation angle. More detailsabout directions and parameterizations will be given in section 2.5.1. We define Ω asthe space of all directions:

Ω = [0,2π]× [0,π]

and introduce the notation ω as the direction (φ,θ). With this notation, we denoteL(x→ ω) as the radiance value from point x into direction ω, and L(x← ω) the radi-ance incident in point x from direction ω. Note that the arrow ’←’ indicates incidentillumination at a point from a direction and the arrow ’→’ indicates illumination leav-ing a point in a direction.

Additionally, we denote the opposite direction of ω as −ω. The ray starting inpoint x in direction ω is written as r(x,ω). Due to the property of radiance beingconstant along a ray (in vacuum), the first point y that the ray r(x,−ω) hits will alsohave radiance value L(x→ ω), if x is a free-standing point (x is not on a surface).

Page 23: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.2. LIGHT 11

2.2.2 Linearity of Light TransportAn important property of light is the linearity of light transport. This means that theeffect of different illumination conditions on an environment can be linearly combinedto obtain the effect of all the illumination conditions together on that environment.Linear combining consists of an additive operation and a scaling operation.

Super-position of Light Transport

This additive operation can be illustrated easily with a simple example. Imagine ascene with two white light sources. We record the irradiance E1 at a small area ofthe scene with one light source on and the other light source switched off. Then weswitch off the first light source, turn on the other light source and record the irradianceE2 for that same area. By adding these two irradiance values (E1 + E2), we obtain thesame irradiance value as we would have recorded at that area in the scene with bothlight sources on simultaneously. This holds for all areas in a static scene. By static,we mean the objects and light sources do not move or chance reflectance properties.

If we store irradiance values for each pixel of an image plane, an example of thissuper-position can be see at the top row of figure 2.1.

Color Modulation of Light Transport

By working in a single RGB space under some assumptions, the color channels ofan irradiance image can be modulated and scaled independently to mimic the colorchanges of the incident illumination. If we have an image with irradiance values of anobject illuminated with a white light source (i.e. RGB values (1,1,1)), then an imageof the object illuminated with a colored light source with RGB values (r,g,b) can becreated by modulating the three color channels red, green and blue of the originalimage of the object with weights r, g and b respectively.

As an example, the two original images of the previous example are each modu-lated with a different color in figure 2.1.

The made assumptions are that the scene is static and that there is no cross-talkbetween color channels in the camera. This means, we assume that when only redlight is emitted, the resulting photograph will have pixels where only the values ofthe red color channel are non-zero. For real cameras this is often not the case: whileonly red light is emitted, the pixels still have a (very low) value for the green and bluechannels, due to the sensitivity of the different color sensors in the camera. In ourwork, we neglect this effect. A better solution would be to measure this cross-talkbetween color channels, as described in [14], based on the work of [61] and accountfor the effect in the relighting. However, there are still a lot of unresolved issues inthis matter [18].

These two operations, the additive operation of light transport and color modulation,can be combined. In our example, the image with the modulated color channels can

Page 24: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

12 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

+

+ =

=

modulate with modulate withcolorcolor

Figure 2.1: A miniature race car is illuminated from the left and right (top left and topmiddle). By summing the two images together, an image is created of the miniaturerace car with both light sources on (top right). By modulating the color channels of thesource images, the color of the light sources can be modulated (bottom left and bottommiddle). As demonstrated at the bottom right, both operations can be combined.

be added to the other modulated image, resulting in an image with one red light sourceand a green light source both illuminating the scene. See the bottom row of figure 2.1for an example. This also allows to obtain an image of the scene illuminated with anynumber of light sources emitting an arbitrary color, starting with an equal amount ofphotographs of the environment, each using a single light source for illumination.

2.2.3 Rendering Equation

Using the linearity of light transport, the light transport in an environment can becomputed as the sum of Le and Lr:

L(p→ ω) = Le(p→ ω) + Lr(p→ ω). (2.1)

This equation states that the radiance at a point p on a surface into direction ωis the sum of the emitted radiance Le of that point in direction ω by the object itselfand the reflected radiance of that point in that direction. The value of Le will only benon-zero if the point p, is a point on a light source. The reflected radiance from pointp in direction ω is the radiance incident in point p that is reflected into direction ω.

We use the function fr(ωin↔ p↔ω) as a function to describe the amount of inci-dent light that is reflected in point p from direction ωin into direction ω. This functionis often known as the Bi-directional Reflectance Distribution Function (BRDF). Thisfunction is reciprocal in the sense that the value for fr(ωin↔ p↔ ω) is equal to the

Page 25: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.2. LIGHT 13

p1

p2

L(p1← ωin)

r(p1,−ωin)L(p2→ ωin)

L(p2← ω′in)

L(p1→ ω)

Figure 2.2: A 2D representation of the computation of the radiance L(p1→ω), leavingpoint p1 into direction ω. To compute this radiance, the incomming radiance from alldirections ωin in the hemisphere above point p1 needs to be known. For any directionωin, this incident radiance L(p1← ωin) corresponds to radiance L(p2→ ωin). To findthis radiance, L(p2 ← ω′in) is required. To find this radiance, radiance leaving pointp1 may be required.

value of fr(ω↔ p↔ ωin), hence the two-way pointing arrows. Using this function fror BRDF, the reflected radiance Lr(p→ ω) can be expressed as the integral over alldirections ωin of radiance L(p← ωin) that is reflected into direction ω:

Lr(p→ ω) =

Z

Ωfr(ωin↔ p↔ ω) L(p← ωin) cos θin dωin. (2.2)

The term cos θin is the foreshortening term, dealing with the fact that light incidentat a grazing angle (large θin) is spread over a wider area than light incident orthogonalto the surface (i.e. θin ≈ 0).

In general, the light transport in a scene can be expressed using the renderingequation [27]:

L(p→ ω) = Le(p→ ω) +Z

Ωfr(ωin↔ p↔ ω) L(p← ωin) cos θin dωin. (2.3)

With this equation the global illumination solution in an environment can be cal-culated. Notice that equation 2.3 is a recursive equation: to calculate L(p1 → ω),L(p1 ← ωin) is required for all directions ωin in the hemisphere above point p1. Toknow L(p1 ← ωin), L(p2 → ωin) will be required, where p2 is the first hit of a rayr(p1,−ωin). However, to know L(p2→ ωin), L(p2← ω′in) will need to be known foreach direction ω′in in the hemisphere above point p2. Here starts the recursion: inorder to compute L(p2 ← ω′in), L(p1→ ω′in) will need to be known. As we are com-puting radiance from point p1 in a direction we fall in a never ending recursion. Seefigure 2.2 for an illustration of this recursion.

Several algorithms exist to solve this equation and a good overview of such globalillumination algorithms is given in [17]. Geometry-based rendering techniques are

Page 26: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

14 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

based on these algorithms and allow to visualize an object by using the geometricdescription of the object and the BRDFs of the materials of the object. However, thesecomplex algorithms are outside the scope of our work.

2.2.4 Plenoptic FunctionThe plenoptic function P, presented in [1], is defined as a seven dimensional (7D)function, representing the flow of light in an environment.

P = P(x,y,z,φ,θ,λ, t).

The function returns the radiance for a point in space (x,y,z) in direction (φ,θ) forwavelength λ at time t. We only allow static scenes and limit the scope to a singlewavelength. Hence, the light in an environment can be described by a reduced 5Dplenoptic function:

P = P(x,y,z,φ,θ).

Figure 2.3: The plenoptic function holds all radiance values for each point in space(i.e. on an object (in red) or not), in any direction.

We refer to figure 2.3 for an iconic representation of the plenoptic function wherethe radiance in each point in all directions is displayed. Although the rendering equa-tion (equation 2.3) only finds the radiance for a point on a surface in a direction, theradiance for any point (x,y,z) in space in direction (φ,θ) can be calculated. This isdone by finding the point on a surface that is the first intersection of the ray, starting inpoint (x,y,z) into the opposite direction of (φ,θ). This operation holds due to radiancebeing constant along a ray in vacuum.

Two plenoptic functions P1 and P2 of the same environment with illumination setsL1 and L2 respectively, can be added together to find the plenoptic function P of thatenvironment with both illumination sets L1 and L2 because of the linearity of lighttransport.

2.2.5 Light FieldsThe plenoptic function is a mathematical tool to represent the flow of light in an envi-ronment. However, it is mainly used to represent the appearance of a single object in

Page 27: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.2. LIGHT 15

a fixed lighting environment. In such an application, the plenoptic function is usuallyrestricted to the radiance originating from the object (i.e. emitted or reflected radianceby the object), and does not comprise other illumination in the environment.

Image-based rendering (IBR) techniques allow to visualize an object based onsuch an appearance of the object, in contrast to the geometry-based rendering tech-niques, based on the geometric description and material properties of the object. Manyof the IBR techniques are based on the 5D plenoptic function. Instead of calculatingthe plenoptic function, like the geometry-based rendering techniques, these IBR tech-niques sample the plenoptic function of an object. This sampling is performed byrecording photographs of the object from different viewpoints. The IBR techniquesthen reconstruct the complete plenoptic function of the object based on these sam-ples. From the reconstructed plenoptic function, the object can be visualized from anarbitrary viewpoint.

Often, IBR techniques assume the object is enclosed in a bounding volume and theviewer is outside this volume. The surface of this volume can be parameterized withtwo parameters (u,v). Since the object is completely in the volume, no changes inradiance values outside the volume are assumed and thus only the radiance values forall points on the surface (2D) of the volume in all directions (2D) out of the volumeneed to be stored. The plenoptic function for the object can thus be expressed as a 4Dfunction (figure 2.4). This dimensional reduction is possible thanks to the propertyof constant radiance along a ray in vacuum. As previously noted, the radiance for apoint x outside the volume in direction ω can be found by tracing a ray r(x,−ω). Thefirst intersection of this ray with the surface is a point (u,v) on the surface and for thispoint we can find the radiance value in direction ω or (φ,θ). This value is the radiancefor point x in direction (φ,θ), since radiance outside the volume does not change. 4Dplenoptic functions are often called light fields.

Figure 2.4: Left, the plenoptic function for the radiance originating from an object atsome selected points is depicted. On the right, the corresponding light field for thisobject, defined on the surface of a cube.

Notice the assumption of the object to be inside the volume and the viewer to beoutside the volume. Light fields defined with this assumption represent the appearanceof the object with a 4D function. These light fields only store radiance values fordirections exiting the volume. We call these light fields radiant light fields.

Different options have been proposed for the surface on which the light field isdefined. In [22, 33] this surface is chosen to be a cube enclosing the object, while

Page 28: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

16 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

in [26], the surface is a sphere. The choice for the surface has an impact on theresulting images. The resulting images of the object, viewed from positions far awayfrom sampled view points contain artefacts, if the chosen surface deviates much fromthe surface of the object. If the chosen surface equals the surface of the object, asproposed in [60, 8, 7], these artefacts disappear. Such radiant light fields are calledsurface light fields.

The reduction of the plenoptic function from five dimensions to four dimensionscan be achieved for environments as well. The partitioning of space is then the otherway around: the viewer is in a volume and the environment is everything outside thatvolume (figure 2.5). The enclosing surface is chosen in a way that radiance values ofthe environment do not chance within the volume and only radiance values enteringthe volume are stored. These light fields are called incident light fields.

Figure 2.5: Left, the plenoptic function for the radiance originating from an environ-ment is depicted. On the right, the corresponding incident light field for this environ-ment, defined on a surface of a cube.

Note that incident light fields and radiant light fields are both just light fields andare similar in every way. The two names just help to make a distinction between alllight entering or exiting an (imaginary) volume.

2.3 Mathematical prerequisites

In the next sections, we will review some mathematical concepts such as the innerproduct and orthogonality. Additionally, we will discuss mathematical tools such asthe orthogonal projection of a function on a subspace.

2.3.1 Inner Product and Orthogonality

The functions used in this thesis are elements of a Hilbert space (i.e. a complete innerproduct space). The inner product of the functions f (x) and g(x) in one variable xover the interval [a,b] is defined as:

〈 f |g〉=

Z b

af (x) g(x) dx. (2.4)

Page 29: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.3. MATHEMATICAL PREREQUISITES 17

For functions in more than one variable, a multiple integral is used to define theinner product.

The two functions f (x) and g(x) are said to be orthogonal to each other if theirinner product is equal to 〈 f |g〉= 0.

2.3.2 Orthogonal Projection of a Function on a SubspaceWe can select a set of n linearly independent functions B = b1(x), b2(x), ..., bn(x) inthe function space H. These functions will span a subspace V of H and can thereforebe called a basis of V . The functions bi are also called basis functions of V .

A set of functions B = b1(x), b2(x), ..., bn(x) is called a bi-orthogonal basis ofbasis B if 〈bi|b j〉= δi j, with δi j Kronecker’s delta. This basis B is also called the dualbasis of B.

A basis B for which holds that the inner product with itself is equal to δi j, iscalled an orthonormal basis. Therefore, it is clear that the bi-orthogonal basis of anorthonormal basis B is this basis itself (i.e. B = B). We will often use an orthonormalbasis, as it makes computations often more stable and practical.

Using the functions bi of a basis of subspace V , the function f (x) in space H, canbe approximated as a weighted sum of the basis functions bi:

f (x) ≈n

∑i=1

wi bi(x). (2.5)

The weights wi can be found by orthogonal projecting the function f (x) onto the basisB, i.e. calculating the inner product of the dual basis function bi with the functionf (x):

〈bi| f 〉= wi. (2.6)

Note that equation 2.5 approximates the function f (x)(∈ H) in subspace V . In therest of this thesis, we will use the symbol ’≈’ to denote this approximation.

2.3.3 Calculating the Inner Product of Two functionsIf B is a set of n functions B = b1(x), b2(x), ...,bn(x) and B′ is a set of m functionsB′ = b′1(x), b′2(x), ...,b′m(x), the inner product of the sets B and B′ is written as[〈B|B′〉], where [〈B|B′〉] is a matrix with the i jth element corresponding to the innerproduct of functions bi(x) and b′j(x) and thus to 〈bi|b′j〉. The inner product of the dualbasis functions bi with the basis functions b j is the identity matrix: [〈B|B〉] = In.

Given two functions f (x) and g(x) and two sets of basis functions B = b1, b2,...,bn and B′ = b′1, b′2, ...,b

′m, we can project f (x) onto the basis functions B and

g(x) onto functions B′:

wi = 〈bi| f 〉w′i = 〈b′i|g〉.

Page 30: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

18 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

The functions f (x) and g(x) can then be approximated using these sets of basis func-tions as:

f (x) ≈n

∑i=1

wi bi(x)

g(x)≈m

∑i=1

w′i b′i(x).

With these expressions, the inner product of the approximations of the functionf (x) and the function g(x) can be computed as:

〈 f |g〉 ≈ 〈(

n

∑i=1

wi bi(x)

)|(

m

∑j=1

w′j b′j(x)

)〉

≈n

∑i=1

m

∑j=1

wi w′j 〈bi|b′j〉,

or

〈 f |g〉 ≈ w [〈B|B′〉] w′T. (2.7)

with w = [w1, w2, ..., wn] and w′ = [w′1, w′2, ..., w′m]. Therefore, to compute the innerproduct of functions f (x) and g(x), the inner product matrix [〈B|B′〉] of the two setsof basis functions B and B′ needs to be computed.

This computation can be very simple if the two functions f (x) and g(x) are bothexpressed using a single set of orthonormal basis functions. In that case, B′ equals Band [〈B|B′〉] = In, resulting in:

〈 f |g〉 ≈ w In w′T

≈n

∑i=1

n

∑j=1

wi δi j w′j

or

〈 f |g〉 ≈n

∑i=1

wi w′i .

For clarity reasons, we will use the long term notation of the inner product (i.e. theright-hand side of equation 2.4) in the remainder of this text. This will allow to viewthe different dimensions of the functions unambiguously.

2.4 Image-based Relighting FrameworkIn this work, we want to relight an object placed on a plane with novel illumination.In order to relight real objects, we introduce an image-based framework for relighting,based on the properties and tools overviewed in the previous section.

Page 31: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.4. IMAGE-BASED RELIGHTING FRAMEWORK 19

2.4.1 Scene SubdivisionSimilarly as in section 2.2.5, we choose a surface, called S, that partitions the spacein which the object is situated into two regions. We make a distinction between theregion of space containing the object, which we will call the object region, and therest of space, the region which we will call the illumination region. We assume theviewer is never in the object region, but always in the illumination region and we willtreat the object region as a black box.

Object Region

The object region of space contains the complete object. As shown in section 2.2.5,the appearance of the object can be represented with a 4D plenoptic function or radiantlight field. We choose the surface S to be a hemisphere centered around the object.The polar coordinate parameterization can be used on a hemisphere and this type ofsurface will be closely related to the way we acquire our data. There are many otheroptions for the surface S:

• Any regular geometric form, such as a cube or dodecahedron, centered aroundthe object can be selected. However, it is hard to find a continuous uniformlydistributed 2D parameterization for these geometric forms.

• The surface of the object is an option as well, similarly as for surface lightfields. This requires to have the geometric description of the object, which isnot always the case. Furthermore, parameterizing free-form geometry is notstraightforward.

• In fact, any non-self-intersecting surface can be used. However, similarly as forthe other options, finding a suitable parameterization can be hard.

With the choice of surface S, the radiant light field of the object can be defined.However, the object is to be visualized with novel incident illumination, hence theradiant light field on the bounding surface S of the object depends on the incidentillumination.

Illumination Region

Treating the object region as a black box implies that the incident illumination mustbe defined in the illumination region (i.e. outside the object region). We can expressthe incident illumination as an incident light field. However, a light field is bound tolie on a surface S′, and this surface must lie in the illumination region. We choosethe surface S′ for the incident light field to be equal to the surface for the radiant lightfield, namely S.

Since the incident illumination is bound to lie in the illumination region, any otheroption for the surface S′ of the incident illumination would enclose a larger volumethan the object region. A light field defined on such a surface can always be resampledto an incident light field on the surface S of the radiant light field, without any changein effect on the object, because we assume the illumination region is vacuum.

Page 32: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

20 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

Figure 2.6: The object with the incident light field (left) and the radiant light field(right) defined on a hemisphere S.

2.4.2 Relighting EquationWith the subdivision of space by surface S, into the object region and the illuminationregion, we can derive the mathematical foundation for relighting. On this surface S,an incident light field, representing the incident illumination and a resulting radiantlight field of the object illuminated with the incident illumination can be defined.

Since the light transport in an environment is linear, the radiant light field Lradiantcan be seen as a sum of two light fields Lemit and Lexitant (equation 2.1):

Lradiant = Lemit + Lexitant ,

where Lemit expresses the radiance emitted by the object (e.g. the light of a candle) andLexitant represents the light field that exits the object region, resulting from incident il-lumination that is transported through the object region. We will neglect the emittancelight field Lemit , since it is independent of the incident illumination and can always beadded to the exitant light field Lexitant . Therefore, we assume the object does not emitlight, without loss of generality.

The exitant light field describes the reflectance of incident illumination on thewhole object region (i.e. the incident light field). Equation 2.2 describes the reflectedradiance in a given scene description as:

L(p→ ω) =

Z

Ωfr(ωin↔ p↔ ω) L(p← ωin) cos θin dωin

or

L(p→ ω) =

Z

Ωf ′r(ωin→ p→ ω) L(p← ωin) dωin, (2.8)

with f ′r(ωin→ p→ω), the function returning the ration of reflected light at point p intodirection ω over light incident from direction ωin, taking the foreshortening term intoaccount. Notice that the function f ′r is no longer reciprocal, due to the incorporationof the foreshortening term.

Equation 2.8 demands knowledge of the scene because it is a recursive equation.This recursion originates from the fact that light can reflect more than once in an en-vironment while function f ′r defines only a single reflection of light at a point on a

Page 33: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.4. IMAGE-BASED RELIGHTING FRAMEWORK 21

surface of the scene. Since we treat the object region as a black box and thus haveno information on the scene, we cannot evaluate this equation. However, the inter-reflections within the object region cannot be neglected, since they are the origin ofbeautiful illumination effects such as indirect illumination, color bleeding and caus-tics, which we want to obtain in the resulting images of the relit object.

Instead of function f ′r , we require a light transport function R that is not limitedto only local illumination effects (i.e. single light reflection) but that holds all globalillumination effects as well. With such a transport function R, knowledge of the sceneis no longer required. However, since this transport function R must describe the lighttransport in the scene instead of a single light reflection, it is more complex than thefunction f ′r . The transport function R must describe the reflected irradiance for agiven point p into a direction ω for incident illumination depending on the directionand the position of the incident illumination.

Suppose we have such a transport function R, denoted as R(pin ← ωin,pout →ωout), describing the reflected radiance from point pout into direction ωout with radi-ance L(pin← ωin) illuminating the environment. As we are only interested in illumi-nating the object with a light field defined on the surface S, we can constrain pin tolie on the surface S only. Similarly, as the user is always outside the object region,we can constrain point pout to lie on the surface S as well. Therefore, the functionR(pin ← ωin,pout → ωout) is an 8D function (2 + 2 + 2 + 2). However, with this8D function, all incident illumination needs to be taken into account, not only thelight incident in a single point, but all light incident on the complete domain of pin.Therefore, the radiance in point pout into direction ωout is:

L(pout→ωout) =Z

S

Z

Ωin

R(pin←ωin,pout→ωout) Lin(pin←ωin) dωin dpin. (2.9)

Notice that the recursion in this equation has disappeared, which makes calculatingL(pout→ ωout) more straightforward.

This 8D function R defines the transport of light through the object region, en-closed by surface S and resembles the 8D light transport function from global illumi-nation literature, called a Global Reflectance Distribution Function or GRDF [57, 30,16] and the dimension of this function R is 1

m2 sr .We call equation 2.9 the relighting equation. This equation allows to visualize an

object from an arbitrary position, illuminated with any incident light field, once thefunction R is known.

2.4.3 Reflectance FieldIn the relighting equation 2.9, we call R the reflectance field, as defined by [12]. Thisreflectance field R is an 8D function and represents a 4D exitant light field of theobject dependent on the 4D incident light field illuminating the object.

Since we treat the object region as a black box, obtaining the reflectance fieldwould require us to measure all values of a continuous 8D function. Even at a verysparse discrete sampling of this function, an enormous amount of data measurements

Page 34: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

22 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

would be required: for all points pin in all directions ωin, we need to measure the radi-ance L(pin← ωin) reflected from each point pout in each direction ωout . In this work,we reduce the 8D reflectance field to a 6D function by selecting a fixed viewpoint tothe object.

For any light field, the viewpoint can be fixed, reducing the 4D function definedon a surface to a 2D function defined on the image plane of the view point. A similardimensional reduction can be obtained for the full 8D reflectance field R. By fixingthe viewpoint to the object, the 4D exitant light field reduces to a 2D function on animage plane. However, this 2D function is still dependent on the 4D incident lightfield, yielding a 6D function. We call this 6D function the restricted reflectance field(figure 2.7).

We denote the image plane of the viewpoint I. A point on I can be representedwith two parameters (u,v) and is called I(u,v). Call direction ωu,v the directionfrom the point I(u,v) to the fixed viewpoint. If p is the first hit point of the rayr(viewpoint,−ωu,v), then I(u,v) corresponds to L(p→ ωu,v) and thus, the restrictedreflectance field for a fixed viewpoint with image plane I is the 6D function R(pin←ωin,viewpoint→ ωu,v) or equivalently R(pin← ωin,I(u,v)).

I

uv

Figure 2.7: The restricted reflectance field R(pin←ωin,I(u,v)) is a 6D function defin-ing the radiance in a point I(u,v) into the direction of the viewpoint when the objectis illuminated with radiance L(pin← ωin).

An image plane is often represented as a grid of finite resolution of pixels (pictureelements). We denote a pixel (x,y) on image plane I as Ix,y, with x,y a grid position.For pixel Ix,y, we define the reflectance function Rx,y as:

Rx,y(pin← ωin) =Z

[x,x+1)

Z

[y,y+1)R(pin← ωin,I(u,v)) du dv. (2.10)

This 4D function describes the irradiance on pixel Ix,y, when the object is illu-minated with radiance L(pin ← ωin) with pin ∈ S and ωin ∈ Ω. The relighting equa-tion 2.9 can be rewritten for a pixel:

E(x,y) =Z

S

Z

ΩRx,y(pin← ωin) Lin(pin← ωin) dωin dpin, (2.11)

with E(x,y) the irradiance at pixel Ix,y. Note that the incident radiance of a pixel isexpressed as irradiance, since a pixel covers a small solid angle.

Page 35: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.4. IMAGE-BASED RELIGHTING FRAMEWORK 23

2.4.4 General Image-based Relighting MethodUsing the relighting equation and the restricted reflectance field, we develop a generalrelighting technique. For this technique, a viewpoint to the object is chosen and fixed.The general relighting method then consists of three stages.

1. Data about the object is acquired using an image-based method (section 2.4.4.1).

2. The acquired data is processed resulting in a reflectance function Rx,y for eachpixel (x,y) (section 2.4.4.2).

3. With the acquired reflectance functions, the object can be relit with novel inci-dent illumination: the relighting equation for a pixel can then be evaluated forevery pixel in the image (section 2.4.4.3).

2.4.4.1 Data Acquisition

Although the 8D reflectance field is reduced to a 6D function, this 6D function stillneeds to be measured. This can be achieved with an image-based technique. By optingfor an image-based technique, the difficulty of finding the light transport in the objectregion is avoided, since nature does all that complex work for us.

By illuminating the object with radiance L(p← ω) and recording a photographfrom the fixed viewpoint, we obtain a sample of Rx,y(p←ω) for the reflectance func-tion for each pixel Ix,y. Recording a single photograph thus allows to take a sample ofall reflectance functions in parallel.

The technique to acquire the 6D restricted reflectance field based on this reason-ing, would still require to take an infinite amount of photographs of the object, eachtime illuminated with an incident light field with radiance L(p← ω) with ω in thecontinuous domain Ω and p defined in domain S. This would be an impractical ap-proach.

In our framework, we choose to select a finite set of N incident light fields, whichwe call illumination conditions C1,C2, ...,CN. For each of these illumination con-ditions Ci, an image Imgi is taken of the object illuminated with that condition. Theresulting set of images Img1,Img2, ...,ImgN is called the set of basis images.

2.4.4.2 Reflectance Function Reconstruction

From this set of basis images, the reflectance functions Rx,y will be reconstructed. Byreconstructing Rx,y for each pixel (x,y), the complete 6D restricted reflectance fieldcan be reconstructed. Note that the reconstructed reflectance functions may be anapproximation of the rue reflectance function: the reconstructed reflectance functionmay differ from the true reflectance function since the true reflectance function isunknown and needs to be reconstructed from a discrete set of measurments. In thistext, we will denote the reconstructed reflectance function as R. Keep in mind thatthis is the approximation of the true but unknown reflectance function.

The selection of illumination conditions C1,C2, ...,CN and the reconstructionof the reflectance functions based on the resulting basis images are what makes the

Page 36: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

24 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

distinction between different image-based relighting techniques. Each technique willapproximate the true reflectance function differently. In the following chapters, wewill introduce several sets of illumination conditions, each with its own assumptionsand properties, and discuss several reconstruction techniques for the reflectance func-tions.

2.4.4.3 Relighting

Once the reflectance functions for all pixels on the image plane are obtained, the objectcan be easily relit. To create an image of the object illuminated with a desired incidentillumination Lin, equation 2.11 needs to be evaluated for each pixel Ix,y.

The time required to compute this inner product for a single pixel Ix,y would beproportional to the number of samples in Lin(ω→ p). This computational effort islarge for densely sampled incident illumination and can be impractical for real-timeor interactive applications.

2.4.5 Fast Relighting

As discussed in section 2.3.2, the function Lin(p←ω) can be expressed using a set ofN basis functions BL

i (p← ω) while the function Rx,y(p← ω) can be expressed usinga set of M basis functions BR

i (p← ω):

Lin(p← ω)≈N

∑i=1

liBLi (p← ω)

Rx,y(p← ω)≈M

∑i=1

rx,yi BR

i (p← ω),

with li the projection of Lin(p←ω) onto the dual basis functions BLi (p←ω) and rx,y

i

the projection of Rx,y(p← ω) onto the dual basis functions BRi (p← ω) respectively.

We denote with l = [l1, l2, ..., lN ] the vector of length N of the coefficients li andwith rx,y = [rx,y

1 ,rx,y2 , ...,rx,y

M ] the vector of length M with coefficients rx,yi .

Note, that Lin(p← ω) and Rx,y(p← ω) have different dimensions: Lin(p← ω)is expressed in radiance values on surface S, while Rx,y(p← ω) expresses the ratio ofirradiance on pixel (x,y) resulting from incident radiance on S. li and rx,y

i are scalarsand thus have no dimensions. Therefore, the basis functions BL

i (p← ω) express ra-diance values, while BR

i (p ← ω) is expressed in the ratio of irradiance on a pixelresulting from incident radiance on S.

Relighting Equation as a Sum of Products

To compute the irradiance at a pixel, equation 2.11 needs to be evaluated. A significantspeedup for this computation can be attained by expressing Lin(p←ω) and Rx,y(p←ω) with basis functions BL and BR: Once functions are expressed with sets of basis

Page 37: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 25

functions, the inner product of these functions can be written as a vector-matrix-vectormultiplication, as discussed in section 2.3.3.

Since equation 2.11 is the inner product of function Rx,y and function Lin, andthese functions are approximated using sets of basis functions BL and BR, this equa-tion can be rewritten as a vector-matrix-vector multiplication, similar as equation 2.7:

E(x,y)≈ rx,y [〈BR|BL〉] lT, (2.12)

with [〈BR|BL〉] the inner product matrix of the sets of basis functions BL and BR.To compute a complete image of a relit image, the vector l (i.e. projection of Lin

onto the basis functions BL) needs to be computed only once.If only a single set of basis functions BR is used for all pixels and BL is known, the

inner product of basis functions BR and BL needs to be computed only once and themultiplication of [〈BR|BL〉] with rx,y can also be precomputed. In this case, the relight-ing is merely a vector-vector multplication of the precomputed vector rx,y [〈BR|BL〉]with lT for each pixel.

As noted in section 2.3.3, equation 2.12 can be simplified further, if the innerproduct of these sets of basis functions BL and BR is equal to the unity matrix IM :

E(x,y)≈ rx,y lT

or

E(x,y)≈M

∑i=1

rx,yi li. (2.13)

The inner product [〈BR|BL〉] of the sets of basis functions BL and BR is only theunity matrix IM if both sets consist of M basis functions and set BL is the dual of setBR or the other way around.

If the basis functions BLi and BR

i , for any i, have the same shape and differ only indimensions, then the set BR (and BR) must be an orthonormal set of basis functions.Because the two sets of basis functions have the same shapes, we call the sets a com-mon set of basis functions. In this case, the incident illumination needs to be projectedonly once on the basis functions and the M resulting li can be re-used for each pixel.

Although the approach with a common set of basis functions is often taken, thebasis functions do not always span the entire functions space on and thus, the functionsare only an approximation of the true function. Therefore, the inner product of thetwo approximated functions contains an error as well. The choice for the set of basisfunctions is thus not straightforward and must be made with caution.

2.5 Practical Image-based RelightingIn the previous section we derived a framework with a general method to relight realobjects. From this framework we learned that in order to capture the appearance of anobject independently of the illumination (i.e. the reflectance functions), photographsneed to be recorded to measure irradiance values. To relight this object, novel incident

Page 38: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

26 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

illumination needs to be provided with which the object is to be illuminated. Thisrequires us to be able to capture incident illumination and to calibrate the camera. Wediscuss these issues in this section. To avoid misunderstanding when talking aboutdirections, we first describe some different parameterization schemes.

2.5.1 Parameterization Schemes

In this work we will use directions to define radiance values, light source positions,and many more. In section 2.2, we defined Ω as the set of all directions and L(p→ω) as the radiance in point p into direction ω. However, we defined directions onlyintuitively. First we will discuss ways to define points in space. Then, with this inmind, we can easily define directions.

Points in space: By choosing an origin o in space with an axis system XYZ, a pointp can be expressed with Cartesian coordinates as a triplet (x,y,z), using the three axesX,Y and Z, and with x the distance of the point to the origin o along the X axis and yand z defined similarly (Figure 2.8 left). A point can also be expressed in this spaceby using polar coordinates (φ,θ,ρ). In this triplet, φ defines the azimuth angle, whichis the angle between the projected line op onto the plane XY and the axis X. θ definesthe tilt or elevation angle, which is the angle between the line op and the Z axis. Theangles are expressed in radians. The last coordinate, ρ, is the distance of the point tothe origin (figure 2.8 right). These are the two most common coordinate systems.

.

x

Y

X

zy

φ

θ

ρ

ppZ Z

X

Y

o o

Figure 2.8: A point expressed in two different coordinate systems. On the left, theCartesian coordinate system, on the right, the polar coordinate system.

Directions as points: With the ability to define points in space, we can easily de-fine directions. A direction is a vector, which can be defined by a point relative to areference point. With a chosen origin of the space as reference point, a direction ωcan be defined by choosing a point. The length of the vector, which is the distance ofthe point to the reference point, is irrelevant when considering directions and thus, avector and the corresponding normalized vector represent the same direction. There-

Page 39: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 27

fore, a direction corresponds uniquely to a point on a sphere with unit radius centeredaround the reference point.

Parameterizing directions: Since a direction maps uniquely to a point on a sphere,a direction can be parameterized using parameterization schemes for a sphere. How-ever, parameterizing the surface of a sphere with two uniformly distributed parametersu and v is not straightforward. There are several different parameterization schemesfor the surface of a sphere, each with advantages and disadvantages. In the nextparagraphs, we will discuss the most common parameterization schemes (u,v) with0≤ u,v≤ 1, of points on a sphere.

2.5.1.1 Latitude-Longitude Parameterization

In computer graphics, the most common parameterization scheme for directions orpoints on a sphere is the latitude-longitude parameterization, based on the polar co-ordinates (φ,θ,ρ) of the point. When working on a sphere centered around the origin,the ρ-parameter of any point is always equal to the radius of the sphere (= 1) and thus,directions can be represented using two parameters (φ,θ). The resulting parameteri-zation is:

u =φ

v =θπ.

When only considering directions on a hemisphere, θ is restricted to lie in theinterval [0,π/2) and the parameterization becomes:

u =φ

v =θ

π/2.

An advantage of this parameterization is the uniform distribution for the tilt angleθ. Equal arc lengths of θ correspond to equal interval lengths in the parameterization,as can be seen in figure 2.9 left. However, there is a discontinuity for the azimuthalangle φ. Furthermore the pole of the hemisphere is smeared out to the top line in theparameterization. This is because texels in an environment map do not all representequal sized solid angles.

2.5.1.2 Projected Disk Parameterization

Another parameterization scheme for points on a hemisphere (or directions) is theprojected disk parameterization also referred to as Nusselt embedding. A directiondefined by a point on a hemisphere around the origin with radius 1, expressed with

Page 40: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

28 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

Figure 2.9: The latitude-longitude parameterization. On the left, the uniform distribu-tion of the tilt angle is demonstrated on a red line. Equal arc lengths of θ correspondto equal interval lengths in the parameterization. On the right the upper hemisphere ofa world map is depicted in the latitude-longitude parameterization. The correspondingred parameterization line of the left diagram is shown for a fixed azimuth angle. Thenorth pole is clearly smeared out over the top line and there is a discontinuity alongthe azimuth angle.

Cartesian coordinates (x,y,z) with a positive z-value, can be expressed with (x,y),since

x2 + y2 + z2 = 1 and thus z =√

1−x2−y2.

The resulting parameterization is the mapping:

u =x + 1

2

v =y + 1

2.

Figure 2.10: The projected disk parameterization. On the left the non-uniformly dis-tributed tilt angle is demonstrated on the red parameterization line. On the right theupper hemisphere of a world map is depicted in the projected disk parameterizationwith the corresponding red parameterization line of the left diagram, for a fixed az-imuth angle. The north pole is a single point and fixing the tilt angle results in acontinuous circle of values for varying azimuth angles.

This parameterization corresponds to the orthogonal projection along the Z axisof the hemisphere onto the plane XY and the rescaling of u and v to interval [0,1],

Page 41: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 29

as can be seen in figure 2.10. The major advantage of this parameterization is thecontinuity for the azimuthal angle φ and the pole being represented as a single pointin the parameterization map. However, the tilt angle θ is non-uniformly distributed(figure 2.10 left), equal arc lengths do not result in equal interval lengths in the param-eterization. Note that this parameterization is only meaningful for couples of values(x,y) with x2 + y2 ≤ 1 and thus the parameterization domain is a disk.

2.5.1.3 Paraboloid Parameterization

Instead of using only the x and y components of the Cartesian coordinates, a more uni-formly distributed parameterization for a hemisphere can be created, called paraboloidparameterization [25], using the following mapping:

u =x

1+z + 12

v =

y1+z + 1

2.

This corresponds to a point projection from the origin of space of the hemisphereonto a parabolic surface (blue in figure 2.11), followed by an orthogonal projection ofthe paraboloid onto the plane XY. The advantages of the projected disk parameteriza-tion are maintained and a more uniform distribution for θ is achieved compared to thedisk parameterization (figure 2.11).

Figure 2.11: The paraboloid parameterization. The nearly uniform distribution for thetilt angle is demonstrated on the left on a red parameterization line. Again, the upperhemisphere of the world map is displayed using this parameterization on the right.The corresponding red parameterization line for a fixed azimuth angle is displayed aswell. The more uniform parameterization can clearly be seen in comparison with theleft of figure 2.10.

2.5.1.4 Concentric Map Parameterization

A disadvantage of the projected disk parameterization and the paraboloid parameter-ization is the loss of data space in a texture representation. A texture is usually a

Page 42: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

30 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

square data space and the parameterization domains are a disk. This results in the useof only π/4 of the texture space. This disadvantage can be overcome using concentricmaps [51]. A concentric map transforms data defined on a disk into data defined on asquare while maintaining region continuity. Figure 2.12 demonstrates the concentricmap parameterization.

Figure 2.12: The concentric map parameterization. This parameterization converts apoint on a disk to a point on a square, while maintaining area continuity, by stretchingthe data on a disk to the corners of a unit square. On the right the concentric map offigure 2.10 is displayed.

2.5.2 Capturing Incident IlluminationIn order to relight an object, a representation of the object and the incident illumina-tion to be applied to the object are needed. In section 2.4, functions were derived torepresent the object with a 6D restricted reflectance field as well as to create images ofthe object illuminated with an incident light field. In this section we discuss what theincident illumination is, overview the most common representations and survey sometechniques to acquire the incident illumination from real world environments.

2.5.2.1 Incident Illumination

The incident illumination of an environment is the illumination entering this environ-ment from all positions and directions. As an example, the incident illumination fora room can be defined by all illumination entering that room through windows anddoors. However, light sources in the room are not considered to be incident illumina-tion to that room. Another example is the incident illumination to the viewpoint of anobserver. In this case, the incident illumination is all illumination hitting the viewpointof the observer.

There are two common representations for incident illumination. In the most gen-eral case, incident illumination can be represented by an incident light field definedon a surface, as discussed in section 2.2.5. The observer is always positioned in thevolume enclosed by that surface. If we recapitulate the example of the room, this vol-ume is the room itself and all windows, doors and walls of the room form the surfaceon which the incident light field is defined (with no illumination entering the parts of

Page 43: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 31

surface that represent the walls). The illumination changes depending on looking upor down through a window (angular change). The illumination also changes depend-ing on the position, even when maintaining the view direction constant. E.g. lookingthrough a window and stepping a meter to a side (positional change). Therefore, theincident illumination expressed with an incident light field may chance angularly andpositionally and is a function of both variables Lin(ωin→ pin).

For a single viewpoint v, the perceived illumination can change depending onlooking up or down and left or right (angular change). Since the viewer is fixed tothat single viewpoint v, the incident illumination is independent of the position andthus, there is no positional change. Therefore, another representation for incidentillumination is often the light incident in a point as opposed to a volume. This il-lumination is represented in an illumination map or more commonly known as anenvironment map [6]. In an environment map, the observer is positioned at the sin-gle point v and only the illumination incident from different directions at that pointis considered. Therefore, an environment map is a function of the direction only:Lin(ωin→ v) = Lin(ωin). The incident illumination for a small region in space aroundthe fixed point v can also be represented with an environmnent map under the assump-tion that the incident illumination in all these points is identical: for points v′ close topoint v (i.e. |v−v′|< ε) we assume Lin(ωin→ v′) = Lin(ωin→ v). This assumptionof positional independence in illumination only holds when radiance originates frominfinitely far away and breaks down if the environment closes in on the viewpoint.However, we can tolerate the introduced error if the incident radiance originates froma large distance to the viewer (e.g. in the middle of a large room).

In figure 2.13, an iconic representation of an environment is displayed, with onthe left, the resulting incident light field and on the right the environment map. Ifthe volume (displayed in blue) of the incident light field collapses to a single point,then only angular variation in the incident illumination is captured, and this degradedincident light field can be represented with an environment map (right).

Figure 2.13: Incident illumination can be represented by an incident light field (left)or with an environment map (right).

Page 44: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

32 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

2.5.2.2 Capturing Environment Maps

To capture an environment map for a particular point, the incident illumination fora single point needs to be acquired. We survey the two most common techniquesto capture an environment map in a real world environment: by means of a lightprobe or the use of a fish eye lens. There exist other environment map acquisitiontechniques, such as especially designed cameras [53], but these are quite expensiveand not commonly available.

Light Probe

A practical way to acquire an environment map of a real environment is the use of alight probe. A light probe is a shiny specular reflective ball. The light probe is posi-tioned at the desired point and photographed using a camera with an orthographic lens.Since an orthographic lens is hard to find, using a large zoom lens and positioning thecamera at a significant distance from the light probe will approximate an orthographiclens well and will result in similar results.

If we assume each pixel in the recorded image of the light probe corresponds witha single direction, this direction can be found easily. The normal N to the surface ofthe ball can be found using the inverse transformation of the projected disk parame-terization. The direction for that pixel is the reflected view direction over that normalN (figure 2.14 left). A photograph of the light probe then results in a set of samples ofLin(ωin) (figure 2.14 right).

Although the acquisition process is easy, using a light probe has some disadvan-tages:

• The camera is reflected in the light probe, thus blocks the incident illuminationfor the directions from where the camera is positioned.

• The use of a light probe does not result in a uniform sampling of directionsof the incident illumination since the normals are distributed in the photographwith the projected disk parameterization. The opposite directions of the cameradirection are poorly sampled in comparison with camera direction.

• All directions sampled at the edge of the image of the light probe representillumination from the same direction. Since the light probe has a small yet non-zero radius, these values may differ, resulting in a singularity in the sampledvalues. These effects are aggravated when using a zoom lens instead of anorthographic lens.

In order to obtain a well sampled environment map, without these problems, twophotographs of the light probe can be taken 90 degrees apart. The samples of bothphotographs can be combined into a single environment map as demonstrated in fig-ure 2.15. All environment maps of real world incident illumination used in our workare acquired with the technique of combining the samples of two images of a lightprobe.

Page 45: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 33

Figure 2.14: Using a light probe, or mirror ball, and a camera with orthographic lens,illumination incident to the center of the light probe from any direction is reflected to-wards the camera (left). On the right, a photograph of a light probe in an environmentis depicted. The resulting environment map represents the incident light in a point inthe middle of one of the salon-rooms of the Arenberg castle in Heverlee, Belgium.

Fish Eye Lens

Another way of acquiring an environment map is by means of a camera with a fish eyelens. Taking two photographs in opposite view direction with a fish eye lens mountedon the camera and combining these two sets of samples results in a single environmentas well. However, there are also some disadvantages with this technique:

• It can be hard to find a fish eye lens and they are often very expensive. The lensalso needs to be calibrated, since radiance values entering at grazing angles willbe recorded as less radiance than the same radiance value entering orthogonalto the lens.

• The two images need to be taken in perfect opposite view directions, otherwiselarge sets of directions will not be sampled. Even if the images are recordedcorrectly, often a seam can be noted where both sets of samples are combined.

These problems are hard to overcome. If only the incident illumination of thedirections on a hemisphere of a point need to be captured, than this technique can bevery practical, since only a single photograph needs to be recorded.

Other Environment Map Acquisition Techniques

Besides the commonly used light probe or fish eye lens, other techniques exist tocapture an environment map. The only requirement is to reflect or refract the lightin a point from all directions in an environment towards a camera. Therefore, othersolutions can be:

• The use of a specular paraboloid surface instead of a sphere [25]. By pho-tographing a paraboloid surface, all illumination in the hemisphere is reflectedtowards the camera. This technique is similar as the use of a fish eye lens andwould require two photographs to obtain an environment map of the full sphere.

Page 46: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

34 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

90

Figure 2.15: A light probe is photographed twice, 90 degrees apart from each other.The resulting images are shown on the left. Using the latitude-longitude parameteri-zation, the singularity and camera can be clearly seen in both environment maps (bluedotted circles). By combining the information of both environment maps (red), a sin-gle environment map can be produced without a singularity or camera, as can be seenin the bottom right.

Page 47: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 35

• Since an eye is very specular and curved, the reflection from an eye of a personcan be use to obtain knowledge from the lighting of environment in which thisperson stands [42]. Although this information will not be a complete environ-ment map, this technique has a lot of potential.

2.5.2.3 Capturing Incident Light Fields

An incident light field describes the illumination incident on a surface S. This illumi-nation varies angularly as well as positionally. We can use the techniques to captureenvironment maps to capture the angularly varying illumination at a single point p. Toacquire the positional variation of the illumination, multiple points p1,p2, ...,pm onthe surface S are selected. For each point pi (with 1 ≤ i ≤m), an environment map iscaptured. The resulting set of environment maps represents the sampled incident lightfield.

Several practical methods have been devised [55, 3] to acquire the incident lightfield on a plane. In these methods, the points p1,p2, ...,pm are positioned on a gridon the plane or define a trajectory on a plane.

One class of methods method is based on the light probe method for environmentmaps [55]. An array of n × n light probes is constructed and a photograph is takenfrom a direction orthogonal to the array. This method samples the incident light fieldwith a single photograph. Hence, only a coarse sampling can be achieved. A lightprobe with fixed camera view can also be mounted on a mobile robotic vehicle. Whilethis vehicle moves, a very dense sampling along the trajectory can be gathered [3].

Another method is based on the fish eye lens technique, where the camera withfish eye lens is mounted on a mechanical gantry and for each selected position pi, thegantry moves the camera to that position [55]. With this technique, very fine sampledincident light fields can be obtained.

Note that these techniques capture only an incident light field on a plane. Tocapture an incident light field on an arbitrary surface, the environment map for eachselected point pi needs to be captured separately due to practical reasons.

2.5.3 Camera Calibration

Camera calibration usually means the estimation of the camera position and the lensdistortion. In our work, only a single viewpoint will be used to acquire the restrictedreflectance field and all images will be taken with same camera. This eliminates theneed for the estimation of the position or lens distortion of the camera.

In this work, camera calibration focuses on the transformation between irradiancevalues captured by the camera and the pixel values in the resulting photograph. Pixelvalues of a digital photograph usually have a limited range of values (e.g. 0 to 255).Irradiance values, on the other hand, can range from zero to an arbitrary large value.Therefore, pixel values have a Low Dynamic Range (LDR) and irradiance values areexpressed in High Dynamic Range (HDR).

A good example to demonstrate the non-trivial transformation from irradiance

Page 48: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

36 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

Figure 2.16: A series of photographs of the light probe in the salon environment as infigure 2.14 with different shutter times (left to right: 1/2000, 1/500, 1/125, 1/30, 1/8,1/2 seconds). When taking a photograph with a shorter shutter time, the pixels rep-resenting the windows are no longer saturated and the trees outside are visible, whilethe rest of the environment becomes too dark to notice anything. With a long shuttertime, the windows become saturated but the paintings, the camera and photographersbecome visible.

(HDR) to pixel values (LDR) is the recording of an image 1 featuring the sun. Ina photograph of the sun all the pixels will have the maximum pixel values, in ourexample 255 because these pixels are saturated. The pixel values are clamped to themaximum pixel value available because the light sensitive sensors in the camera areover exposed. This problem can be easily solved by using a very short shutter time onthe camera and avoiding to over-expose the light sensitive sensors. For a photographof the sun with a very short shutter time, the sun will be portrayed as a yellow disk,because the light sensitive sensors of the camera are not over-exposed.

However, things may become more complicated. If the photograph portrays thesun as well as a shadowed region, an appropriate shutter time can never be found forthe camera to capture both features. Either the pixels representing the sun are saturatedand the shadowed region is captured fine (figure 2.16 right), or the sun is representedas a yellow disk (i.e. no saturation) but the pixels for the shadowed region are all setto zero, because of under-exposure (figure 2.16 left). The transformation from HDRto LDR is thus not only a scaling function because of threshold cut-off. We want toknow the irradiance values, the scaled pixel values. The origin of the problem is thenon-linear transformation from HDR to LDR by a common camera and the limitedrange of pixel values. Each camera has a response curve describing this non-lineartransformation.

Debevec and Malik [13] proposed a technique to recover this response curve func-tion by using a series of P photographs of the same environment with different knownexposure times (∆t1, ∆t2, ..., ∆tP) from the same viewpoint. An example of such aseries of photographs can be seen in figure 2.16. Since we used this technique onmany occasions in our work, we overview this technique here briefly and demonstrateit with the data of figure 2.16. As in [13], the value of the ith pixel of the photographwith exposure time ∆t j is called Zi, j. These pixel values Zi, j are integer values and wecall Zmin the minimum pixel value and Zmax the maximum pixel value over all pixelsand photographs. If pixel values can range between 0 and 255, as in our example, Zminwill be positive and very small (in our case value 4), while Zmax will be equal to 255

1To acquire an environment map in HDR, see also [58]

Page 49: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 37

due to saturation. We know the pixel value is the result of the response curve functionfor the product of the irradiance of the pixel with the exposure time of the camera or:

Zi, j = f (Ei ∆t j),

with Ei the irradiance on the ith pixel, and f () the response curve function which isreasonably assumed to be monotonic and thus, f−1() exists. The irradiance values Eifor each pixel need to be found, based on the known exposure times ∆t j and the pixelvalues Zi, j. In order to find these values, [13] gives the following derivation:

Zi, j = f (Ei ∆t j)

f−1(Zi, j) = Ei ∆t j

ln( f−1(Zi, j) ) = ln( Ei ∆t j )

ln( f−1(Zi, j) ) = ln( Ei ) + ln( ∆t j ),

or by substituting ln( f−1()) by function g():

g(Zi, j) = ln( Ei ) + ln( ∆t j ),

or

ln( Ei ) = g(Zi, j)− ln( ∆t j ). (2.14)

Since, Ei and the function g() are unknown, the values Ei will only be knownup to a scale factor a with this equation. This can be easily demonstrated with thefollowing example: suppose, we know the value Ei and replace it with value aEi,which corresponds to replacing ln( Ei ) with ln( Ei )+ ln( a ), then the equation 2.14still holds if g() becomes g() + ln( a ), which is possible, since g() is also unknown.

Using equation 2.14, the pixel values of a single pixel in the different photographscan be plotted (figure 2.17). Call z′i the pixel value for that pixel in the ith photographand the accompanying irradiance value is called ln( E ′i ). The horizontal axis repre-sents the pixel values (ranging between 0 and 255), the vertical axis represents theln( Ei ). Since the shutter time was increased with a constant number of f-stops be-tween two photographs, the values ln( E ′i ) of the pixel differ a constant factor betweensubsequent photographs (equation 2.14). Since this is known up to a scale factor, arandom value for ln( E ′1 ) can be chosen and the subsequent values ln( E ′i ) are thendefined as (ln( E ′1 ) + iC), with C a constant determined by the number of f-stopsbetween two photographs.

The values Ei need to be recovered and thus also the function g. However, notthe complete function g needs to be found but only g(z) for z a pixel value. Since weknow all pixel values are integer values in the interval [ Zmin,Zmax ], g(z) is requiredfor only (Zmax − Zmin + 1) values.

Suppose we have N pixel values, then the function values g(z) can be found byminimizing the error in equation 2.14 for all pixels in all photographs with a minimalleast squares optimization:

Page 50: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

38 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

C

C

C

C

C

0 50 100 150 200 250 Pixel values

ln(i

rrad

ianc

eva

lues

)

ln( E ′1)

Figure 2.17: The six pixel values of a single pixel in the six different photographs offigure 2.16.

O =N

∑i=1

P

∑j=1

[g(Zi, j) − ln( ∆t j ) − ln( Ei ) ]2

However, the function values g(z) may not be optimized for each value separately.A relation holds for consecutive values g(z−1), g(z) and g(z+1): g(z−1) − 2g(z) +g(z+1) should be small. This relation originates from the fact that these values shouldlie on a smooth function g and have a the second derivative g′′(z) (equal to g(z−1) −2g(z) + g(z + 1) in case of a discrete problem) as small as possible. Adding this factto the optimization equation leads to:

O =N

∑i=1

P

∑j=1

[g(Zi, j) − ln( ∆t j ) − ln( Ei ) ]2

+ λZmax−1

∑z=Zmin+1

[g(z−1) − 2g(z) + g(z + 1)]2 ,

with λ a weight for the smoothness of the resulting function g, relative to the datafitting. In [13], an extra weighting term is introduced in the optimization to favorvalues in the middle of the interval [ Zmin,Zmax ].

From an intuitive point of view, this optimization shifts all the data values perpixel up and down, as depicted in figure 2.17, for all pixels until all data points line upon the best curve through all values possible. We demonstrate this with an example.The pixel values of three different pixels are plotted in figure 2.18. The values of thedifferent pixels are depicted as circles, crosses and squares. Since these values are

Page 51: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

2.5. PRACTICAL IMAGE-BASED RELIGHTING 39

known up to a scale factor, we plotted them with the same relative irradiance values.The optimization will shift all the values of the different pixels vertically until theyline up in a smooth curve (figure 2.19). The result is the response curve of the camera.Since the values are known up to a scale factor, only the relative position is important(e.g. pixel x is double the irradiance value as pixel y) and the exact vertical positionof the curve is not known.

0 50 100 150 200 250 Pixel values

ln(i

rrad

ianc

eva

lues

)

Figure 2.18: The six pixel values for three different pixels from figure 2.16.

With this technique, the Ei values can be known relative to each other and theresponse curve function g can be found by fitting a function through the values g(z).Once the function g is obtained for the camera, the pixel values of a single photographcan be transformed into irradiance values using equation 2.14. However, only thepixel values in the interval [ Zmin,Zmax ] will be transformed correctly. Therefore, thedynamic range of the photographed scene may not exceed the available dynamic rangeof the camera. The dynamic range of a standard digital camera is about 1 : 500. Ifthge dynamic range of the environment is larger, saturated pixels will be transformedin false irradiance values. Choosing the shutter time adaptively for each scene, mayhelp to some extent, but for really high dynamic range scenes, such as environmentmaps, the optimization must be used.

In this work, we only use HDR images in our computation, no matter if theyare reconstructed from a series of photographs or from a single photograph using theresponse curve. However, images printed in this text are always transformed back intoLDR images, due to the lack of HDR ink. To make a distinction with HDR imagesand LDR images, a recorded picture with a camera will be called a photograph.

Page 52: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

40 CHAPTER 2. GENERAL IMAGE-BASED RELIGHTING

0 50 100 150 200 250 Pixel values

ln(i

rrad

ianc

eva

lues

)

Figure 2.19: The recovered response curve of the camera using the three pixels offigure 2.18. By shifting all the values of the pixels up and/or down, the response curveof the camera can be recovered.

2.6 ConclusionIn this chapter, we presented a framework for image-based relighting. We derived therelighting equation, enabling to relight a real object, based on the appearance of theobject and the incident illumination. In this context, we provided a general image-based method to acquire the appearance of an object, independently of the incidentillumination (i.e. the reflectance field). This general method will be instantiated andimplemented in different ways in the next chapters.

Additionally, parameterization schemes for directions, several incident illumina-tion acquisition methods and camera calibration techniques were surveyed.

Page 53: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 3

Relighting with 2D Illumination

In the previous chapter, a general framework was developed for relighting real objectswith an arbitrary incident light field. Instead of using an incident light field as novelillumination, objects can also be relit with only directionally varying incident illumi-nation. This is often done because it is easier to accomplish, regarding data acquisitionand data processing, than using incident light fields. By restricting the incident illumi-nation to depend only on the direction, the illumination can be represented with a 2Dfunction, hence the term relighting with 2D illumination. In this chapter we discussseveral techniques to relight objects with restricted 2D incident illumination.

In section 3.1, we approach relighting with directionally dependent illuminationfrom a mathematical point of view, and simplify the general relighting frameworkdescribed in chapter 2. With this background, we proceed in section 3.2 with anoverview of the most common method to relight objects with directionally varyingillumination, represented as an environment map. In this section we also refine thisstandard technique to a more practical relighting method. Similar relighting methodswith directionally varying incident illumination are surveyed briefly and are fit in ourframework as well. Section 3.3 discusses other published relighting methods withdirectionally varying incident illumination.

Restricting the incident illumination to only directionally varying incident illumi-nation is not the only option to reduce a (4D) incident light field to 2D illumination.One could also restrict the incident illumination to only spatially varying incident il-lumination. In section 3.4, we revisit our general relighting framework again for thiskind of relighting and demonstrate that published relighting techniques with spatiallydependent incident illumination fit in our framework.

Working with 2D illumination requires to make some assumptions. In the lastsection (section 3.5), we survey these required assumptions and the consequences forthe relighting.

41

Page 54: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

42 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

3.1 IntroductionTechniques to relight with directionally varying incident illumination usually havethe illumination expressed in an environment map (2.5.2.1). An environment consti-tutes of the illumination incident at a single point and the assumption is made thatthe incident illumination at all other positions in space is identical to the measuredillumination at that single point. For relighting, this corresponds to assuming that theincident illumination at any point on the surface S enclosing the object, is identicalto the incident illumination at the point where the environment map is captured. Thisassumption holds for environments that are very large in comparison to the size of theobject, but breaks down if the environment, in which the environment map is taken, isclose to the object.

Therefore, an environment map can be seen as a function defining the directionalillumination for all directions of an environment. With this in mind, relighting an ob-ject with an environment map allows us to illuminate the object with any combinationof directional light sources and thus with directionally varying illumination.

Since only directional illumination independent of the position, is used, the objectis either completely illuminated from a direction or is not illuminated from a direction.Therefore, local illumination effects such as a spotlight cannot be achieved when anobject is relit with an environment map.

3.1.1 Relighting Equation RevisitedThe relighting equation, developed in chapter 2, can be simplified when objects areto be relit with illumination dependent only on the direction. Therefore, we revisitour relighting framework and start by recapitulating equation 2.11 (on page 22). Thisequation states the general relighting equation for a pixel:

E(x,y) =Z

Ω

Z

SRx,y(p← ω) Lin(p← ω) dp dω.

Using only directionally varying incident illumination corresponds with incident illu-mination independent of the position. Therefore, the incident illumination for positionp from direction ω is equal to the radiance from direction ω in a fixed point pfixed:

Lin(p← ω) = Lin(pfixed← ω).

The relighting equation then becomes:

E(x,y) =

Z

Ω

Z

SRx,y(p← ω) Lin(pfixed← ω) dp dω

=

Z

ΩLin(pfixed← ω)

Z

SRx,y(p← ω) dp dω

or

E(x,y) =Z

ΩLin(pfixed← ω) Rx,y

dir(ω) dω.

Page 55: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.1. INTRODUCTION 43

with

Rx,ydir(ω) =

Z

SRx,y(p← ω) dp.

The reflectance function for pixel (x,y) then becomes the 2D function Rx,ydir . In this

case, the restricted reflectance field can be reduced from a 6D function to a 4D re-flectance field, because only the incident direction is of importance and the positionno longer matters.

Similarly as the reflectance function, the incident light field can be written with-out the positional argument pfixed as Lin(ω). With the two dimensionally reducedfunctions Lin and Rx,y

dir the relighting equation for a pixel becomes:

E(x,y) =

Z

ΩRx,y

dir(ω) Lin(ω) dω. (3.1)

The function Rx,ydir(ω) is the 2D reflectance function for pixel (x,y) and Lin(ω) is

the directionally dependent incident illumination. The object can be relit using thisequation once the reflectance functions Rx,y

dir are obtained for each pixel (x,y).In the remainder of this text we will drop the subscript dir from the function

name Rx,ydir(ω) and call the 2D reflectance function Rx,y(ω). Note that the 2D function

Rx,y(ω) and 4D function Rx,y(p←ω) have different dimensions ( 1sr and 1

m2 sr respec-tively). However, from the context it will always be clear if the reflectance functionRx,y is a 4D or a 2D function.

3.1.2 General Image-based Relighting Method RevisitedIn section 2.4.4 a general image-based relighting method was proposed based on threestages: data acquisition, data processing and relighting using the processed data. Wesurvey these three stages for relighting with directionally varying incident illumina-tion:

1. In the general relighting method, the data acquisition method requires to se-lect a set of N illumination conditions C1,C2, ...,CN, which are incident lightfields (i.e. dependent on direction and position). For relighting with direction-ally varying incident illumination, the reflectance functions Rx,y(ω) are onlydependent on the direction ω, and the illumination conditions can be restrictedto varying only in direction as well. Therefore, the illumination conditions arenoted as functions Ci(ω) and can be represented with environment maps. Abasis image Imgi is then to be recorded of the object illuminated with eachillumination condition Ci separately.

2. The resulting set of basis images Img1,Img2, ...,ImgN, are used in the dataprocessing stage to construct the reflectance functions Rx,y(ω).

3. After the construction of the 2D reflectance functions, the object can be relitwith arbitrary directionally varying illumination, according to equation 3.1.

Page 56: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

44 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Note that the choice of C1(ω),C2(ω), ...,CN(ω) will have a great influence onthe construction method for the reflectance functions. We distinguish between illu-mination conditions that are pulse functions (section 3.2), and structured illuminationconditions (section 3.3).

3.2 Pulse ApproachThe N illumination conditions Ci can be based on N chosen directions ω1,ω2, ...,ωN.Therefore, they are pulse functions on the Ω domain and correspond with the N direc-tional light sources from the directions ωi:

∀i, 1 ≤ i ≤ N : Ci(ω) =

1 Wm2×sr if ω = ωi

0 Wm2×sr if ω 6= ωi.

(3.2)

Each basis image Imgi, resulting from illuminating the object with condition Ci,holds a single measured sample of each reflectance function. In fact, the pixel (x,y)of basis image Imgi is a measured value of the reflectance function at direction ωiand is equal to Rx,y(ωi). With these N measured values, one for each basis image, thereflectance functions Rx,y(ω) will be constructed on the complete Ω domain.

In section 3.2.1, we instantiate this acquisition technique and obtain the most basicdata acquisition method for relighting with environment maps, called the Light Stage.This data acquisition method and an intuitive method to relight the object with thiscaptured data are discussed. In addition, we demonstrate that our presented relightingframework can also be used to relight the object: we describe a basic reconstructiontechnique for the reflectance functions, using the acquired data, in order to be able torelight the object with the relighting equation. Fitting the relighting method in ourframework allows to express the relighting method mathematically.

We extend this basic data acquisition technique in section 3.2.2, by reducing somelimitations of the Light Stage. The resulting new data acquisition technique, calledFree-form Light Stage, is more practical but requires a more sophisticated reflectancefunction reconstruction method.

3.2.1 Light StageFor the N illumination conditions N directions ωi need to selected. The object is to beilluminated from these directions and for each direction an image is to be recorded.For the most basic data acquisition method, these directions ωi are chosen to be regularsampled on the latitude-longitude parameterization and are denoted as (φi,θi).

Debevec et al. [12] introduced a Light Stage for this data acquisition. A LightStage is a directional light source mounted on a manually operated two-axis rotating

Page 57: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 45

system. One axis allows to rotate in steps around a vertical axis, which correspondswith the azimuthal rotation (i.e. φ-rotation). The other axis allows to tilt the lightsource in steps and change the θ-angle. A camera is placed outside this device andallows to record images of the object positioned in the center of the two-axis system.Note that the light source may block the camera from recording the object. The LightStage allows to record images of the object illuminated from a set of regularly sam-pled directions in the latitude-longitude parameterization (φi,θi), which are the basisimages Imgi. In the proposed technique, the object is a human face and it is illumi-nated from 64 × 32 (= 2048) regularly sampled directions on a sphere (i.e. the stepsize for azimuthal rotation is 64/360 degrees and the step size for tilt rotation is 32/180degrees). The data acquisition duration is about one minute.

An evolution of the Light Stage was proposed in [23], called a Light Stage 2.0,able to acquire a similar data set of an arbitrary object, fully automated and extremelyfast (less than 10 seconds). This Light Stage 2.0 consists of 27 light sources mountedon a semi-circular brace. This brace with the light sources replaces the single lightsource and the axis allowing to tilt that light source of the original Light Stage. Thebrace can still be rotated around the object in 64 steps allowing to illuminate the objectfrom 64 × 27 directions.

In this setup, the light sources are small area light sources and not directional lightsources. However, as long as the object is small compared to the distance to the lights,these light sources are a good approximation for directional light sources. This isdemonstrated in figure 3.1.

Figure 3.1: On the left, a directional light source is illuminating the object. In themiddle a standard light source illuminates the object from a large distance, while onthe right the light source is closer to the object. As can be seen, the larger the distanceof the light source to the object, the more the rays of the light source converge to therays of a directional light source.

Both Light Stage and Light Stage 2.0 are practical devices allowing to acquire Nbasis images Imgi for all directions ω1,ω2, ...,ωN or similarly, (φ1,θ1), (φ2,θ2), ...,(φN ,θN).

We built a similar device as the Light Stage 2.0, but restricted the incident illu-mination to the upper hemisphere only. This is not a severe restriction, since we willplace the object on a ground plate, in order to see the shadows of the object cast on

Page 58: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

46 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

this ground plate. In our setup, the object and camera are placed on a turntable. Asemi-circular brace with 40 light sources is mounted over the turntable. The tilt angleof the light direction is defined by which light source on the brace is used, the azimuthangle can be changed by rotating the turntable. Using 40 light sources and rotatingthe turntable 180 degrees in 32 steps, allows to illuminate the object from 64 × 20(= 1280) regularly sampled directions in the latitude-longitude parameterization. Aschematic drawing and an image of the setup can be seen in figure 3.2, in figure 3.3a subset of 32 × 10 basis images resulting from our version of the Light Stage isdisplayed. Note that in this setup the camera can block a light source.

...

Figure 3.2: On the left, a schematic drawing and on the right a photograph of the setupof our version of a Light Stage 2.0 acquisition device. The scene, (in the photographa set of stones), and the camera are placed on a turntable. A semi-circular brace with40 light sources is mounted over the turntable. By rotating the turntable 180 degreesin 32 steps and switching on one light source at a time, the object can be illuminatedfrom 64 × 20 regularly sampled directions.

Figure 3.3: The toy miniature race car, as depicted in figure 3.2 is used as objectfor acquisition of basis images. A subset (1/4) of all recorded images is depicted andorganized in a latitude-longitude parameterization corresponding with the illuminationdirection used for each image.

Page 59: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 47

Intuitive Approach to Relighting

The aforementioned techniques [12, 23] assume the data acquisition is a very densesampling of the possible directions and this allows to relight the object with an intu-itive approach. If we look at the data acquired with a light stage, relighting with di-rectionally varying incident illumination can be seen as an application of the linearityof light transport. The plenoptic functions for a fixed viewpoint with light originatingfrom a set of directions are recorded in the different basis images. These plenopticfunctions can be linearly combined to mimic the relighting of the object accordingto a given incident illumination. Suppose, for example, this incident illumination tobe red light originating from a direction ωred and yellow light coming from ωyellow.We can relight the object with this illumination by finding the directions ωi and ω jused during the data acquisition, closest to ωred and ωyellow respectively. The basisimage Imgi taken with illumination from ωi is modulated with the red radiance tripletand is called Imgred

i . Similarly, the image Img j taken with illumination from ω j ismodulated with yellow radiance to make up the image Imgyellow

j . The sum of images

Imgredi and Imgyellow

j results in an image of the object illuminated with the desiredillumination.

A linear combination of basis images can of course be calculated for any incidentillumination: the radiance value Lin(ω) of the incident illumination is assigned tothe nearest direction ωi and the basis image Imgi is modulated with this radiancevalue. This is done for all directions in Ω, and all modulated basis images are summedtogether.

Formulated more generally, the object can be relit with the incident illuminationLin(ω), represented as an environment map, by resampling the environment map tothe same resolution as the data acquisition sampling resolution (i.e. N incident illumi-nation values). The recorded basis images Imgi can then be modulated with the cor-responding re-sampled incident illumination (which is irradiance, since pixels cover asolid angle). This corresponds with modulating the basis images with the light origi-nating from all the directions closest to the sampled direction ωi.

Because of the regularly sampled set of directions in latitude-longitude parameter-ization, this resampling is merely a down-sampling of the environment map expressedin this parameterization. We denote this discrete down-sampled incident illuminationas wφi,θi . The irradiance value wφi,θi represents the approximation of the incident illu-mination from direction (φi,θi) for the relit object. Since each irradiance value wφi,θi

corresponds to a single direction (φi,θi), which in its turn corresponds to a basis imageImgi, each irradiance value wφi,θi corresponds uniquely to a single Imgi. We modu-late each Imgi with the value wφi,θi , resulting in Img

wφi,θii and add all these images

together. Thus, to compute the image of the relit object, we create a weighted sum ofthe recorded basis images Imgi with corresponding irradiance values wφi,θi as weights:

Image =N

∑i=1

wφi,θi × Imgi . (3.3)

This can be illustrated with a simple example as shown in figure 3.4. In this

Page 60: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

48 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Figure 3.4: An example of relighting the miniature race car from figure 3.2 with theenvironment map of the salons in the Arenberg castle of Heverlee. The environmentmap is normalized and down-sampled to the data acquisition resolution of the basisimages. These basis images are modulated with the corresponding down-sampledillumination color and added together, resulting in an image of the race car illuminatedwith the illumination of the salons of the Arenberg castle.

Page 61: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 49

example, only 32 × 10 directions are used, and hence only 32 × 10 basis imagesare used to make a clear illustration.

The incident illumination, in this example the illumination in the salons of theArenberg castle in Heverlee Belgium, first needs to be normalized. Due to the pa-rameterization, pixels in an environment map do not always correspond to equal solidangles, as was already mentioned in section 2.5.1. For example, a pixel on the topline of the environment map expressed in the latitude-longitude parameterization cor-responds to a very narrow solid angle, whereas a pixel on the bottom line of this en-vironment map of the hemisphere corresponds to a wider solid angle. In order for theenvironment map to represent the captured illumination, this needs to be taken intoaccount. Since the pixel sizes cannot be altered, the irradiance values of the pixelsneed to be adjusted. This compensation consists of normalizing the irradiance valuesin the environment map according to solid angle of the pixel. The relation betweena pixel and the solid angle of this pixel, is given by the Jacobian of the transforma-tion (i.e. parameterization). For the latitude-longitude parameterization, the Jacobiancorresponds with the function sin(θi) and is independent of φ. Therefore, by modulat-ing all irradiance values with the function sin(θi), the environment map is normalized(first row of figure 3.4).

The resulting environment map is down-sampled to an environment map with res-olution 32× 10 (figure 3.4, second row). Subsequently, the recorded basis images aremodulated with the corresponding illumination from the down-sampled environmentmap (third and forth row of figure 3.4). By adding all these modulated basis imagestogether, an image of the relit object can be created (figure 3.4, last row).

Framework Approach

Although this linearly combining of images is very intuitive, we can also fit this re-lighting technique in our relighting framework. In the framework, the reflectancefunctions for each pixel are reconstructed based on the recorded basis images. Thesefunctions are then used in the relighting equation to create an image of the relit object.

The reflectance functions Rx,y(ω) need to be reconstructed over the complete do-main Ω, using the selected illumination conditions C1(ω),C2(ω), ...,CN(ω), whichare defined by the set of directions ω1, ω2, ..., ωN as in equation 3.2 and the accom-panying basis images Img1, Img2, ..., ImgN.

The directions are regularly sampled in latitude-longitude parameterization, there-fore, each direction ωi can be written as a couple of elements from the sets φ1, φ2,..., φN1 and θ1, θ2, ..., θN2 with N1×N2 = N. This results in a set of N directions(φ1,θ1), (φ1,θ2), ..., (φ1,θN2 ), ...,(φN1 ,θN2 ). Because of this regular sampling inthat parameterization, we can define the constant sampling distance between φi andφi+1 as ∆φ and the constant sampling distance between θi as θi+1 the ∆θ.

To reconstruct the reflectance functions, we create a set of N disjunct partitions Ωiof Ω. The union of all these partitions Ωi covers Ω completely:

[

i

Ωi = Ω and Ωi\

Ω j = /0 if i 6= j ∀ i, j ∈ 1, ...,N. (3.4)

Page 62: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

50 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

a

c

e

b

db

d

e

a

c

Figure 3.5: Five reconstructed reflectance functions from the miniature race car ex-pressed in a latitude-longitude parameterization. From top to bottom: a diffuse surfaceon the ground plate (a), a surface with a long highlight (b), a surface with a soft highlight (c), a surface with complex occlusion (d) and a diffuse surface on the helmet(e).Note the dark spots at the lower right of each reflectance function. this is the cameraoccluding the light source.

We define each Ωi as a subset of Ω:

∀ ω = (φ,θ) ∈ Ω : ω = (φ,θ) ∈ Ωi if

(φi − ∆φ

2

)≤ φ <

(φi + ∆φ

2

)

and

(θi − ∆θ

2

)≤ θ <

(θi + ∆θ

2

)

(3.5)With these subsets Ωi, we can approximate the reflectance functions Rx,y(ω) as:

Rx,y(ω) =

Img1(x,y) if ω ∈ Ω1Img2(x,y) if ω ∈ Ω2

...ImgN(x,y) if ω ∈ ΩN

(3.6)

or written in a more compact way:

Rx,y(ω) = Imgi(x,y) with ω ∈Ωi. (3.7)

Page 63: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 51

These reconstructed reflectance functions are thus constant over each subset Ωiand the value for each subset Ωi is equal to the measured reflectance value Imgi(x,y).Since the union of all Ωi covers Ω completely, the constructed reflectance functionsare defined over the complete domain Ω, yielding a piecewise constant function overΩ. The true reflectance function for each pixel can be approximated with this recon-struction. Five selected reconstructed reflectance functions are depicted in figure 3.5.

Relighting can now be achieved by taking the inner-product of the reflectancefunctions with the incident illumination (relighting equation 3.1). This is illustrated infigure 3.6 for the selected reflectance functions of figure 3.5 and the complete imageis depicted at the bottom of figure 3.6.

=

=

=

=

=

,

,

,

,

,

a

c

e

b

d

b

d

e

a

c

Figure 3.6: The inner-product of the selected reflectance functions and the normalizedenvironment maps is computed, yielding the colors of the pixels in the relit images.At the bottom, the complete image of the relit object is depicted.

The resulting image of the relit object is equal to the image created using theweighted sum of the basis images (section 3.2.1). Both techniques perform exactly

Page 64: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

52 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

the same operations but in a different order: the sum of the pixel values Imgi(x,y),modulated by the down-sampled normalized environment map values is equal to theinner-product of the reconstructed reflectance function Rx,y with the normalized envi-ronment map, due to the reconstruction method.

Limitations of the Pulse Approach

It is important to note that the reflectance functions are reconstructed with a discreteset of measured values of the true reflectance function and that this reflectance func-tion is assumed not to change but to be constant over each region of directions Ωi.However, the real reflectance function may not be constant over an entire region Ωi.Moreover, high frequency features of a reflectance function, such as a highlight, couldbe missed during sampling, resulting in reflectance functions not incorporating theseimportant features. By increasing the number of measured values (i.e. increasingthe number of directions N) this problem can be avoided. However, the reflectancefunctions may contain very high frequency features requiring an infinite amount ofdirections to be sampled, resulting in a reflectance function with a measured value forany direction. This is impractical, hence, this sampling approach only works for lowfrequency reflectance functions, and thus, specular objects can not be relit with thisapproach.

As an example, in figure 3.7 the object is a diffuse white ball and the illuminationconditions are 64×36 = 2304 regularly sampled direction created in a high resolutionLight Stage. The sum of the basis images is shown on the left. On the right thereconstructed reflectance functions for two neighboring pixels are displayed. Thesetwo reflectance functions do not differ much. The camera can be clearly noticed inthe reflectance functions, since each illuminant direction contributes to the reflectancefunction but for some directions the camera blocks the illumination.

In figure 3.8, a chrome sphere is used instead of a diffuse ball. All the 2304different highlights can be seen in the sum of basis images and the camera can berecognized, since each basis image contributes only to a set of pixels in the resultingimage that corresponds with the highlight. The two reflectance functions of the neigh-boring pixels differ signifigantly: the reflectance function on the top right features ahighlight, while at the right bottom, the reflectance function seems to reflect no inci-dent illumination. This is because the sampling resolution, even at 2304 samples, wastoo coarse.

In figure 3.9, a glass ball was the object. Here the incident illumination is reflectedand refracted multiple times, resulting in multiple highlights in the reflectance functionfor some pixels and in no reflected illumination in other reflectance functions.

3.2.2 Free-form Light StageThe Light Stage, its evolution (Light Stage 2.0) and our version of the Light Stage usea gantry to move the light source to known positions. This use of a gantry poses somelimitations on the relighting itself:

• The object needs to be small relative to the distance to the light sources, hence

Page 65: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 53

Figure 3.7: On the left, a diffuse ball illuminated from 64×36 = 2304 directions. Onthe right two reconstructed reflectance functions of neighboring pixels in the middleof the left image. Note that these two reflectance functions are quite similar. Thecamera can clearly be recognized in the reflectance functions.

Figure 3.8: On the left, a chrome ball illuminated from 64×36 = 2304 directions. Thetwo reconstructed reflectance functions are from two pixels next to each other on theleft image. As can be clearly seen not all pixels have reflected incident illuminationtowards the camera for any sampled direction.

Figure 3.9: On the left, a glass ball illuminated from 64×36 = 2304 directions. Theincident illumination from certain directions is reflected and refracted, resulting inmultiple highlights in the accompanying reflectance functions. However, due to thehigh frequency, not all pixels reflect incident illumination towards the camera.

Page 66: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

54 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

the size of the objects is limited. If the light sources can be positioned on ahemisphere with a radius of 1 meter, then the object can only be a few decime-ters in diameter, otherwise the assumption of a directional light source is notmet (see figure 3.1 on page 45).

• Using a gantry requires to place the object into the Light Stage. This is a prob-lem for objects that may not be moved. As an example, an ancient old mosaicor fresco is usually fixed to a floor or wall and cannot be drilled out, simplyto acquire reflectance data. Moving a Light Stage to the object can be veryimpractical, due to the size of the device and the necessary calibration.

• Most importantly, using a gantry limits the sampling resolution of the illumi-nant directions (i.e. the number of possible illuminant directions). In our setup,the brace with 40 light sources allows 20 fixed tilt positions and the turn table isdesigned to rotate over 180 degrees in 32 steps, resulting in 64× 20 light sourcepositions. One can easily use half of the tilt positions or double the azimuthalrotation step size to sample at 64 × 10 directions or 32 × 20 directions respec-tively. However, one cannot ask for sampling 64 × 40 directions or any otherrandom number of directions.

All of these limitations and restrictions can be removed using a free-form LightStage. With a free-form Light Stage, the user acts as gantry. A hand-held light sourceis moved freely around the object, and thus without the use of a gantry. The free-form Light Stage relighting technique works again in three stages, with a very simpledata acquisition, a more sophisticated data processing and a relighting step, similarto using a standard Light Stage.

In the first stage, the basis images need to be acquired. These basis images areagain images of the object, taken from the fixed camera position, whereby the objectis illuminated with a single light source held by the user at freely chosen positions. Theobject is surrounded by four diffuse white spheres, which are also visible in the basisimages. Since the hand-held light source is positioned freely for each basis image, theillumination per basis image is different and unknown.

In the second stage, the acquired images are processed. In each basis image, theilluminant direction needs to be estimated since it is required for relighting, but un-known due to the free-form movement of the user. The parts of the basis imagesshowing the diffuse spheres are used to estimate the illuminant direction. Since dif-fuse spheres and a single light source are used, the illuminant direction can be re-trieved from the shading pattern of the spheres. Using these illuminant directions thereflectance functions of the pixels representing the object, can be constructed. This isnot straightforward, since no regular sampling can be assumed.

In the third and final stage, the object is relit with an environment map. Simi-lar to for the Light Stage, once the reflectance functions are obtained from the dataprocessing stage, the relighting boils down to evaluating equation 3.1.

Page 67: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 55

3.2.2.1 Data Acquisition

White diffuse spheres (we use table tennis balls, spray-painted with white diffusepaint) are placed around the object. These spheres will be used to estimate the lightsource direction, since the hand-held light source will be moved freely but the lightdirection still needs to be known. Therefore, it is important that the spheres are visiblefrom the camera viewpoint.

This setup can be done in as few as a couple of minutes, since it requires only tofix the camera, position the diffuse spheres and the object in the viewing range of thecamera and switch on the hand-held light source. Typically, 400 to 500 images arerecorded, but any number of images can be used. Each resulting image correspondsto a single light source position (i.e sampled illumination direction) and thus a singlemeasured value of each reflectance function. Therefore, the user can decide at anytime to stop or to continue sampling light source directions and thus sampling valuesof the reflectance functions.

The photo shoot is easily realized in 25 to 30 minutes for 400 to 500 images. Inour case, the main bottleneck is the computer controlled Canon EOS D30 camera. Theimages are downloaded from the camera to the computer via USB 1.0 which takes 4seconds for a single photograph. However, this slow transfer gives the user some timeto move the light source to another position. Faster transfer is possible when usingUSB 2.0 or IEEE1394 transfer (also known as Firewire).

The hand-held light source is moved freely but held still every time a picture istaken to avoid illumination blur in the images. Illumination blur is similar to mo-tion blur and occurs when the light source is moved during the exposure time. Thisillumination blur would make the illuminant direction estimation more difficult. Toprevent illumination blur even more, a very short exposure time is selected. Becauseof this short shutter time, a very bright light source is needed, otherwise the result-ing images would be too dark and under-exposed. The light source used, is a hand-held Xenon light source, emitting uniformly distributed light and very small in area(10cm×10cm). This type of light source is adequate for our assumptions (i.e. a non-directional point light source) if we do not move it to close to the object. The lightsource emits a yellowish light. To compensate for this color shift, the light sourceis calibrated using a Gretag McBeth ColorChecker DC, and the recorded images aremodulated to simulate images recorded with a white light source.

The light source movement can be irregular, so large sections of the hemispherecould be under-sampled. To prevent this from happening, it helps to end the photoshoot with a series of images in a semi-regular pattern, emulating the gantry from theoriginal Light Stage. In figure 3.10 the free-form Light Stage can be seen in action.

The original Light Stage was built to relight objects from all directions on a sphere.For a practical implementation, the object is placed on the floor so the user can walkaround the object and move the light source freely around and over the object. Thislimits the incident illumination to the upper hemisphere. However, this is not a severerestriction since we want to see the shadows cast by the relit object onto the ground.If the full sphere of directions has to be sampled, an extension to a full sphere of lightdirections can still be achieved (e.g. placing the object and spheres on a tripod).

Care has to be taken that the spheres do not affect the illumination of the object,

Page 68: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

56 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Figure 3.10: The free-form Light Stage in action.

since they are not part of the object to be relit and their influence on the illuminationof the object should not be noticeable. Different problems could arise:

• The spheres are in front of the object. In the resulting image of the relit object,the spheres will block the view of the object as well. However, the spheres willbe correctly relit, as they are part of the scene.

• A sphere casts a shadow on the object. The object can be relit with a light map,but an unwanted blurred shadow of the sphere will be seen in the image of therelit object.

• The object casts a shadow on one of the spheres. This will not have an effecton the image of the relit object. It will however complicate the extraction of theilluminant direction.

These effects can be avoided by placing the spheres with caution. By placing thespheres at the side of and behind the object, shadows of the spheres will be cast on theback of the object, thus unseen. Basis images with shadowed spheres can be rejectedfor further use, since no decent illuminant direction can be estimated.

The spheres are best placed as far as possible from the object, reducing the inter-action and casting of shadows to a minimum. However, the further the spheres are setfrom the object, the more the camera needs to be zoomed out to capture all the spheresas well as the object. This would result in only a small set of pixels in each basis im-age representing the object. Although modern digital cameras have a very high imageresolution, still a lot of detail would not be captured. This can be overcome by usingtwo synchronized cameras. The first camera is zoomed out to capture the whole set ofspheres and these images are used to estimate the light source direction. The secondcamera is focused on the object alone and these pictures are used to relight the object.

Other methods for free-form sampling of the reflectance functions have been pre-sented in the last year:

• An array of similar chrome balls next to each other can be used instead of a setof four diffuse white spheres [32]. By taking an image with the light source

Page 69: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 57

very close to the camera position and finding the resulting highlights in therecorded image, the positions of the light probes can be found. The distancebetween two highlights determines the size of the chrome balls. The rest ofthe photo shoot then remains similar with the diffuse white spheres. For eachrecorded photograph the light source position can be estimated by calculatingthe illuminant direction for each chrome ball based on the recorded highlight inthat chrome ball and finding the intersection of all these estimated directions ofall chrome balls.

With an array of chrome balls, the position of the illuminant can be found veryprecisely. A disadvantage is that it becomes difficult to choose the shutter timefor the data acquisition. The chrome balls will produce specular highlights inthe image and therefore, the shutter time should be chosen very short. However,the shutter time should be large enough to capture the reflectance of the object.Another downside to this approach is that it requires a set of chrome balls whichcan be expensive.

• Another method is to mount an additional camera on the light source [44]. Thisextra camera is to be synchronized with the main camera and both cameras takephotographs at precisely the same instant. Instead of positioning a set of spheresaround the object that may cast unwanted shadows on the object, only a calibra-tion pattern, which is usually flat, needs to be visible in the neighborhood of theobject. The object can for instance be placed on a checker board pattern. Fromthe images of the extra camera, using camera calibration techniques, the cam-era position of the extra camera relative to the calibration pattern can be found.Since this camera is mounted on the light source, the position of the cameracorresponds with the illuminant direction. A disadvantage of this approach isthat it requires a second camera.

Again, radiance values instead of pixel values are needed of the recorded pho-tographs. Usually this is done using a series of photographs of the same scene withdifferent shutter times. This is not possible with this data acquisition method. Wecannot ask the user to keep the hand-held light source perfectly still to take a seriesof photographs. We take LDR photographs but minimize the saturation by tuning theaperture and exposure time of our camera. The response curve of the digital cameracan be extracted with the technique proposed in [13] before the photo shoot. The sin-gle LDR pictures can then be transformed into HDR images using the inverse responsecurve, as discussed at the end of section 2.5.3.

3.2.2.2 Data Processing

The data processing consists of the estimation of the illuminant directions from theshading of the white spheres and the construction of the reflectance functions of eachpixel for all directions in Ω, based on the estimated illuminant directions.

Illuminant direction estimation is required since the hand-held light source wasmoved uncontrolled and freely. We first discuss the illuminant direction estimation

Page 70: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

58 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

for a single sphere. By using a diffuse white sphere, the model of Lambert shadingcan be used on the spheres:

Lp = ρ Lin(Np ·ωin). (3.8)

This shading equation states that the radiance Lp at a point p, is dependent on theincident radiance Lin in that point p, the direction of the incident illumination ωin, thenormal Np of the surface at point p and the materials diffuse hemispherical reflectivityρ. The extraction of the illuminant direction ωin from a diffuse sphere is possible,based on this Lambert shading model.

Since we are using recorded images, we will have to assume that each pixel cor-responds to a small region on the sphere. For each pixel, we have a recorded value ofirradiance. However, due to the use of HDR images, these irradiance values are onlyknown up to a scale factor S :

S Ep = ρ Lin(Np ·ωin). (3.9)

We can assume the illumination radiance Lin is constant for all points on the whitesphere, because the light source has a uniform emittance distribution and the sphere issmall in comparison with the light source. Furthermore, the hemispherical reflectivityρ of the material is constant over the white homogeneous sphere. These two values areconstant and can be omitted, since we have an unknown scale factor S in the equationanyway:

S Ep = (Np ·ωin). (3.10)

This scale factor S can be well approximated using all N pixels on the sphere:

N

∑i=0

(S Epi) =N

∑i=0

(Npi ·ωin)

SN

∑i=0

Epi =N

∑i=0

(Npi ·ωin)

S =∑N

i=0(Npi ·ωin)

∑Ni=0 Epi

.

Therefore, we can write equation 3.10 as:

(∑N

i=0(Npi ·ωin)

∑Ni=0 Epi

)Ep = (Np ·ωin). (3.11)

Using this equation, the unknown direction of the illumination ωin can be foundonce the normals Np for each point p are found. These can be found using a numeri-cal algorithm, assuming all information is averaged out for each pixel separately andthe position of the sphere center C with respect to the camera viewpoint is known.Therefore, the normals Np are found in the following way:

Page 71: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 59

1. All pixels psi , representing the silhouette of the sphere, are located using aCanny Edge detection algorithm on the part of the image representing the sphere.

2. The directions from the camera viewpoint corresponding to these pixels psi arecomputed and called si. Note that these directions are normalized. The pointin space where the ray from the viewpoint in direction si grazes the sphere is inpoint (si ·C ) (i.e. the projection of C on direction si).

3. Using the measured radius R of the sphere, a good estimation for the sphereposition C can then be found by an optimization technique, minimizing theerror in the following equation:

‖C‖2 = R 2 + (si ·C )2. (3.12)

4. The normal Np for a pixel p can then be computed easily by intersecting thefound sphere with center C and radius R with the ray from the camera viewpointinto the direction corresponding to this pixel p.

The normals and shading information together can be used to find the illuminantdirection of a single sphere. Least square optimization techniques can solve equa-tion 3.11 for the illuminant direction ωin using the irradiance Ep and normal Np datafrom all N pixels containing shading information.

To test the accuracy of this illuminant direction estimation, we placed a diffusesphere in our version of the light stage and recorded images of this sphere from a cam-era viewpoint that was tilted 90 degrees from the zenith of the hemisphere covered inour light Stage. On top of figure 3.11, the set of images is displayed, representingthis diffuse sphere illuminated from a set of regularly sampled directions in latitude-longitude parameterization. The bottom left image (0,90) is completely dark, sincethe camera blocks the light source completely. As can be seen, the image correspond-ing to the light source being positioned behind the sphere (180,90) is dark as well,since the sphere occludes the light source. For directions close to being behind thesphere, lens flare effects prohibit from recording the sphere clearly.

Since we use a Light Stage and therefore know the illumination direction for eachrecorded image and we can estimate the directions with our optimization technique,we can find the accuracy of the illuminant direction estimation. At the bottom of fig-ure 3.11, the angle between the estimated illuminant direction and the true illuminantdirection is expressed in degrees. As can be noted, the error is on average less than2.5 degrees. In our setup, we used an area light source of radius 2cm positioned ata distance of 1m from the diffuse sphere. Since we assume the use of directionallight source in our illuminant direction estimation algorithm, the error can easily bearctan(2/100) = 1.14 degrees, with our setup. For ill-conditioned directions (i.e. di-rections opposite to the camera direction) the error increases significantly. However,there is little we can do about this, since these directions result in very few pixelsbeing illuminated (i.e. only the “edge” of the sphere). Therefore, there are only fewpixels with valuable information while all other pixels represent noise. Furthermore,lens flares cannot be eliminated and the occlusion cannot be prevented.

Page 72: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

60 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Note the symmetry in the data and the error angles. The image of a sphere illu-minated from (φ,θ) is similar to the horizontally mirrored image of the sphere illumi-nated from (360− φ,θ). The difference between two nearly identical images, exceptfor the mirroring, gives an idea of the noise on our data.

3.7 2.7 1.8 1.9 1.2 3.3 12.2 18.8 141 16.4 2.2 1.6 2.8 3.2 3.0 2.72.4 0.9 1.4 2.4 1.7 1.9 6.7 11.8 42.4 10.6 2.0 1.3 2.6 1.8 2.2 1.71.9 0.1 0.6 1.9 0.9 2.5 5.7 9.1 13.8 10.0 2.9 0.7 1.7 1.9 1.4 1.30.8 1.1 1.2 1.3 0.8 2.9 6.9 9.6 11.7 12.2 4.8 1.8 1.6 1.1 1.0 1.41.3 1.5 1.9 2.1 1.3 1.5 4.2 6.4 7.8 6.2 2.4 0.9 0.8 0.4 1.5 1.60.4 1.0 1.3 1.6 1.1 0.2 2.1 4.1 4.4 3.1 2.5 2.7 2.3 2.2 1.1 1.11.8 2.4 2.4 2.0 1.8 2.6 3.6 4.7 3.3 2.7 2.4 2.2 2.0 2.6 2.8 2.80.8 0.9 0.7 0.8 1.5 2.8 3.3 3.0 1.9 2.0 1.7 1.3 0.3 0.8 1.8 1.7

n/a 3.0 3.1 2.5 2.6 1.4 6.0 17.7 116 20.6 2.0 2.9 4.0 4.0 3.5 3.5

0 360 90

0

Figure 3.11: On top, a diffuse sphere illuminated from 16× 9 regularly sampledknown directions. At the bottom, the angle, expressed in degrees, between the es-timated illuminant direction and the exact known illuminant direction.

From our experience in using this direction estimation technique, we found somepractical issues that improve this direction estimation procedure to some extent:

• We opt for placing four spheres symmetrically around the center of our realscene. The illuminant direction is then computed for each sphere separately andthe average of the found directions is used to find the illuminant direction withrespect to the center of the scene. This also allows to find the direction even ifthe real object casts a shadow on one of the spheres.

• Since all cameras record color images, we can perform the optimization onthree data sets: one for each color channel. By taking the average over the three

Page 73: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.2. PULSE APPROACH 61

resulting directions we reduce the factor of noise in the data.

• From our data set in figure 3.11, especially the marked area, we learn that thedirection cannot be well estimated when the illumination originated from direc-tions opposite to the view direction. Therefore, we suggest to tilt the cameraonly 45 degrees or less, relative to the pole of the hemisphere. In such a setup,the illumination never originates from these ill-conditioned directions.

• We found it useful to position the object and spheres on black velvet in a darknon-reflective environment. Otherwise, due to inter-reflections in the scene, thespheres might contain some additional information. This would add structured“noise” to the spheres, resulting in incorrectly estimated illumination directions.

Reflectance function reconstruction can be performed once all the N illuminantdirections ω1,ω2, ...,ωN are estimated. With this data and the basis images, a sim-ilar reconstruction technique can be used, as with the Light Stage. The reflectancefunctions Rx,y(ω) over the complete domain Ω can again be constructed as a piece-wise constant function defined over a set of N disjunct partitions Ωi of Ω satisfyingthe constraints of 3.4. However, the definition of these Ωi is not straightforward dueto the non-regular sampling of the illuminant directions. These illuminant directionswill generally not be regularly sampled in any parameterization.

The estimated illuminant directions ω1,ω2, ...,ωN can be used as sites for anangular Voronoi diagram on the hemisphere, using the angles between directions as adistance measure. The N directions ωi will result in a set of N Voronoi cells. TheseVoronoi cells are the subsets Ωi. The Voronoi diagram brings an elegant solution tothe non-uniform distribution of light source directions over the hemisphere, whichresults from the uncontrolled movement of the hand-held light source. Constructingan angular Voronoi diagram is independent of the parameterization used to representthe incident light map. Any hemispherical parameterization can be used.

The reflectance functions can then again be approximated as:

Rx,y(ω) = Imgi(x,y) with ω ∈Ωi. (3.13)

For the example, we use the same object as we used for the Light Stage and se-lected the same 5 pixels, in order to be able to compare the different reconstructed re-flectance functions. Figure 3.12 displays examples of reconstructed reflectance func-tions.

With this formulation, the Light Stage with regularly sampled illumination direc-tions becomes a special case of the free-form Light Stage. The Voronoi diagram ofregular sampled directions in latitude-longitude parameterization is equal to the subsetdefinition of equation 3.5.

3.2.2.3 Relighting

Once the reflectance functions for all pixels have been constructed, an image of the ob-ject, illuminated with any environment map can be created using the relighting equa-tion for each pixel (equation 3.1). Using the reflectance functions of figure 3.12 to

Page 74: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

62 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

a

c

e

b

db

d

e

a

c

Figure 3.12: The five selected pixels from figure 3.5, with 320 randomly selectedillumination directions from the total of 1280 regular sampled directions.

relight the miniature race car with the environment map of the Salons of the Arenbergcastle results in a similar image as in figure 3.6.

In figure 3.13, some selected basis images resulting from the free-form Light Stagedata acquisition technique are displayed. A set of toy soldiers, surrounded with fourdiffuse white spheres is depicted, illuminated by one single light source from an un-known direction. For this scene, we recorded 500 basis images for which the cor-responding illuminant directions were estimated, based on the shading of the whitespheres. With this information, the reflectance functions for each pixel could be re-constructed.

In figure 3.14, the set of toy soldiers is displayed four times with different novelincident illumination. The applied incident illumination for each image of the relitscene is depicted as well. As can be seen on the bottom left of this figure, the tripodof the toy camera in the scene casts a long shadow over the ground surface, which isto be expected for light incident at grazing angles.

Note that instead of reconstructing the reflectance functions, one could also use theapproach of the weighted sum of basis images. Down-sampling the normalized en-vironment map cannot be done, since no regular sampled illuminant directions wereused. Therefore, the incident illumination is re-sampled to the same resolution as theilluminant directions (i.e. N). This can be accomplished by the use of an angularVoronoi diagram: the angular Voronoi diagram is overlayed on the normalized envi-ronment map and each Voronoi cell acts as a box filter. This results in the N incidentillumination values, which are used to modulate the basis images. By summing to-gether the modulated basis images, the final image of the relit object is created.

Page 75: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.3. STRUCTURED APPROACH 63

Figure 3.13: Some selected images of the 500 recorded basis images of a set of toysoldiers. The four white diffuse spheres can easily be noted, and form the shadingpattern, the illuminant direction can be estimated. As can be seen on the top right, thespheres may cast a shadow onto the scene for illumination incident at grazing angles.

3.3 Structured ApproachInstead of using illumination conditions representing directional illumination froma single direction, one could also use illumination conditions defined over a set ofdirections. For example, the problems with specular objects discussed on page 53,when using a Light Stage approach, can be overcome: one could use the illuminationconditions Ci defined as:

∀i, 1 ≤ i ≤ N : Ci(ω) =

1 Wm2×sr if ω ∈Ωi

0 Wm2×sr if ω /∈Ωi

(3.14)

with the partition of Ω into Voronoi cells Ωi based on a set of directions. With theseillumination conditions, high frequency features such as highlights, would not bemissed for some reflectance functions. However, features in the reflectance functionwould be blurred together. Also note that the reconstruction of the reflectance func-tions would correspond better with these illumination conditions. These illuminationconditions would then allow to measure reflected irradiance for all reflectance func-tions, and avoid situations where no information can be measured of some reflectance

Page 76: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

64 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Figure 3.14: The set of toy soldiers from figure 3.13, is visualized with four differentillumination conditions, using the free-form Light Stage technique.

functions as in figure 3.8. However, it is very difficult to illuminate a real object withillumination conditions, continuous over (a part of) Ω. In the following sections wesurvey relighting techniques based on structured directionally dependent illuminationconditions.

Page 77: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.3. STRUCTURED APPROACH 65

3.3.1 Steerable FunctionsNimeroff et al. [41] developed a technique based on steerable functions to relightobjects, which we now review briefly. A steerable function is a function that canbe expressed as a linear combination of rotated versions of itself. An example of asteerable function is the function c(u;s) = u · s, called the directional cosine function,which takes the inner product of direction u with a given direction, both expressed inCartesian coordinates. However this function c(u;s) can also be written as

c(u;s) = sxc(u;ex) + syc(u;ey) + szc(u;ez),

with ex, ey and ez the unit vectors spanning the Cartesian space. As can be seenc(u;s) can be written as a linear combination of rotated versions of itself and is there-fore a steerable function. Of course, any three non-coplanar directions s1, s2 and s3could be used instead of ex, ey and ez. The directional cosine function values of s1, s2and s3 can be written as:

c(u;s1)c(u;s2)c(u;s3)

= M3

c(u;ex)c(u;ey)c(u;ez)

,

with M3 a 3×3 matrix and the directional cosine function c(u;s) can then be writtenas:

c(u;s) = sT M−13

c(u;s1)c(u;s2)c(u;s3)

.

Selecting the 3 directions s1, s2 and s3 results in three functions c(u;s1), c(u;s2)and c(u;s3) of direction u. These functions can be used as illumination for an object.The object can then be visualized, illuminated with any function c(u;s), using thesuper-position principle of light transport and by creating a weighted sum of the threebasis images. However, c(u;s) can be negative. To solve this problem, a functionL(u;s) = 1 + c(u;s) can be used:

L(u;s) = 1 + sxc(u;ex) + syc(u;ey) + szc(u;ez).

This function is no longer a steerable function but can be made steerable in asimilar way as before, by choosing 4 directions s1, s2, s3 and s4 and defining L(u;s)as:

L(u;s) =

1sxsysx

T

M−14

c(u;s1)c(u;s2)c(u;s3)c(u;s4)

.

With this approach, selecting the 4 directions s1, s2, s3 and s4 results in four func-tions L(u;s1), L(u;s2), L(u;s3) and L(u;s4) of direction u. These functions have only

Page 78: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

66 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

positive values and can be used as illumination functions and hence, illumination con-ditions.

In the technique of Nimeroff et al. [41], a polynomial of the directional cosinefunction is used as steerable function f (u;s):

L(u;s) = a0 + a1c(u;s) + a2c(u;s)2,

and requires to select 10 directions, defining 10 functions L(y;si) and thus 10 basisimages. Although more basis images need to be recorded, the steerable functions canbe made narrow by tweaking the values of a0, a1 and a2. From these 10 basis images,the object can then be visualized with any illumination function L(u;s).

Higher order steerable functions could be used as well, allowing to use even nar-rower functions, with the cost of having to record more basis images.

The technique of Nimeroff et al. [41] fits in our framework since it requires toselect N (i.e. 10) illumination conditions L(y;si) and illuminate the object with theseconditions while an image is recorded. The image can then be visualized with anarbitrary illumination function L(u;s) by creating a linear combination of the basisimages, depending on the weights of L(u;s) expressed as weighted sum of L(u;si).Instead of doing this for the whole image at ones, this can be done per pixel. As aresult, a continuous reflectance function can be constructed, expressing the irradianceof the pixel when to object is illuminated with L(u;s).

Although this technique is very elegant and well constructed, it can only be ap-plied to virtual objects and scenes like the technique of [4], since the used steerablefunctions are very hard to create in reality as illumination conditions.

3.3.2 Basis Functions as Illumination ConditionsSince in theory, any set of functions can be used as illumination conditions, a set ofbasis functions can be used as well. Although no practical image-based relightingmethods for real objects have originated from this idea, we still describe it using ourframework. This short mathematical deduction allows us to demonstrate the powerof having a general relighting framework: with an intuitive relighting approach, onewould never come up with such a mathematically elegant technique.

With equation 3.1, an object can be relit, once the reflectance functions for allpixels have been obtained. In section 2.4.4.3, it was also noted that faster relightingcan be achieved if the incident illumination Lin and the reflectance functions Rx,y areprojected onto a common set of basis functions Bi. By projecting the functions Linand R onto sets of basis functions, the values li and rx,y

i are obtained (section 2.4.5).Instead of obtaining the reflectance functions Rx,y and projecting them onto a set

of basis functions Bi, the values rx,yi can also be obtained directly by using the illumi-

nation conditions Ci that are equal to the basis functions Bi:

∀i, 0 ≤ i ≤ N : Ci(ω) = Bi(ω). (3.15)

The illumination conditions are the set of basis functions and nature does the pro-jection of Rx,y onto this set of basis functions for us. Although this is mathemati-cally very elegant, it may be impractical. Since basis functions are often continuous

Page 79: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.4. RELIGHTING WITH SPATIALLY DEPENDENT ILLUMINATION 67

over Ω and may contain negative values, the illumination conditions C1(ω), C2(ω),...,CN(ω) may not be easily reproducible in reality. Therefore, not all sets of basisfunctions can be used, even if they are theoretically well chosen.

This approach is therefore restricted to virtual environments until arbitrary angu-larly dependent illumination conditions can be created in the real world.

3.4 Relighting with Spatially Dependent IlluminationAlthough this chapter focuses on relighting with directionally varying incident illumi-nation, relighting with spatially dependent incident illumination fits in our relightingframework as well. Spatially dependent incident illumination describes the angularlyuniform distributed emitted light, originating from each position on a surface (i.e. a2D location and thus 2D illumination as well).

In section 3.4.1, we refine the general framework, this time for relighting withspatially varying incident illumination. In subsequent sections, we demonstrate thatpublished relighting methods for relighting with spatially dependent illumination fitin our framework.

3.4.1 The General Framework Revisited, againIn section 3.1.1 we revisited the general relighting equation and transformed it inorder to use it for relighting with angularly varying incident illumination. The samededuction can be performed for spatially dependent incident illumination. Using onlyspatially dependent incident illumination allows to eliminate the angular argument inthe illumination function:

Lin(p← ω) = Lin(p).

This simplification then assumes the light emitted from position p on surface S, isuniformly distributed in all directions. With this illumination, the relighting equationcan be written as:

E(x,y) =Z

S

Z

ΩRx,y(p← ω) Lin(p) dω dp,

=

Z

SLin(p)

Z

ΩRx,y(p← ω) dω dp

or

E(x,y) =

Z

SLin(p) Rx,y

pos(p) dp,

with

Rx,ypos(p) =

Z

ΩRx,y(p← ω) dω .

Page 80: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

68 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

The reflectance function for pixel (x,y) thus becomes the 2D function and the re-stricted reflectance field can be reduced from a 6D function to a 4D reflectance field,because only the position of the incident illumination and not the angle is of impor-tance. Similarly as for relighting with directionally dependent incident illumination,even though the dimensions of function Rx,y

pos(p) and Rx,y(p← ω) differ, we will dropthe subscript pos and denote the function as Rx,y(p).

The general relighting framework requires to choose a set of N illumination con-ditions C1,C2, ...,CN, which are functions of the position on the surface S. Animage is then to be recorded of the object, illuminated with each of these illuminationconditions.

3.4.2 Pulse ApproachSimilar to the angularly dependent incident illumination, the N illumination conditionsCi can be defined by a set of pulses on the domain S, being positions p1, p2, ..., pN:

∀i, 0 ≤ i ≤ N : Ci(p) =

1 Wm2×sr if p = pi

0 Wm2×sr if p 6= pi.

(3.16)

The illumination conditions correspond with point light sources placed at positionsp1,p2, ...,pN on surface S, emitting light in all directions. Of course, an ordinarylight source (e.g. light bulb) emitting uniformly distributed light on the object can beused as well.

An example of this relighting approach is the technique proposed by Lin et al. [34].In this technique, the point light source positions are defined by a grid on a plane S′and parameterized as (q,r) (with 0 ≤ q,r < N and q,r ∈ N). Note that S′ is only aplane and not a hemisphere enclosing the object. The N2 basis images of the objectare recorded with the point light sources illuminating the object from each grid pointseparately.

A reflectance function Rx,y(p) is defined on this plane S′ (i.e. p ∈ S′) and isconstructed using bi-linear interpolation: the value of S′(q′,r′), with q′,r′ ∈ R and(q′,r′) on S′, is computed by bi-linear interpolation of the values bq′c, dq′e, br′c anddr′e. In the article of Lin et al. [34], this technique is further extended to allow theobject to be viewed from an arbitrary viewpoint and directional light sources can beapproximated.

A similar approach was taken in the work of Koudelka et al. [38] where multiplesets of data are acquired with the light source at different distances to the object, toallow to move the virtual point light source to an arbitrary position. However, in thistechnique, only direct illumination effects can be achieved.

3.4.3 Structured ApproachInstead of using pulse functions as spatially varying illumination conditions, struc-tured illumination conditions can be used as well, as in the case of directionally vary-ing illumination conditions. No real devices exist to create such angularly dependent

Page 81: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.4. RELIGHTING WITH SPATIALLY DEPENDENT ILLUMINATION 69

continuous illumination conditions. However, there do exist illumination devices ca-pable of emitting arbitrary continuous spatially dependent illumination conditions.Plasma screens, LCD-screens and even common CRT monitors are capable of per-forming this task (if pixels are assumed to be continuous points on the screen). Eventhough these devices usually do not allow to emit light from an enclosing surface of theobject, they allow to emit light from a small continuous area. We overview some pub-lished techniques using such devices to demonstrate that these relighting techniquesalso fit in our general framework.

3.4.3.1 Environment Matting

Blue screen matting is a matting technique introduced in 1984 by Porter and Duff[47]. In that technique, an object is put in front a blue screen and an image is recorded.From this information, a matte can be derived. This matte has the same resolution asthe image and defines for each pixel if the pixel represents the object (object pixel)or the background (background pixel). Creating this matte is done per pixel and isstraightforward: if the pixel in the recorded image is blue, the corresponding pixel inthe matte is defined as background pixel, otherwise, the pixel of the matte is set to bean object pixel.

This matte then allows to visualize the recorded object in front of an arbitrarybackground image. This can be accomplished by replacing the background pixels ofthe matte with the corresponding pixels of the new background image and replacingthe object pixels of the matte with the corresponding pixels of the recorded image.

There can be a problem if the object is blue as well. During the creation of thematte, no distinction can be made between these blue pixels of the object and bluepixels of the background. Thus, the blue object pixels would be assumed to be pixelsof the background. This would generate a “hole” in the object. However, any colorcan be used for the back screen, and thus a color not appearing in the object can beselected for the back screen during the data acquisition.

Although this allows to find a matte for an object, there are some problems withspecular and refractive objects. These objects will reflect or refract the blue screenand make it difficult to distinguish between pixels of the blue screen and pixels rep-resenting the object that reflect or refract light from the blue screen. Using anothercolor for the back screen will not help in this case since any background color will bereflected or refracted.

Environment matting, introduced by Zongker et al. [62], was developed to over-come this problem. An environment matte considers 3 types of pixels:

• pixels not representing the object,

• pixels representing a part of the object that reflects or refracts light of the back-ground,

• other pixels representing a part of the object.

Page 82: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

70 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

Suppose the category for each pixel is known, then for background pixels, the bluescreen matting procedure can be used: replace the color with the novel backgroundcolor. For pixels representing parts of the object that do not reflect or refract light ofthe background, the blue screen matting procedure can be used as well: copy the colorof the pixel from the original photograph of the object. However, the color of pixelsthat represent parts of the object that do reflect or refract light of the background can-not be easily found. For these pixels the reflectance functions need to be constructed inorder to find the color of the pixel for an arbitrary background (i.e. lighting). Environ-ment matting techniques are matting techniques that allow to capture the informationnecessary to construct these reflectance functions in order to visualize the pixels ofthis category and have a correct matting of the object with an arbitrary background.

The environment matting techniques can be described using our general relightingframework for pixels representing a part of the object that reflects or refracts lightof the background illumination. We can describe the standard environment mattingtechnique [62] with our framework:

The incident illumination can only originate from a background plane (i.e. a CRTmonitor). Therefore, the surface S′ on which the incident illumination is defined, canbe parameterized with (u,v), with 0 < u ≤ U, 0 < v≤ V. The incident illuminationcan be represented as:

Lin(u,v) with (u,v) ∈ S′.

A set of 2×P illumination conditions is selected. We denote this set of illumina-tion conditions as Ch

1(u,v),Ch2(u,v), ..., Ch

P(u,v), Cv1(u,v), Cv

2(u,v), ..., CvP(u,v).

These illumination conditions can be separated in the horizontal illumination condi-tions Ch

1, Ch2, ..., Ch

P where Chi is independent of u and the vertical illumination

conditions Cv1, Cv

2, ..., CvP where Cv

i is independent of v.Because a CRT monitor is often used to emit the illumination conditions, only a

discrete set of points are available. Therefore, the extra constraint of u,v∈N is added.To be able to relight the object with a large variation of possible incident illuminationsets, large U and V (e.g. 1024) are chosen.

In the technique of [62], patterns based on the one-dimensional Gray codes aredisplayed on the CRT twice, once horizontally and once vertically. For each singlepattern, a basis image is then recorded. The length of a sequence of one dimensionalGray codes always is a power of two and can be constructed in the following manner:

1. Start with the Gray code sequence ”0,1”.

2. Make a copy of the sequence.

3. Reverse the order of the copy.

4. Prepend a ”0”to each number in original sequence.

5. Prepend a ”1”to each number in the copy.

6. Concatenate the original and the copy together to create a new Gray code se-quence twice as long as the original.

Page 83: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.4. RELIGHTING WITH SPATIALLY DEPENDENT ILLUMINATION 71

7. To double the length of the Gray code, go back to step 2.

Using our framework, we can define Chi (u,v) as the uth pattern based on the one-

dimensional Gray codes of length U. Similarly, Cvi (u,v) is the vth pattern based on the

one-dimensional Gray codes of length V. For instance, the sequence of Gray codesof length 3 is 000, 001, 011, 010, 110, 111, 101, 100. The three patterns are then00001111, 00111100 and 01100110. Additionally, the pattern 11111111 will be usedas well.

From the pixels of these basis images, the reflectance functions need to be con-structed over the domain S′ of each pixel. Since these pixels contain the informationof the reflection or the refraction of a pixel, Zongker et al. [62] chooses to representthe reflectance function of each pixel as an axis aligned box filter on the incident illu-mination defined on S′. By choosing an axis aligned box filter, only the top t, bottomb, left l and right r of the boundaries of the box on the domain S′ need to be found.Moreover, by using only the basis images using the illumination conditions Ch, thevalues for r and l can be found by optimization. Similarly, using the basis imagesfor the illumination conditions Cv, the values for t and b can be found by optimiza-tion. If ρ is the color modulation factor defined by the absorption and reflectance ofthe surface of the object, the reconstructed reflectance function for pixel (x,y) thenbecomes:

Rx,y(u,v) =

ρ 1(r−l)×(t−b) if l≤ u≤ r and b≤ v≤ t

0 otherwise.(3.17)

Visualizing the object with a novel background can then performed using the re-lighting equation for the pixels with a reflectance function and using the blue screenmatting procedure for the two other categories of pixels.

To categorize the pixels in the matte, the set of photographs of the object illumi-nated with the patterns on the CRT monitor are recorded, followed by an extra set ofimages with the same patterns but without the object in front of the CRT monitor. Thisresults in a set of couples of photographs, where each couple consists of a photographof a pattern illuminating the object and a photograph of only the pattern. Each pixelin the matte can then be categorized in the following way:

• The pixel is equal in all couples of photographs. Since all patterns have a one-to-one mapping of the background images with the object images, the pixel canbe categorized as representing the back screen.

• The pixel differs in some couples. The illumination patterns have an effect onthe pixel color and thus, the pixels represents a part of the object that reflects orrefracts the illumination of the back screen.

• The pixel differs in all couples of photographs and the pixel is the same for allphotographs of the object. Since no matter which pattern is illuminating the

Page 84: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

72 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

object, the pixels remains unchanged, this pixel can be categorized as a part ofthe object not reflecting nor refracting the object.

Environment matting is only practical for relighting over a small area (i.e. thesize of the CRT monitor). This can be ameliorated by performing the data acquisitionprocess multiple times, each time with the CRT monitor at a different position, forexample, as a back drop, a left side drop, a right side drop and a top.

The standard environment matting technique can be further extended: Chuanget al. [9] extended the idea of environment matting in two ways. The first extensionconsists of using a large set of Gaussian convoluted band patterns instead of Graycoded stripe patterns as illumination conditions. Based on the basis images of theobject illuminated with these patterns, an optimization method allows to create thereflectance functions of the pixels. These reflectance functions consist of a set oforiented elliptical Gaussian functions on the surface S′ for each background or sidedrop. A second extension allows to find the reflectance functions of the object withonly a single illumination condition and thus a single basis image. However, in thiscase, the object may not modulate the color of the incident illumination and the objectneeds to be perfectly specular or refractive.

Peers and Dutre [45] subdivided the surface of the CRT monitor (i.e. S′) into aset of tiles or patches. For each separate patch, an optimized standard environmentmatting data acquisition method is executed, resulting in a an axis aligned box func-tions for each patch, or a set of box functions for the entire CRT monitor. Since thereflectance functions are now defined as a set of box functions, diffuse objects as wellas specular objects can be relit or matted faithfully. By using 4 CRT monitor positions(back, left side, right side and top) this matting techniques is extended to a relightingtechnique.

Recently, a matting technique was proposed that progressively refines the reflectancefunctions in an environment matting approach [36]. The illumination conditions area set of natural illumination conditions such as images or parts of environment maps.The reflectance functions are expressed as a set of rectangular box functions.

3.4.3.2 Wavelet Environment Matting

In section 3.3.2, we theoretically deduced the idea that the coefficients of the re-flectance functions, projected onto a set of basis functions could be captured directly,enabling fast relighting. However, this requires to emit continuous directional illumi-nation conditions, which is practically impossible for real objects, since such displaydevices do not exist currently. Since spatially dependent structured illumination condi-tions can be created in reality, this process can be practically executed for real objectsfor relighting with spatially dependent incident illumination.

Using the setup of environment matting, basis functions can be emitted on the CRTmonitor behind the object. Peers and Dutre [46] followed this approach, and used aset of basis functions based on wavelets. This wavelet environment matting approachfits in our relighting framework as well. We outline this technique very briefly.

Page 85: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.5. SOME NOTES ON RELIGHTING WITH 2D ILLUMINATION 73

Suppose we use a wavelet function Ψ(u,v) and note the wavelet functions result-ing from a translation b and a contraction a as Ψa,b(u,v). The N illumination condi-tions C1(u,v),C2(u,v), ..., Cn(u,v) correspond to different wavelet basis functionsΨi with i = 2a + b. By illuminating the object with these illumination conditions, weobtain a set of basis images Img1,Img2, ...,ImgN .

To relight the object, both the incident illumination Lin and the reflectance func-tions can be projected onto a common set of basis functions. The corresponding co-efficients of incident illumination and reflectance functions need to be multiplied andsummed together to obtain the pixel values of the image of the relit object (2.4.4.3).Since we used the basis functions as illumination conditions, the reflectance functionsneed not be projected onto the set of basis functions since each basis image Imgi al-ready contains the coefficients for basis function i. The incident illumination Lin canbe expressed in wavelet space using the wavelets used for the data acquisition, result-ing in coefficients li. The irradiance of the pixel (x,y) of the image of the relit objectcan then be calculated as:

E(x,y) =N

∑i=0

li× Imgi(x,y), (3.18)

and the resulting image can be calculated as a linear combination of the basis images:

Image =N

∑i=0

li× Imgi. (3.19)

In order to work, this technique would require a tremendous amount of illumina-tion conditions. To prevent this, the wavelet environment matting technique [46] usesa feedback loop during the data acquisition process to select the next most importantbasis function. With this approach the data acquisition can be halted at any time andthe reflectance functions are expressed as a non-linear approximation.

3.5 Some Notes on Relighting with 2D IlluminationIn the previous sections, a distinction was made between relighting with directionallyvarying incident illumination and spatially varying incident illumination: on the onehand, if the illumination is only dependent on the direction, then the position in theillumination is of no importance. On the other hand, if the illumination only dependson the position, then the directional parameter can be neglected for the illumination.

In this section we briefly overview the assumptions required to make these restric-tions and discuss the consequences of these restrictions for relighting with 2D illumi-nation. Additionally we demonstrate that environment maps can express directional2D illumination as well as positionally dependent incident illumination.

3.5.1 Restriction to 2D IlluminationBoth positional and directional illumination are tools to restrict the 4D illumination toa 2D parameterizable data set and are based on the assumption:

Page 86: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

74 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

The incident illumination L(p←ω), with point p on S, is assumed to be independentof the direction ω and can be written as L(p).

This assumption allows to restrict the 4D illumination to 2D illumination. Depend-ing on the choice of S, the assumption allows to express this restricted incident illu-mination as spatially varying incident illumination or directionally varying incidentillumination.

• If S is a surface enclosing the object, then this assumption leads to positionallydependent incident illumination. For each point p on surface S the radiance isindependent of the direction (i.e. p is a point light source) and thus L(p).

• If S is an imaginary surface at infinity (e.g. a sphere with an infinitely largeradius), this assumption corresponds to directionally dependent incident illumi-nation. Since a point on a sphere corresponds to a direction, the position p onthis hemisphere can be defined with the direction ωp and thus the illuminationcan also be represented as L(ωp) = L(p). A point light source at infinity corre-sponds to a directional light source, hence the name ’directional illumination’.

The dimensional reduction from 4D to 2D thus originates from the assumptionthat the distribution of emitted light for each point on surface S is independent of theangle. The difference between the two restrictions is based on the choice of the surfaceS.

3.5.2 Consequences for Relighting with 2D IlluminationThis assumption and the resulting dimensional reduction of the incident illuminationfrom 4D to 2D has the following consequences.

3.5.2.1 Dimensional Reduction of the Relighting Equation

As discussed in section 3.1.1 and in section 3.4.1, the relighting equation can be di-mensionally reduced as well, resulting in 2D reflectance functions. This allows verypractical data acquisition techniques as was demonstrated throughout this chapter.However, this reduction also imposes some extra constraints on the data acquisition.

Extra Constraints for Directional illumination

Assuming the surface S is at infinity, requires the use of directional illumination. Sincedirectional light sources are hard to find, we use a common light source. In this case,an additional requirement has to be made:

• The object needs to be very small in comparison to the distance to the lightsources. This allows to emulate a directional light source (figure 3.1) with acommon light source. If the object is not small compared to the distance to thelight sources, the light sources do not emulate a directional light source.

Page 87: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

3.5. SOME NOTES ON RELIGHTING WITH 2D ILLUMINATION 75

Extra Constraints for Positional illumination

With positionally dependent incident illumination, the surface S cannot be any surface.The incident illumination needs to be expressed as point light sources on the surface Sand thus as L(p). To represent the illumination of an environment with such a functionL(p), an extra requirement is necessary:

• The surface S must coincide with the geometry of the surface emitting the illu-mination.

If the surface S deviates from the true geometry an error is made and this error in-creases the more the surface S deviates from the true geometry.

3.5.2.2 Limited Illumination Effects

By assuming angular independence of the incident illumination, some illuminationeffects are no longer possible in the resulting images of the relit object. Relightingwith such 2D illumination will never result in objects being illuminated only partially(i.e. spot light effects or objects partially in shadow).

3.5.3 Environment Maps Expressing 2D illumination

An environment map, as introduced in section 2.5.2.1, captures the illumination foreach direction incident at a single point L(pfixed← ω). The following assumption canbe made:

• For all points p on S, the incident illumination L(p ← ω) = L(pfixed ← ω).This assumption holds only for environments for which the radiance originatesfrom a large distance. If the environment closes in, this asumption is not met,and the captured incident illumination does not correspond to the true incidentillumination.

Under this assumption, an environment map can represent directional incident illumi-nation.

An environment map can also represent positional incident illumination defined forsurface S. Since positional incident illumination is defined by point light sourceson surface S and each point light source emits the same radiance in all directions, itsuffices to know what radiance is emitted by each point in a single direction.

An environment map is a function defining the incident illumination in a singlepoint pfixed for all directions ω as L(pfixed← ω). Therefore, the radiance emitted by apoint light source at position p on surface S can be found from this environment map:if the direction from point p to point pfixed is noted as ωp, then this radiance value isL(pfixed ← ωp). However, when using an environment map as incident illuminationfor relighting with positionally varying incident illumination, an extra assumption hasto be made:

Page 88: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

76 CHAPTER 3. RELIGHTING WITH 2D ILLUMINATION

• For each point of the geometry of this environment, the radiance must be thesame for all directions, otherwise, the environment map does not hold all nec-essary information.

3.6 ConclusionIn this chapter, the general relighting framework was adapted twice to handle relight-ing with 2D illumination: once for relighting with incident illumination restricted toonly directionally varying incident illumination and once for relighting with spatiallyvarying incident illumination. For both cases, the relighting equation was dimen-sionally reduced and techniques were discussed where the reflectance functions aresampled and reconstructed or where structured illumination conditions were used forthe data acquisition so as to obtain the reflectance function coefficients.

Sampling the reflectance functions results in very practical relighting techniques,such as the Light Stage, Free-form Light Stage and the technique of Lin et al. [34].However, due to the use of sampling, high-frequency features, such as highlights,may be missed, rendering these relighting methods impractical for near-specular tospecular objects. Nonetheless, these techniques are applicable to a wide range ofobjects.

Techniques using structured illumination conditions, such as the use of steerablefunctions or basis functions in general, do not suffer from these problems and areable to visualize near-specular to specular objects under arbitrary novel illumination.However, these techniques often demand to recreate illumination conditions impossi-ble in reality, due to the continuous character of the functions or the use of negativeillumination.

As a conclusion, the user has to choose between sampling the reflectance functionsand not being able to relight specular objects, or relight objects of any kind of materialbut not being able to relight these objects from all directions, due to the restriction ofthe display devices.

Page 89: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 4

Relighting with 4D Illumination

In this chapter we relight real objects with 4D incident illumination, expressed in aincident light field, as introduced in chapter 2. The use of incident light fields willallow to achieve all possible illumination effects in the relit images, without any as-sumptions concerning relations between the novel illumination and the object. How-ever, this will also require to use the relighting equation, as presented in our generalrelighting framework (section 2.4), without any dimensional reductions. Therefore,more complex techniques need to be designed to acquire the 6D reflectance fields, orequivalently, the 4D reflectance functions for each pixel.

4.1 IntroductionThe use of 2D illumination in a relighting context allows us to reduce the dimension-ality in the relighting equation, as described in chapter 3, and allows to express thereflectance functions with 2D functions. In case of applying only directionally de-pendent incident illumination, the object needs to be very small in comparison to thedistance to the environment in which the incident illumination (i.e. environment map)is recorded in order to obtain faithful results. For example a large environment andsmall object, like a miniature race car in a large room, can be relit faithfully using2D illumination. However, not all illumination effects can be achieved, due to theseimplied assumptions. Once the object becomes large with relation to the illuminationenvironment, the relit images contain large errors and are no longer faithful to whatthe object would look like in reality.

As an example: visualizing the effects of an object illuminated directly by thesun can be easily produced using relighting with 2D illumination, since the object isinfinitely small compared to the distance to the sun. However, visualizing the effectsof an object illuminated by the sun, that is partially blocked by Venetian blinds on thewindow (figure 4.1), cannot be achieved with 2D illumination. In this last case, notonly the direction of the sun defines the incident illumination to the object but alsothe Venetian blinds, and thus, the distance to what defines the incident illumination ismuch smaller.

77

Page 90: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

78 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

Figure 4.1: A paper and a little soccer ball are partially illuminated by the sun becauseof some Venetian blinds on the window.

This problem is due to the fact that the 2D incident illumination cannot representthese features of the incident illumination. In case of the 2D incident illumination onlydependent on direction, the incident illumination is captured at a single point. In theexample of the sun blocked by the Venetian blinds, the point where the environmentmap was recorded was either illuminated by the sun or was in the shadow of theVenetian blinds. Therefore, using only an environment map would result in relightingthe object as if it was completely illuminated by the sun (and thus lacking shadowbars) or as if the object was completely in the shade (and thus lacking sun bars). Thisis because the illumination depends on both position and direction and neither one ofthem can be omitted.

Incident light fields, or 4D illumination, can represent all the illumination featuresin an environment. Therefore, we need to be able to relight objects with incidentlight fields. In order to relight real objects with 4D illumination, we will follow thegeneral image-based method 2.4.4 (on page 23). This method consists of three steps,which are discussed for relighting with 4D illumination in the following three sec-tions: acquisition of the data (section 4.2), construction of the reflectance functions(section 4.3) and the actual relighting (section 4.4).

In section 4.5, an improvement of the basic data acquisition method is discussed:an elegant way to speed up the data acquisition process while using the same dataacquisition setup.

Page 91: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.2. DATA ACQUISITION 79

4.2 Data Acquisition

To acquire the data for the reflectance function construction, illumination conditionsC1,C2, ...,CN need to be chosen. For each of these illumination conditions Ci, animage Imgi is then to be taken of the object illuminated with that condition. First, wewill define algebraic illumination conditions. Then we create a setup to create theseillumination conditions on the object in order to record images of the object underthese conditions.

4.2.1 Illumination Conditions

Since the reflectance functions are dependent on the position as well as the directionof the incident illumination, the illumination conditions will be dependent on boththese parameters (i.e. position and direction). Therefore, in this case, the illuminationconditions are incident light fields Ci(p← ω), with p a point on the surface S, and ωa direction of Ω.

When using incident illumination dependent only on the direction, sampling a setof N directions returns a set of illumination conditions. The resulting set of basisimages allows to reconstruct the 2D reflectance functions accurately, especially if thepixel does not represent a specular or glossy surface. In case of a glossy surface,using more directions, and thus more illumination conditions, will reduce the error. Incase of perfectly specular objects, the sampling technique will not work (i.e. a Diracfunction cannot be constructed from a discrete set of samples), and therefore, specularobjects can not be relit with this approach. In figure 3.8 on page 53, a chrome ball wasused to demonstrate this. In general, the use of few illumination conditions will only“blur” the illumination, but not the image of the object.

We can try to apply this sampling approach for 4D illumination as well. The 4Dillumination space can be decomposed in two 2D spaces: the space of positions onsurface S and the direction space Ω. Sampling in the 4D illumination space is equalto a couple of samples from both spaces. Suppose a set of couples of samples ofthese two spaces (p1← ω1), (p2←ω2), ..., (pN←ωN) is chosen and basis imagesare recorded of the object with these illumination conditions. In this case, it can notbe assumed that each visible part of the object will be illuminated at least once inthe resulting set of basis images. Therefore, it can not be guaranteed that for eachpixel there is enough information to construct a reflectance function. In an extremesituation, the irradiance for a pixel can be zero for each basis image. However, if thesurface that such a pixel represents is directly illuminated, the pixel will surely havesome resulting irradiance. Therefore, we cannot merely conclude that this pixel isalways zero for any incident illumination, based on the chosen illumination conditionsand the resulting basis images. Solutions to this problem are:

• Using more samples. This will not resolve this situation since it can never beguaranteed that each position on the object is illuminated at least once, no matterhow many samples. This is a similar situation as for perfectly specular materialswhen relighting with only directionally dependent illumination.

Page 92: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

80 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

• Combine measured information of neighboring pixels. Although this couldsolve the problem to some extent, this solution needs to be rejected as well,since the resulting relit images would be blurred, due to information interchangeamong pixels.

• Ensure that all parts of the object are illuminated from sampled positions, byusing sets of directions instead of sampled directions. This is the method weused in our work.

We can sample a set of P positions p1,p2, ...,pP, on the surface S. We subdividethe space of directions Ω into M disjunct partitions: Ω1,Ω2, ...,ΩM. The union of allthese partitions Ωi cover Ω completely as defined in 3.4 (page 49). The N (= P×M)illumination conditions C1, C2, ..., CN can then be defined as C1,1, C2,1, ..., CP,1,C1,2, C2,2, ..., CP,2, ..., C1,M, C2,M, ..., CP,M:

∀i, j 1≤ i ≤ P, 1≤ j ≤M : Ci, j(p← ω) =

1 if p = pi and ω ∈Ω j

0 otherwise.(4.1)

For position p, we only need to consider the set of directions that are directedtowards the object and we call all these directions the set of directions Ωp. On theleft of figure 4.2, all directions in Ω are displayed, in the middle, only the directionsimportant to us are displayed. As one can clearly see, the important directions are asubset of all directions. therefore, instead of subdividing Ω into M partitions, we cansubdivide Ωp into M partitions, called Ωp

1 , Ωp2 , ... Ωp

M .

Figure 4.2: On the left, all directions for a position p are represented. In the middle,only the directions towards the object for position p are displayed. At the right, alldirections within the projection cone (dashed lines) of a projector positioned at pointp towards the object are displayed. As can clearly be seen, all important directions areincluded in the projector cone.

The illumination conditions are then defined as:

∀i, j 1≤ i ≤ P, 1≤ j ≤M : Ci, j(p← ω) =

1 if p = pi and ω ∈Ωpij

0 otherwise.(4.2)

Page 93: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.2. DATA ACQUISITION 81

These sets Ωpij actually define a set of disjunct box functions BOXpi

j (ω) on thedomain Ωpi and can be extended to the complete domain Ω. In order for the boxfunctions to be defined over Ω, they require to return a value for any direction ω in Ω.Therefore, we choose to assign the value 0 to all directions outside Ωpi . For directionsinside Ωpi , the values are defined by the subsets Ωpi

j in Ωpi . The box functions can bewritten as:

BOXpij (ω) =

1Ωpi

jif ω ∈Ωpi

j

0 otherwise.

We can now easily express these illumination conditions of equation 4.2 with the boxfunctions BOXpi

j (ω) as:

∀i, j 1≤ i ≤ P, 1≤ j ≤M : Ci, j(p← ω) =

BOXpij (ω) with p = pi

0 otherwise.

The illumination condition Ci, j corresponds to radiance originating from point piand the radiance value for each direction ω depends on the function BOX pi(Ω).

4.2.2 Data Acquisition SetupIt is clear that a standard Light Stage setup cannot be used to recreate these illumina-tion conditions, since a standard Light Stage is only able to illuminate an object fromdifferent directions. However, in most practical Light Stage setups, a common lightsource (e.g. light bulb) is used to illuminate the object from different positions on asphere around the object. By allowing only to measure objects that are very small incomparison to the distance to the light sources, the object is illuminated by an approx-imation of a directional light source. Therefore, a Light Stage setup is only able toilluminate an object from different positions on a sphere.

Our illumination conditions require to illuminate the object from a set of positionspi on a hemisphere (surface S) and for each of these positions pi into a set of solidangles Ωpi

j . A common light source emits light in all directions at once and can not berestricted to emit light in a chosen solid angle. Instead of using a common light source,an LCD-projector can be used as light source in a Light Stage setup. This allows toilluminate the object from different positions on a hemisphere and for each position indifferent sets of solid angles. However, a projector can not emit light in only directionsΩpi but is only able to emit light in a fixed range of directions (i.e. the projection cone,also known as the angular range of the projector). By choosing a small object that fitsinto the projection cone of the projector at position p, all directions of Ωp fall into theprojector range (figure 4.2).

Since the angular range of the projector is a superset of all directions of Ωp andwe know the object fits in this angular range, we extend Ωp to the complete angular

Page 94: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

82 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

range of the projector. Now, we choose to subdivide the angular projector range, nowcalled Ωp, into the M partitions Ωp

1 , Ωp2 , ..., Ωp

M. The illumination condition Ci, jthus corresponds to light originating from position pi in all directions defined by Ωpi

j .

The subdivision of Ωp in j sets Ωpj , can be directly related to the projected image

of the projector. We choose to subdivide the projected image of the projector intoa m×m(= M) grid of square patterns. Each square pattern thus defines a partitionof directions Ωp

j . Note that by subdividing the projected image of the projector, theresulting subdivision of Ωp into M partitions is disjunct and exhaustive and complieswith the requirements. Therefore, the illumination condition Ci, j corresponds to lightoriginating from the projector at position pi, emitting the jth white square pattern.If only a single white pattern is used, covering the complete angular range of theprojector, this setup results in a similar data acquisition as a Light Stage. This isillustrated at the top of figure 4.4.

Projector

Camera

Turntable

Figure 4.3: The setup for the data acquisition for relighting with 4D illumination of areal object (outlined in blue). The projector (outlined in red), moves on a circular pathover the object. The object itself is positioned on a turntable, to which the camera isattached (outlined in green).

In order to acquire the data, we need to sample a set of P points pi on surface S,which is a hemisphere. The projector then needs to be positioned with the apex ofits projection cone at each of these points pi and its angular projection range needs tocover the complete object. Therefore, the projector does not only need to be movedto each position, but must be oriented correctly in order for its angular range to coverthe object. In practice, we use an LCD projector mounted on a movable gantry, andthe object and camera are placed on a turntable. The gantry is able to tilt the projectorfrom 0 to 90 degrees, while the center of projection remains the center of the turntable.The turntable is able to rotate the full 360 degrees (Figure 4.3). The combination ofboth movements allows us to place the projector at any position on the hemisphere

Page 95: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.2. DATA ACQUISITION 83

relative to the object, thus at any sampled position pi, while the center of the turn tableis always in the angular range of the projector.

For each of the sampled positions pi, the projector emits a series of m×m whitesquare patterns of constant intensity, corresponding to Ωpi

j . Each bundle thus pro-jected, accounts for a single illumination condition of the object. Figure 4.4 gives aschematic overview of the acquisition.

Figure 4.4: The topmost picture shows the object illuminated by the projector emittingwhite light over its entire angular range. This would be similar to data acquisition bythe Light Stage. For relighting with 4D illumination, the projector emits various whitesquare patterns, resulting in a partially illuminated object.

With this setup, the illumination conditions can be recreated and the object can berecorded while being illuminated with these illumination conditions.

In our setup, we positioned the projector at 7 regularly spaced tilt angles (i.e. tiltedthe gantry with projector in 7 steps) and for each tilt angle moved the projector over32 regularly spaced azimuth angles (i.e. rotated the turn table in 32 steps). This resultsin P equal to 224 (= 32×7). For each projector position we used a grid of 16×16 orM equal to 256 patterns.

Note the requirement stated for the data acquisition process: the object size isconstrained by the angular range of the projector. Objects need to be small enough

Page 96: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

84 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

to fit in the angular range of the projector for all sampled positions. If this is notthe case, a larger gantry must be used, in order to move the projector away from theobject, allowing a larger object size to be measured. Alternatively, a lens needs to bemounted on the projector to widen the angular range.

4.3 Reflectance Function ConstructionThe reflectance functions need to be defined for the complete 4D illumination space,using the illumination conditions Ci, j and the corresponding basis images Imgi, j . Thiscorresponds to constructing reflectance functions defined over both the complete sur-face S (2D) and the space of directions Ω (2D).

When fixing the reflectance function to a point pi, an approximation of the re-flectance function Rx,y(pi← ω) can be easily found in the following way:

In section 2.3.2, we showed that a function f (x) can be approximated as a sumof weighted basis functions where the weights are the orthogonal projections of thefunction f (x) onto these basis functions. We can approximate the reflectance functionRx,y(pi← ω) with basis functions BOXpi

j (ω) as:

Rx,y(pi← ω)≈M

∑j=1

(w j×BOXpi

j (ω)). (4.3)

where the weights w j are the projection of Rx,y(pi←ω) on the dual basis functions ofBOXpi

j (ω). Since the set of functions BOXpij (ω) is an orthonormal set of functions,

the function BOXpij (ω) is its own dual function.

Due to the data acquisition method, the basis image Imgi, j is the result of recordingan image of the object illuminated with condition Ci, j. Since illumination conditionCi, j corresponds to a function BOXpi

j , the basis image Imgi, j is the projection offunction BOXpi

j on the true reflectance field of the object, or denoted per pixel:

Imgi, j(x,y) =

Z

ΩRx,y(pi← ω) BOXpi

j (ω) dω. (4.4)

Therefore, w j (in equation 4.3) is equal to Imgi, j(x,y). Substituting w j by Imgi, j(x,y)in that equation results in:

Rx,y(pi← ω)≈M

∑j=1

(Imgi, j(x,y)×BOXpi

j (ω)).

Note that the reflectance function for such a pixel is only an approximation of thereal reflectance function, and thus contains some error compared to the real reflectancefunction. This error owes to the fact that the real reflectance function for a single se-lected point is not piecewise constant in the angular domain, but should be defined forevery direction separately. However, due to the choice of the illumination conditionsas a set of box functions in the angular domain, the reconstructed reflectance functionsare an approximation of the real reflectance functions.

Page 97: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.4. RELIGHTING 85

Since only a discrete set of P positions were used during the data acquisition, thereflectance function at non-sampled positions needs to be based on the measured data.We construct a Voronoi diagram on the surface S, using the P sampled positions pias Voronoi sites. This results in P Voronoi cells, which we call Spi , corresponding toposition pi. The union of these Voronoi cells is the complete surface S. We define thevalue for the reflectance function for all positions in Spi to be equal to the reflectancefunction value for the measured position pi. Therefore, the reconstructed reflectancefunction can be defined as:

Rx,y(p← ω)≈M

∑j=1

(Imgi, j(x,y)×BOXpi

j (ω))

for all p ∈ Spi . (4.5)

Note that the reflectance functions have a value over the complete 4D illuminationspace, albeit the value 0 for some portion of this illumination space. This portioncorresponds to illumination that is assumed to have no influence on the reflectance ofthe object, since this illumination is not directed towards the object.

4.4 RelightingOnce the reflectance functions are obtained, the object can easily be visualized withnovel incident illumination Lin(p← ω), using the relighting equation for a pixel (re-capitulation of equation 2.11):

E(x,y) =

Z

S

Z

ΩRx,y(p← ω) Lin(p← ω) dω dp . (4.6)

Relighting as a Weighted Sum of Basis ImagesEquation 4.6 states that the irradiance for pixel (x,y) is the inner product of functionsRx,y(p←ω) and Lin(p←ω). In section 2.3.3, we demonstrated that the inner productof two functions can be approximated with a sum of products of coefficients and thus:

E(x,y)≈P

∑i=1

M

∑j=1

wi, j(x,y)×w′i, j(x,y),

where the coefficients wi, j and w′i, j are respectively the projections of the functionsLin(p← ω) and Rx,y(p← ω) onto the orthonormal basis functions. If we assumethe set of box functions BOXpi

j , used as illumination conditions Ci, j, to be set ofnormalized box functions, this set of functions is a set of orthonormal basis functions.The coefficients wi, j and w′i, j can then be calculated as follows:

• Coefficients w′i, j can be found by projecting the function Rx,y(pi← ω) onto thebasis functions BOXpi

j :

w′i, j(x,y) =

Z

S

Z

ΩRx,y(p← ω) BOXpi

j (ω) dω dp.

Page 98: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

86 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

Since the function BOXpij (ω) returns value 0 for points p 6= pi for any i we can

write:

w′i, j(x,y) =

Z

ΩRx,y(pi← ω) BOXpi

j (ω) dω.

Equation 4.4 states that the recorded irradiance value for pixel (x,y) in the basisimage Imgi, j is in fact this projection of the reflectance function Rx,y(p← ω)

on the function BOXpij and thus,

w′i, j(x,y) = Imgi, j(x,y).

• The coefficients wi, j can also be found by projecting the function Lin(p← ω)on the basis functions BOXpi

j :

wi, j(x,y) =

Z

S

Z

ΩLin(p← ω) BOXpi

j (ω) dω dp

=

Z

Spi

Z

Ωpij

Lin(p← ω) dω dp.

With these weights we can write the irradiance of a pixel as:

E(x,y)≈P

∑i=1

M

∑j=1

wi, j(x,y)× Imgi, j(x,y).

Since wi, j = wi, j(x,y) for any pixel position (x,y), we can generalize this to full im-ages and write:

E ≈P

∑i=1

M

∑j=1

wi, j× Imgi, j, (4.7)

and thus, one can also define this relighting technique as a linear combination of basisimages, similar to relighting with 2D illumination (section 3.2.1).

4.4.1 ResultsFigure 4.5 depicts the two toy monkeys, we have been using all along in this chapter.The objects are placed on black velvet since only the illumination effects on the objectare of interest. On the left of this figure, the illumination is neutral (i.e. white lightoriginating from all positions and directions) in order to show the objects without anyillumination effects. For this scene, 224 projector positions were sampled regularlyin a latitude-longitude parameterization on the hemisphere S, and for each projectorposition, 256 white square patterns were projected (i.e. cells of a 16×16 grid). Thisrequired us to record 57344 basis images. On the right of figure 4.5, the object is

Page 99: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.4. RELIGHTING 87

Figure 4.5: On the left, two toy monkeys lit with neutral illumination. On the right,the toy monkeys are illuminated with a red and blue colored spot light.

Figure 4.6: The toy monkeys illuminated with an artificial lighting environment ofthree colored spotlights: red, green and blue. On the left, the toy monkeys are illumi-nated with 4D illumination, on the right, only 2D illumination incident at the centerof the environment is used.

Figure 4.7: The two monkeys illuminated with real world captured illumination usingthe techniques of [55]. On the left, the complete incident light field was used, whileon the right, only a 2D subset of the illumination was used.

Page 100: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

88 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

Figure 4.8: An arrangement of chess pieces. On the top left, the scene is lit withneutral illumination. On the top right, three different colored spot lights illuminate thescene. The bottom left demonstrates that light incident at grazing angles illuminatesthe top of some chess pieces but not the bottom. At the bottom right, artificial ’ring’illumination is used.

illuminated with two colored spot lights: the brown monkey is illuminated with a redspot light, while the white monkey is illuminated with a blue spotlight.

In order to demonstrate the need for relighting with 4D illumination, we createda virtual environment in which three spotlights are the only source of illumination.A green spot light illuminates the back of the objects placed in this environment,while a red and blue spot light illuminate the object from the front left and frontright, respectively. On the left of figure 4.6, our two toy monkeys are illuminatedwith this artificial lighting environment. The green spot clearly illuminates the backof the brown monkey, while not illuminating the white monkey. Similar effects arenoticeable for the other two spot lights. We also synthesized an environment map inthe center of our lighting environment and applied this 2D illumination on the twomonkeys. The resulting image is displayed on the right of figure 4.6. While the imagelooks visually pleasing, all positional illumination effects are completely lost. Forexample, the green spot light in the back now illuminates both toy monkeys.

As noted in chapter 2, there exist several methods to capture incident light fieldsof real world environments. In [55], this is done by acquiring multiple environmentmaps at different positions. Jonas Unger was so kind to let us use his acquired dataof different real world incident light fields. One such incident light field represents

Page 101: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.5. DATA ACQUISITION SPEEDUP 89

the illumination of a spot light. We applied this illumination to the toy monkeys andthe resulting image can be seen on the left of figure 4.7. One can clearly see the spotlight effect on the monkeys, since only a part of them is illuminated, while everythingoutside of the spot light beam is dark. We also illuminated the object with only a singleenvironment map of the recorded data of the incident light field and the resulting imageis displayed on the right of figure 4.7. Again, all positional illumination information islost, which results in an image of the object being illuminated with a directional lightsource.

At the top left of figure 4.8, an arrangement of chess pieces is displayed on awooden surface. This scene is captured with the same projector position resolutionas the scene with the two monkeys, but only a fourth of the hemisphere was sampled(i.e. 56 projector positions). The patterns for this scene are based on a grid of 32×32 white squares, resulting in 56× 1024 or 57344 basis images. This is the sameamount of basis images as for the two toy monkeys, but the illumination is restrictedto originate from positions of the sampled quarter of the hemisphere. However, theangular resolution is quadrupled. Note at the top left of figure 4.8 that the two cornersof the image are darker, since these places on the wooden plate lie outside of theprojection cone for some projector positions.

On the top right of figure 4.8, three different colored spotlights illuminate thevarious chess pieces. Due to the wooden surface, the spotlight effects can clearly beseen. At the bottom right, a wide spotlight illuminates the scene from a grazing angle.This results in the chess pieces on the right being only illuminated at the top. On thebottom right, an artificial lighting condition is used to illuminate the scene, resultingin a ’ring’-like lighting effect.

4.5 Data Acquisition Speedup

Although the previous sections provide a robust method to relight real objects with 4Dillumination, the technique is not very practical: the number of basis images is verylarge. This calls for improvements of the technique. In this section we propose a dataacquisition speedup. This speedup significantly reduces the number of images to berecorded, while using the same data acquisition setup.

Capturing the basis images for a single object in the way we presented it requires toplace the projector at P positions. For each position, M patterns need to be projected,while the resulting illuminated object is recorded. This calls for P×M basis images.In the figures on page 87 we used 224× 256 = 57344 basis images, similar to thescene in figure 4.8 (56×1024 = 57344 basis images).

In case M = 1, the projector emits white light over the entire angular range andthe the data acquisition process is similar to a Light Stage. Therefore, the numberof projector positions P can only be reduced when the object is fairly diffuse. Thus,P depends on the kind of object and cannot be reduced at will. We will reduce theamount of patterns M per projector position. In our setup, these M patterns are whitesquare patterns, tiled in a grid. Suppose this grid has a resolution of m by m, thenM = m×m.

Page 102: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

90 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

During the data acquisition for a particular projector position pi, directions nottowards the object (i.e. directions outside Ωpi) are omitted, since these directionsdo not contribute to the reflectance of the object. This notion can be taken a stepfurther. In our proposed method, the reduction of M is based on the assumption that thebundles of light have a limited, local influence on the object and not a global influenceon the complete object. This means that illuminating the object with a narrow bundleof light at one side, will not have an influence on the other side of the object and thus,this other side of the object remains dark, as if the object would not be illuminated atall. With this assumption, we can emit more than one square pattern for each recordedimage, as long as the effect of the different patterns does not overlap. After the imagerecording, the images of each individual square pattern can be reconstructed using theimages of the multiple emitted square patterns.

4.5.1 Basic Speedup Technique

We create a horizontal set of patterns Hi as shown on the left of figure 4.9. Each Hiis constructed in such a way that each individual square pattern does not influenceanother square pattern in the same horizontal pattern Hi, once projected. A similar set,the vertical set of patterns Vi is created as well, as depicted on the right of figure 4.9.An original square pattern occurs twice: once in a horizontal set Hx and once in avertical set Vy. By construction we know that all other squares in the pattern do notinfluence this pattern and that the intersection of Hx and Vy equals that pattern.

...

...

Figure 4.9: On the left, the set of horizontal patterns is shown, on the right, the setof vertical patterns. Each set of patterns is split up into 3 sets, in such a way that thepatterns are spaced out evenly.

The basis image corresponding to the projection of an individual square pattern atgrid position (x,y) can be reconstructed from the recorded images, by taking for eachpixel the minimum of the corresponding pixel values in the images resulting fromprojecting Hx and Vy. This reconstructed image is an approximation of the photographresulting from emitting pattern at grid position (x,y). This technique works becauseof the assumption that two different squares do not interfere with each other. In 4.10,an illustration of this technique is given.

Page 103: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.5. DATA ACQUISITION SPEEDUP 91

Figure 4.10: An illustration of the use of patterns on the two toy monkeys. On top, onevertical and one horizontal pattern are projected and the resulting images are shown.At the bottom, we project a pattern with one white square and the resulting image isdisplayed. This resulting image at the bottom is approximated by taking the minimumper pixel of the two top images which are the result of the projection of the horizontaland vertical patterns above.

4.5.2 Influence ParameterWe assume that a square pattern has only local influence and that the square patterns ineach horizontal and vertical pattern do not interfere with each other once projected. Ifthis assumption is not met, this speedup technique will fail. Therefore, the choice forthe horizontal patterns Hi and vertical patterns Vi depends greatly on this assumption.

The influence of a light bundle can be large or very small. Both extremes arepossible:

• If a single light bundle has an influence on the complete object, the horizontaland vertical patterns cannot have multiple squares and thus, no reduction ispossible. For example, if the object were a diffuse concave bowl pointed towardthe camera, there would be a lot of inter-reflection in the bowl for any lightbundle. In this case, no speedup can be achieved.

• If, on the other hand, each bundle has only an influence on the parts of the objectdirectly illuminated by the light bundle, then the set of horizontal patterns canbe chosen to be m white banners of all connected white patterns in a row, and theset of vertical white patterns can consist of m white columns of white squares. Inthis case, the speedup is spectacular: instead of m2 patterns, only 2×m patternsare required.

For most objects, the influence is more than the directly illuminated portions of theobject, but less than the complete object. We use a distance called d, that states at least

Page 104: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

92 CHAPTER 4. RELIGHTING WITH 4D ILLUMINATION

how many squares must lie between two squares in a pattern, in order to have no in-terference between the two squares. For example, if the distance d is 0, then no squaremust lay between two squares in a pattern, and thus the light bundles have no influ-ence on the object except for the portion of the object directly illuminated. If d is 1,then at least one square must be skipped between two squares in a pattern, otherwise,the light bundles corresponding to the squares will interfere. In general, the larger thedistance d, the more influence each bundle has. If d is large, few squares can lie in ahorizontal or vertical pattern and thus more patterns are needed in the horizontal andvertical pattern sets. In figure 4.11, the different values for d are displayed with thecorresponding patterns for the second row of the horizontal patterns. In general, thefollowing rule can be deduced: with a distance of d, at least 2×m× (d + 1) patternsare needed.

...

...

1

2

m−1

0

Figure 4.11: The second row of the horizontal patterns, for different values of distanced. If d is 0, only a single pattern is needed, is d is larger, each square has a largerinfluence, but more patterns are required.

This distance enables us to choose the amount of allowed interference at the timeof data acquisition and the reduction in number of images to be recorded. If we onlyallow direct illumination, then d can be chosen to be 0 and only 2×m images needto be recorded, but all inter-reflection effects are lost. However, if we assume a lightbundle has influence on the whole object, all inter-reflection effects are kept but dmust be set to m− 1 and m2 images need to be recorded. Therefore, the choice of dis a trade-off between reduction in images to record and inter-reflection effects in theresulting images.

In general we can reduce the complexity of our data acquisition from O(P×M) orO(P×m2) to O(P×m). Therefore, the reduction in number of images to be recordedis more significant for large values of m. In our implementation, we split up the setsof patterns for each row or column into 3 separate sets (i.e. d = 2). This reducedthe number of photographs per projector for the scene with the toy monkeys from256 (= 16× 16) to 96 (= 2× 16× (2 + 1)). In total, 21504 images needed to be

Page 105: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

4.6. CONCLUSION 93

taken, almost a reduction of 1 : 3. The speedup is even better for the scene with thearrangement of chess pieces, where instead of 1024 (= 32× 32), only 256 imagesper projector needed to be recorded. And thus, 14336 images needed to be recorded,instead of recording 57344, a reduction of 1 : 4.

4.6 ConclusionBased on the general image-based relighting method, a practical relighting techniquehas been proposed allowing to visualize real objects with novel 4D illumination.

First, a basic set of illumination conditions was chosen and a practical data acqui-sition setup was discussed, allowing to recreate these illumination conditions. Fromthe recorded images using this setup, a straightforwardreflection function reconstruc-tion technique was deduced. Relighting with these reflectance functions correspondsdirectly with the linear combination of the recorded basis images.

In a second step, this basic relighting method is improved to make it more prac-tical: the amount of images to be recorded is reduced in order to speed up the dataacquisition. By choosing the influence parameter to be small, a significant reductionin the number of required images can be achieved. However, this also requires the in-fluence of the separate light bundles to be local, otherwise, the underlying assumptionswill not be met and errors will be made in the relighting.

Page 106: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 107: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 5

2D Reflectance Functions

Relighting techniques using 2D illumination have been surveyed in chapter 3. Thesetechniques mainly focused on the data acquisition methods and used straightforwardtechniques to construct reflectance functions from the acquired data.

In this chapter, we focus on the processing of the acquired data while using the dataacquisition methods of chapter 3. By using more intelligent reconstruction techniquesfor the reflectance functions, better images of the objects with novel illumination willbe produced while the same data acquisition methods are used. In order to limit thedata storage space and speed up the relighting, efficient compression techniques forthe reflectance functions are discussed as well.

5.1 Introduction

In the first part of chapter 3, we sampled the reflectance functions by using direc-tional light sources as illumination conditions. This resulted in a set of basis imageseach corresponding to an illuminant direction. Using this data, a complete reflectancefunction for each pixel was created, describing the reflectance of that pixel for eachincident direction. This reconstruction was performed using box functions. However,there are other signal reconstruction methods performing better than box functions. Insection 5.2, we will survey the effect of different reconstruction techniques for sam-pled 2D reflectance functions and we compare these techniques in section 5.3. Sincethe resulting 2D reflectance functions are algebraic formulas describing the reflectancefor each pixel, we discuss possible compact representations of the reconstructed 2Dreflectance functions (section 5.4).

5.2 2D Reconstruction Techniques

In chapter 3, the reflectance function for relighting with directionally varying incidentillumination in pixel (x,y) is defined as:

95

Page 108: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

96 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Rx,y(p← ω) = Rx,y(ω),

and thus the function only depends on the direction ω. In the same chapter, tech-niques were discussed to capture samples of these reflectance functions at a set ofregular selected directions (section 3.2.1) or, more generally, from a set of arbitraryselected directions (section 3.2.2). Both techniques return, for pixel (x,y), the pixelvalues Img1(x,y),Img2(x,y), ...,ImgN(x,y), which are the measured values of thereflectance function Rx,y at directions ω1,ω2, ...,ωN.

From this set of N measured values Imgi(x,y) at selected directions ωi, the re-flectance function Rx,y(ω) on the complete domain Ω needs to be reconstructed. Thisproblem can be seen as a signal reconstruction problem, and several reconstructiontechniques can be applied for this task. In our work, several techniques were analyzedto create the reflectance function on a hemisphere from the discrete set of samples,expressed as algebraic formulas. These techniques do not assume any structure in theselected set of directions ωi. In the following subsections we review these reconstruc-tion techniques.

To demonstrate the different reconstruction techniques, we use the miniature racecar from previous chapters. This object was chosen because it contains many specularsurfaces and fine geometrical details, causing numerous self-occlusion features. Thereflectance functions of this object were sampled from 1280 directions. Four pixelswere selected and the corresponding sampled reflectance functions reconstructed withthe Voronoi-based box functions as discussed in section 3.2.2, are displayed in fig-ure 5.1. To visualize the reflectance functions, defined on a hemisphere, we use theprojected disk parameterization (section 2.5.1).

Note that in this example, the directions are regularly sampled in latitude-longitudeparameterization, since a standard Light Stage was used. However, irregularly sam-pled directions and measured values can be used (captured with a Free-form LightStage) for the discussed reconstruction techniques as well, unless noted otherwise.

The four reflectance functions include a wide range of features: a diffuse red pixelon the side of the driver’s helmet, as can be seen in figure 5.1 (a) and a yellow pixelon the top of the helmet featuring a soft highlight (b). We also selected two pixelscontaining high-frequency features: a pixel on the side of the car with a sharp highlight(c) and a pixel on the rear of the car complexly occluded by the rear wing (d). Atthe bottom of the reflectance functions, a dark spot can be seen. This is the cameraoccluding the light sources.

In the following sections, only 320 measured values (i.e. 1/4 ) will be used toreconstruct the reflectance functions in the example figures. The reflectance functionsfrom figure 5.1, reconstructed with all 1280 measured values, can then be used asreference to compare the reconstructed reflectance functions with.

5.2.1 Zero-order Hold Reconstruction

In section 3.2.1 and 3.2.2, the reflectance functions are reconstructed using a set ofbox functions. This reconstruction technique is called zero-order hold in signal re-

Page 109: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 97

a b c d

c

bd

a

Figure 5.1: The miniature race car featuring diffuse, glossy and specular materialproperties along with fine detail in geometry. The reflectance functions of this object,depicted at the bottom, are sampled at 1280 illumination directions and reconstructedusing the Voronoi-based box functions technique.

construction techniques, hence we will use this name as well. We recapitulate thistechnique briefly.

The N directions ωi, are plotted on a hemisphere and an angular Voronoi diagramis created using the angles between different directions as a distance measure. Theresulting Voronoi cells are called Ωi and the union of these cells covers the completedomain Ω.

[

i

Ωi = Ω and Ωi\

Ω j = /0 if i 6= j ∀ i, j ∈ 1, ...,N. (5.1)

The reflectance function Rx,y is reconstructed over the domain Ω by assigning themeasured reflectance value Imgi(x,y) of each selected direction ωi to all the directionsin the corresponding Voronoi cell Ωi:

Rx,y(ω) = Imgi(x,y) for all ω ∈Ωi. (5.2)

The resulting reflectance functions are piecewise constant functions, as can be seenin figure 5.2.

Since all reflectance functions are constructed using the angular Voronoi diagrambased on the selected directions ωi, the angular Voronoi diagram can be constructedonce and be re-used for the construction of each reflectance function independently.Therefore, we introduce the function V (ω), defined on Ω as:

Page 110: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

98 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Figure 5.2: The selected reflectance functions reconstructed from 320 samples, usingthe zero-order hold method. The reflectance functions are clearly piecewise constantfunctions.

V (ω) = i for all ω ∈Ωi, (5.3)

representing the angular Voronoi diagram. The reflectance function Rx,y can be easilywritten using this function as:

Rx,y(ω) = ImgV (ω)(x,y). (5.4)

The function V (ω) is a reference to the index of the measured value.If the reflectance functions are to be represented and stored as textures, these re-

flectance function textures can be created very rapidly using a simple Voronoi-texture.This Voronoi-texture, which is a sort of master texture or mould for the reflectancefunction reconstruction, then represents the function V (ω) and each texel in thisVoronoi texture is the index of the site of Voronoi cell to which the texel belongs.The Voronoi texture requires only a single value per texel (i.e. the index). However,each texel can only represent a single Voronoi cell. This can create artefacts if a texelrepresents a small solid angle lying in two Voronoi cells. This can be improved tosome extent by choosing a large resolution for the Voronoi texture.

This Voronoi-texture can then be used to create all the reflectance functions in aswift manner: for each texel of a reflectance function texture, the value of the corre-sponding texel in the Voronoi-texture is looked up. Suppose this texel value is i, thenthe color for the texel in the reflectance function texture is the color Imgi(x,y). TheVoronoi-texture is thus a reference to the index of the measured value.

5.2.2 Interpolation

Instead of using box functions based on an angular Voronoi diagram of the selecteddirections ωi, as in the previous section, linear functions can be used for the recon-struction of the reflectance functions as well. These linear functions will allow tointerpolate between measured values. By interpolating the recorded reflectance val-ues, a C0 continuous reflectance function can be created.

We implemented the two most common interpolation techniques for scattered data:interpolation based on triangulation and inverse distance weighted interpolation.

Page 111: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 99

Interpolation Based on Triangulation

The linear functions can be based on a Delaunay triangulation, constructed on thehemisphere, using the selected directions ωi. Each vertex point in a triangle sectionis weighted using spherical barycentric coordinates. This results in a linear “hat”-function for each selected direction. The hat-function for a direction ωi is defined byall triangle sections that have a point on the hemisphere, corresponding to directionωi, as a vertex, and have a linear fall-off from that vertex to the neighboring vertices.The Delaunay triangulation is the dual of the angular Voronoi diagram and will create“equal” triangles on the hemisphere, resulting in “well constructed” neighborhoodsfor each vertex.

The resulting reflectance functions are depicted in figure 5.3 and they are visuallysmoother than the results obtained with the zero-order hold technique (figure 5.2).

Figure 5.3: The reflectance functions reconstructed from 320 measured values, usingthe linear interpolation based on a Delaunay triangulation. The resulting reflectancefunctions are visually smooth while interpolating the measured data.

Similar as when using a Voronoi diagram, the Delaunay triangulation needs to beconstructed only once and can be re-used for reconstructing every reflectance function.We introduce two functions D(ω,n) and W (ω,n), with ω a direction and defined ondomain Ω and n an element of the set 1,2,3. The function D(ω,n) represents theDelaunay triangulation and returns the nth vertex of the triangle section in which ωfalls. The function W (ω,n) represents the weights for a point and returns the nth

spherical barycentric coordinate of the point defined by ω in the triangle section inwhich ω falls. With these functions, we can write the reflectance function as:

Rx,y(ω) =3

∑i=1

W (ω, i)× ImgD(ω,i)(x,y). (5.5)

Again, if the reflectance functions are to be represented and stored in a texture,the use of master textures can accelerate the production of these reflectance functiontextures. Instead of a single master texture, as with the zero-order hold technique,two three-channel textures are required. These two textures will store the functionsD(ω,n) and W (ω,n), called the vertices-texture and the weights-texture, respectively.

To create a texel for the reflectance function texture for reflectance function Rx,y

at direction ω, the corresponding texel values, v1, v2 and v3 are looked up in thevertices-texture and the corresponding texel values w1, w2 and w3 are looked up inthe weights-texture. The corresponding measured values, Imgv1

(x,y), Imgv2(x,y)

Page 112: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

100 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

and Imgv3(x,y) are then weighted with the values w1, w2 and w3 respectively and

summed together:

Rx,y(ω) =3

∑i=1

wi× Imgvi(x,y).

Inverse Distance Weighted Interpolation

Reconstructing a function or signal from scattered data is often based on InverseWeighted Distance Interpolation (IDW) also called Shepard’s method. This methodcan be used for reconstructing a reflectance function as well. The IDW method usesall the N samples of the function to reconstruct the function in a single direction, asopposed to the previous method that only uses the three closest surrounding sampleddirections. Again, the angle between directions is used as a distance measure. To findthe value of the reflectance function for a direction ωi, the distance (i.e. the angle) toall the directions for which the reflectance function values are measured, is calculated.We call the distance from ω to ωi distance di. The measured reflectance functionvalues are then weighted and added together, with the weights of the values of thesampled directions inversely proportional to the distance to the sampled directions.Therefore, directions that are close to the direction for which the function value is tobe known are given higher weights than directions far away.

Lout(ω) =N

∑i=1

wi× Imgi(x,y), (5.6)

with weights inversely proportional to the distance to the direction, that sum to unity:

wi =di−p

∑Nj=1 d j

−p . (5.7)

with p the power parameter. This p is usually set to 2 but can also be 1 or any otherpositive natural value.

We do not want measured values for opposite directions to contribute to the cal-culation. Therefore, we do not take all measured values into account but only apre-defined number of the nearest neighbors. For a direction ω the M closest mea-sured values, called s1,s2, ...,sM are retrieved and their corresponding distancesd1,d2, ...,dM are calculated. The interpolated reflectance value Lout for ω is:

Lout(ω) =M

∑i=1

wi× si, (5.8)

with

wi =

[dMAX−didMAX di

]p

∑Mj=1

[dMAX−d jdMAX d j

]p , (5.9)

and dMAX = maxdi. In our implementation, p is equal to 2.

Page 113: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 101

The resulting reflectance functions, using Shepard’s reconstruction method, aredepicted in figure 5.4, using M = 32 and p = 2.

Figure 5.4: The selected reflectance functions reconstructed from 320 measured val-ues, using Inverse Distance Weighted interpolation.

Note that using only the 3 closest neighbors (i.e. M = 3) will not result in thesame reflectance function as when using interpolation based on triangulation, sincethe 3 closest neighbors do not necessary form a triangle section in which the directionω lies.

Similar as with previous reconstruction techniques, reflectance function texturescan be created very rapidly using two textures. However, the textures require to haveM channels, which can be a problem if to be implemented on graphics hardware.

5.2.3 Multilevel B-Splines

Bicubic B-Splines can be used to create a smooth continuous function (C2 continu-ous) on the acquired data. However, B-Splines are not easily defined on a hemisphere.Therefore we opt for representing the sampled data using the paraboloid map param-eterization, as proposed by [25] and discussed in section 2.5.1. This representationoffers a continuous projection of the hemisphere, in which the boundaries are iden-tical as on a hemisphere and the pole is defined in a single point. Furthermore, thesolid angle to projected area ratio is close to constant (i.e. the Jacobian is almost theconstant function 1).

A bicubic B-Spline can be fitted on the data by creating a 4×4 grid of controlpoints on the projected data, as can be seen in the first column of figure 5.5. The re-sulting reflectance function is poorly reconstructed because a single bicubic B-Splinecannot represent all features. A possible solution reconstructs the reflectance func-tion using a set of B-Splines, each defined on a different but partially overlapping 4×4grid of control points, which are defined on a lattice over the projected data. Figure 5.5demonstrates the influence of the resolution of the lattice and the resulting reflectancefunction. Using few B-Splines results in a good global fit of the data, but with almostno local detail (figure 5.5, first three columns) while using more B-Splines produces areflectance function with a good local fit but lacking global smoothness. In fact, whenusing too many B-Splines, each measured value will be represented with a singlebicubic B-Spline and in-between measured values no information will be available,resulting in a set of peaks (figure 5.5, last column).

Page 114: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

102 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

...

1 (1 × 1) 64 (8 × 8) 256 (16 × 16) 4096 (64 × 64)Number ofB-Splines

Lattice withsingle 4×4supportmarked

functionreflectanceResulting

Figure 5.5: Influence of number of B-Splines on the coverage of 4×4 support andresulting reflectance function. The red dashed rectangle shows the size of a 4×4support for each lattice.

...

+ + + + +

sω i- R

1 (ω i)

1 2 3 4 5 6

num

bero

f

sω i- R

3 (ω i)

sω i- R

5 (ω i)

sω i- R

4 (ω i)

sω i- R

2 (ω i)

== = = = =

(2 × 2)4 16

(4 × 4)64(8 × 8)

256(16 × 16)

1024(32 × 32)

4096(64 × 64)

B-S

plin

esL

evel

Refl

ecta

nce

func

tion

Rle

vel

s ωi

-Rle

vel−

1 (ω

i)B

-Spl

ines

of

R2R1 R3 R4 R5 R6

Figure 5.6: Construction of the multilevel B-Splines. For each level, the 4×4 grid ofcontrol points is subdivided on which a set of B-Splines is defined. These B-Splinesare fitted on the difference of the already reconstructed set of B-Splines and the mea-sured values sωi . The difference is displayed dark for negative values and bright forpositive values.

Page 115: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 103

Multilevel B-Splines, presented by [31], allow to fit a smooth approximation throughthe projected samples without these problems. Multilevel B-Spline interpolation is ahierarchical method that first tries to fit a set of globally smooth B-Splines through thesampled data each defined on a distinct, yet overlapping 4×4 grid of control pointswith large coverage. In each successive step the number of control points in the gridof each B-Spline is doubled in each direction and a new set of bicubic B-Splines iscreated on the four smaller 4×4 grids of control points. These new B-Splines on thesmaller grids are fitted through the difference of the sum of the already computedB-Splines and the measured reflectance function values, called sωi . The method isdemonstrated in figure 5.6. The hierarchy of B-Spline sets can then be reduced to asingle set of B-Splines defined on the 4×4 grid of control points with the smallestcoverage used (i.e. highest hierarchy level).

In our implementation we used four starting grids, which produce a smooth globalfit of the measured data and we refine to six levels in the hierarchy, which allows fora good local fit. The number of levels in the hierarchy was empirically determinedon the magnitude of differences of the measured values and the values of the sampleddirections in the already constructed reflectance function. Using six levels allows tofit the data values, while not fitting noise on the data. This method results in a setof 64×64 B-Splines. The reconstructed B-Spline reflectance functions can be seen infigure 5.7.

Figure 5.7: The reflectance functions, using the multilevel B-Spline reconstructiontechnique, from 320 measured values. The resulting reflectance functions are smoothand represent all measured features.

5.2.4 Other Reconstruction Methods

Other reconstruction techniques for the reflectance functions have been published,previous to our developed techniques: the use of biquadric polynomials and spheri-cal harmonics. These mathematical tools were introduced to compress measured orrendered reflectance function values. However, as a side effect, the reflectance func-tion value for any direction can be found during the decompression of the data. Thisinspired us to look at another popular compression technique: wavelets. With thewavelet transform, reflectance functions can be reconstructed as well, albeit in ourimplementation only in a restricted case.

Page 116: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

104 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Biquadric Polynomial Fitting

Malzbender et al. [35] introduced a technique to represent illumination dependent tex-tures which are called polynomial texture maps (PTM). These textures are independentof the view-direction but incorporate the foreshortening of incident illumination andself-shadowing effects of the texture. Although this technique was not developed torelight arbitrary objects, but only generally flat objects (e.g. a surface covered withsmall stones), it can still be regarded as a relighting technique. Moreover, it was one ofthe first techniques capable of constructing an algebraic form for reflectance functionsfor each texel.

The main idea is to project the data of each texel onto a disk (see section 2.5.1) andthus ω can be parameterized with u,v on the plane of the disk. A biquadric polynomialis then used to represent the reflectance function, which is written as:

Lout(u,v) = a0u2 + a1v2 + a2uv + a3u + a4v + a5. (5.10)

The values a0, a1, a2, a3, a4 and a5, are unknown and need to be found in orderto obtain the reflectance function at each texel. The N captured data values Imgi(x,y)corresponding to directions ωi allow to create a set of equations. Using an optimiza-tion algorithm (least squares optimization), values for the unknown parameters ai canbe obtained.

The resulting reflectance functions, are C∞ continuous and defined on a global sup-port, containing the whole domain Ω. This reconstruction method significantly blursout the high frequency features such as highlights and self-shadowing boundaries,similar to using a single or a small set of bicubic B-Splines. However, the techniqueallows to represent the reflectance functions in a very compact representation (only 6coefficients).

Results for the selected pixels can be seen in figure 5.8. Using a biquadric poly-nomial can result in negative values for some directions, which are clamped to zero inthe visualization.

Figure 5.8: The reflectance functions resulting from fitting 320 measured values on abiquadric polynomial. The reflectance functions clearly do no represent all measuredfeatures, but only gives the global shape of the reflectance function.

Spherical Harmonics

A function represented in the polar coordinate system (i.e. using latitude-longitude)can be expanded in a series of spherical harmonic basis functions:

Page 117: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 105

f (φ,θ) =∞

∑l=0

l

∑m=−l

al,mYl,m(φ,θ),

where al,m are real constant coefficients with values given by the projection of eachspherical harmonic function onto the function:

al,m =Z 2π

0

Z π

0f (φ,θ)Yl,m(φ,θ)dθ dφ,

The spherical harmonic basis functions are the extension of the Fourier transfor-mation on a sphere and can be written as:

Yl,m(φ,θ) =

Nl,mPl,m(cos(φ))cos(mθ) if m> 0

Nl,0Pl,0(cos(φ))/√

2 if m = 0

Nl,mPl,|m|(cos(φ))sin(|m|θ) if m< 0,

with

Nl,m =

√2l + 1

2π(l−|m|)!(l + |m|)!

,

and

Pl,m(x) = the associated Legendre polynomials.

A function on a sphere can be approximated with only a limited number (L2) ofspherical harmonic basis functions, by breaking off the series:

f (φ,θ) ≈L

∑l=0

l

∑m=−l

al,mYl,m(φ,θ).

A reflectance function can then also be approximated with a linear combination ofspherical harmonic basis functions. The coefficients al,m can be approximated by thesum of the projections of the N measured data values Imgi(x,y), corresponding to thedirection (φi,θi) onto each spherical harmonic basis function Yl,m(φ,θ):

al,m ≈N

∑i=1

Imgi(x,y) Yl,m(φi,θi).

With these coefficients, a continuous approximation of the reflectance functioncan be obtained. However, spherical harmonics are defined on a sphere. This poses aproblem since the reflectance functions are only defined on a hemisphere. We solvedthis by mirroring the data to the other hemisphere, resulting in a set of data over thecomplete sphere without introducting extra discontinuities. Recently, hemisphericalharmonics basis functions have been presented in [21]. Although these functions are

Page 118: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

106 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Figure 5.9: The selected reflectance function, reconstructed from 320 samples, using25, 255 or 625 spherical harmonics coefficients in the first, second and last row re-spectively. Using few spherical harmonics coefficients results in reflectance functionswithout the fine measured details, while using a large number of coefficients results inreflectance functions suffering from Gibbs ringing.

defined on a hemisphere and the data would not have to be mirrored, this would notlead to noticeable improvements for our application.

Wong et al. [59] projected sampled reflectance functions of synthetic scenes onthe first 16 or 25 spherical harmonic basis functions, corresponding to four or fivefrequency bands respectively. The reconstructed reflectance functions using the firstfive bands, are very smooth but do not capture all the details, similarly as using abiquadric polynomial (see figure 5.9 first row).

In the second and last row of figure 5.9, the reflectance functions are depictedusing 225 and 625 spherical harmonics coefficients, for each color channel. The re-constructed reflectance functions are able to capture all features to some extent, butsuffer from severe Gibbs ringing or aliasing artefacts. Negative values in the visual-izations are clamped to zero.

These ringing artefacts can be avoided by using windowed spherical harmon-ics. This windowing corresponds to weighing the coefficients with lower weights forhigher frequency bands and corresponds to blurring the reflectance function. Since wewant to obtain the best possible reflectance function, we will not follow this path.

Page 119: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.2. 2D RECONSTRUCTION TECHNIQUES 107

Wavelet Transform

Wavelets are a flexible tool used in many domains including computer graphics. Agood survey can be found in [54, 50].

A continuous signal can be reconstructed from a sampled signal using wavelets,by repeatedly inserting additional samples halfway between each sample point. Themagnitude of each new sample point is characterized by the scale function of thewavelet. Intuitively this can be seen as composing a signal using the wavelet transformin which the added wavelet coefficients (high frequency features) are set to zero.

A logical choice would be to use spherical wavelets [49] since the reflectancefunctions are defined on a hemisphere. The highly irregularly spaced samples in thespherical domain are a disadvantage and require a resampling step. However whenthe measured values are recorded using a standard Light Stage, the samples are regu-larly spaced in the latitude-longitude parameterization and we can opt for the wavelettransform in the latitude-longitude parameterization. Since this is not the most gen-eral case, we will discuss briefly how the wavelet transform can be used in this specificcase to reconstruct the reflectance functions.

Figure 5.10: The reflectance functions, reconstructed from 320 measured values usingthe wavelet transform. On top, the 5/3 LeGall wavelet, at the bottom, the Daubechies9/7 wavelet was used.

In our work, we opted for two wavelet sets: the first wavelet is the 5/3 LeGall,or Integer 5/3, which is the shortest symmetrical biorthogonal wavelet with two van-ishing moments. Its scaling function is a linear B-Spline. The second wavelet is thewell known 9/7 Daubechies wavelet, which is the shortest symmetrical biorthogonalwavelet of order four, and is by construction a cubic B-Spline. Both wavelets are partof the JPEG2000 standard, and are therefore widely implemented. Traditionally thesewavelets are mirrored around images boundaries. However, in our implementationwe opted for repeating the signal on vertical boundaries, since this fits better to theoriginal spherical domain. We still mirror the wavelets on horizontal boundaries.

Results of using these wavelets to reconstruct the reflectance functions can be seen

Page 120: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

108 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

in figure 5.10. The 5/3 LeGall wavelet gives similar results as linear interpolation.The 9/7 Daubechies wavelet, however, result in a very smooth reflection function,but suffers from aliasing artefacts, which become noticeable when animated incidentillumination is used.

The famous Haar wavelet can be used in this context as well. However, sincethis wavelet is based on a constant function and we use the regularly sampled data inlatitude-longitude parameterization, the Haar wavelet would correspond to the zero-order hold technique of section 5.2.1.

5.3 Comparison for 2D Reconstruction TechniquesIn this section we will compare the discussed reconstruction techniques in three ways.First, we will investigate the mathematical errors of the different reconstruction tech-niques in a numerical comparison. This is done by calculating the differences of thereconstructed reflectance function with a reference reflectance function. Second, weinspect the visual differences of the images of the relit object when using differentreconstruction techniques, since the ultimate goal is to generate an image of the ob-ject with novel incident illumination. Finally, we look at the temporal coherence of aset of relit images, resulting from animated novel incident illumination, for differentreconstruction techniques.

5.3.1 Numerical ComparisonSuppose we reconstructed the reflectance function RN(ω) using N sampling directionsand thus N measured values. Reconstructing a reflectance function with a discrete-to-continuous technique introduces an approximation error defined as:

E2avg =

RΩ (R(ω)−RN(ω))2 dω

RΩ (R(ω))2 dω

, (5.11)

with Eavg the average error over directions on the reconstructed reflectance function.Note that Eavg is actually not the average error, but the relative average error. We usethe relative average error, in oder to be able to compare Eavg of different reflectancefunctions. Since R(ω) is unknown, Eavg cannot be computed exactly. However thiserror can be approximated by sampling an additional large set of M ( N) directionsω1,ω2, ...,ωM. Using the M captured samples, denoted as sωi , E2

avg can be approxi-mated as:

E2avg ≈

∑Mi=1 (sωi −RN(ωi))

2

∑Mi=1 sωi

2. (5.12)

We illuminated the objects from 1280 (= M) regularly sampled directions and takea subset of 320 (= N) samples to estimate the error on each reconstructed reflectancefunction.

Eavg will mainly depend on the used reconstruction technique and the kind ofreflectance function. Most of the reconstruction techniques will perform well if a

Page 121: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.3. COMPARISON FOR 2D RECONSTRUCTION TECHNIQUES 109

pixel represents a diffuse un-occluded surface, thus a low frequency function, andconsequently results in a low value for Eavg. However, the error can be high for somereconstruction methods when the reflectance function features a highlight or complexself-shadowing.

Instead of comparing Eavg for a single reflectance function, we compute the av-erage of Eavg over a set of pixels with similar occlusion and material properties. Wecompare these errors for different reconstruction techniques. We use four sets of pix-els:

• all pixels representing the complete object and close-by ground surface,

• the set of pixels representing diffuse un-occluded surfaces,

• pixels representing diffuse surfaces with complex self-shadowing and

• the pixels located on glossy and specular materials.

The average relative errors Eavg of these four sets are plotted in figure 5.11. Thefour sets are visualized as false color images in which the relevant pixels are shown inwhite.

In general, the error for un-occluded diffuse reflectance functions is low, while re-flectance functions featuring high frequency details result in larger errors. Ramamoor-thi and Hanrahan [48] noted that diffuse unblocked reflectance functions can be repre-sented with the first three spherical harmonics bands, yielding an error of less than 1%.In our experiments, we come to a similar conclusion, having noisy real data (includingcamera occlusion) and using five frequency bands.

Reconstructing the reflectance functions by fitting the data to a biquadric polyno-mial or using spherical harmonic basis functions results in large errors. This is dueto the loss of high frequency features and the use of a set of globally smooth func-tions. Using the zero-order hold reconstruction technique produces a mediocre result.It preserves most features, but also introduces new high frequency features in the re-flectance functions. Our proposed methods, the interpolation methods and the multi-level B-Spline method, have relatively low errors, because the resulting reconstructedreflectance functions are smooth while maintaining most of the features.

The standard deviation on the calculated errors was small for pixels representingdiffuse or occluded surfaces. Pixels representing a specular surface produced a largestandard deviation which was expected due to the high frequency features in the re-flectance functions.

5.3.2 Visual Comparison for a Single ImageAlthough these errors Eavg give a good indication of the performance of the intro-duced reconstruction methods, and the accuracy of the resulting reflectance functions,it still is the effect of incident illumination on the object which is most important inour work, not the reflectance function itself. In figure 5.12, the miniature race car isilluminated by a vertical sliver of light. Using a line of illumination results in imagescontaining both low frequency illumination features such as long soft shadows (along

Page 122: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

110 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

0%

3%

4%

5%

6%

7%

8%

9%

1%

2%

Zero-order Hold Spherical Harmonics

Linear Interpolation

IDW Interpolation

Biquadric Fitting

multilevel B-Spline

5/3 LeGall Wavelet

9/7 Daubechies Wavelet

Completeobject un-occluded

DiffuseSpecularGlossy &

occludedDiffuse

Figure 5.11: The average relative error of the reconstructed reflectance functions forthe object from figure 5.1 for four different sets of pixels using different reconstructiontechniques.

the direction of the line of illumination) and high frequency illumination features suchas very short soft shadows (orthogonal to the line). The reflectance functions werereconstructed using several reconstruction techniques and using all 1280 sampled di-rections.

Fitting the data with a biquadric polynomial or using 25 spherical harmonics co-efficients does not result in a satisfying image. The shadows, especially close to theobject, are distorted and the highlights are completely lost. These aliasing effectsare due to Gibbs ringing. However, these methods require very few coefficients torepresent the data which is of importance for real time relighting.

The zero-order hold, linear interpolation, and multilevel B-Splines reconstructiontechniques deliver a good visual result (IDW interpolation performed almost identi-cally to linear interpolation). The shadows are faithfully recreated, as are the high-

Page 123: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.3. COMPARISON FOR 2D RECONSTRUCTION TECHNIQUES 111

Zero-order Hold

Biquadric fitting

Multilevel B-Spline

Linear Int.

25 SH Coefficients

9/7 Daubechies

Figure 5.12: The miniature race car from figure 5.1, relit with a vertical sliver of light,using different reconstruction techniques for the reflectance functions.

lights. The LeGall wavelet transform reconstruction technique (not shown) performssimilarly as the linear interpolation and the Daubechies wavelet transform suffers fromGibbs ringing in the reflectance functions, which cause disturbing visual effects (e.g.over-accentuated shadow boundaries).

5.3.3 Visual Comparison for a Series of ImagesThe findings concerning the visual error on the relit images correspond well with theconclusions from the mathematical errors. The only difference is that the zero-orderhold reconstruction technique gives almost exactly similar relit images as the interpo-lation techniques or the B-Spline technique, while a larger mathematical reconstruc-tion error is made. However, the reconstruction error made by the zero-order holdtechnique is clearly visible when using animated novel incident illumination.

Since we are working with directionally incident illumination, we can rotate theincident illumination around the object. The incident illumination can then be repre-sented as Lin(ωin, t):

Page 124: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

112 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Lin(ωin, t) = Lin(ωin + tωstep),

where t represents time and ωstep represents the angle over which the incident illu-mination is rotated each unit of time. With this animated incident illumination, timecorresponds with a series of images, each image representing the object being illumi-nated with the incident illumination at time t.

For the incident illumination Lin(ωin), we choose the vertical sliver of light fromfigure 5.12. Using a narrow sliver of illumination results in images containing bothlow frequency illumination features such as long soft shadows (along the direction ofthe line of illumination) and high frequency illumination features such as very shortsoft shadows (orthogonal to the line).

To visualize the temporal coherence of the relit images using animated incidentillumination in a single image, we use a technique called video cubism, as presentedin [19]: a video-clip can be seen as a pile of images chronologically stacked on topof each other. This results in a 3D data structure, with 3 axes: the X axis and Y axisof the images and the time axis. The ith frame of the video-clip corresponds to theintersection of this 3D data structure with the plane parallel to the XY plane that cutsthe time axis at time step i. Instead of intersecting the data structure with a planeparallel to the XY plane, one can also intersect the data structure with a plane parallelto the Y − time plane that cuts the X axis at value x. The resulting image represents thechanges of the xth column of pixels over time. We call such an image a “sweep”. Ontop of figure 5.13, the selected single column of pixels from the image of the miniaturerace car is marked. The animated incident illumination is depicted as well. The rest offigure 5.13 shows the different sweeps for different reconstruction techniques. In thesesweeps, the horizontal axis represents time. Therefore, a sweep that is very smooth inthe horizontal direction (i.e. time direction) corresponds to high temporal coherencein the series of relit images.

As can be clearly seen, the multilevel B-Spline technique delivers a smooth sweep(i.e. no horizontal high frequency features), corresponding to high temporal coher-ence in the resulting animation of the object under varying incident illumination. Theopposite happens using the zero-order hold reconstruction technique, where the sweepis clearly piecewise constant, resulting in a jittered animation of the rotating incidentillumination. The sweep corresponding to the linear interpolation reconstruction, hassome sudden changes when the illumination rotates as well, although these changesare not as severe as with the zero-order hold technique. Similar results are obtainedusing Shepard’s reconstruction technique.

On the other hand, the sweep resulting from using the spherical harmonics recon-struction is very smooth. However, the shadows are not reproduced faithfully as wasalready demonstrated in figure 5.12. A similar smooth sweep resulted when the mea-sured data was fit to a single biquadric polynomial to obtain the reflectance function.

The wavelet transform reconstruction also produces a very smooth sweep, compa-rable with the sweep of the multilevel B-Spline. However as already mentioned in theprevious section, the shadow boundaries are over-accentuated due to Gibbs ringing inthe reconstructed reflectance functions.

Page 125: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.3. COMPARISON FOR 2D RECONSTRUCTION TECHNIQUES 113

360270180900

Linearinterpolation

9/7Daubechies

Zero-orderHold

25 SHcoefficients

MultilevelB-Spline

Time IncidentIllumination

Figure 5.13: On top, the selected column of pixels and the animated incident illumi-nation are displayed. Using the video cubism technique [19], sweeps of this columnare displayed for the varying illumination with different reconstruction techniques.

Page 126: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

114 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

5.4 Compact Representation of 2D Reflectance Func-tions

The reconstructed reflectance functions, resulting from using the techniques surveyedin section 5.2, are originally described with an algebraic expression. To find the re-flectance function value for a certain direction, this algebraic form needs to be evalu-ated, which may require a significant amount of computation time. Moreover, comput-ing the irradiance for a pixel (x,y) of the image of the relit object requires to evaluatethe restricted relighting equation (equation 3.1) :

E(x,y) =Z

Ωin

Rx,y(ωin) Lin(ωin) dωin. (5.13)

The time required to compute this inner product for such a pixel is thus proportionalto the time required to evaluate a reflectance function and the number of samples inLin(ωin).

As discussed in section 2.4.5, this inner product of the functions Rx,y(ω) andLin(ω) can be approximated as:

E(x,y)≈M

∑i=1

rx,yi × li, (5.14)

requiring only a sum of M products. In this approximation, a set of M orthonormal ba-sis functions Bi is used to approximate the functions Rx,y(ω) and Lin(ω) as discussedin section 2.3.2:

Lin(ω)≈M

∑i=1

liBi(ω), (5.15)

Rx,y(ω)≈M

∑i=1

rx,yi Bi(ω), (5.16)

with the coefficients rx,yi and li, the projection of the dual basis functions on the re-

flectance function Rx,y and the incident illumination Lin (equation 2.6).Besides fast computation, these basis functions enable to compactly represent

the reflectance functions, since only the M coefficients need to be stored. We willoverview three types of basis functions, box functions, spherical harmonic basis func-tions and wavelet basis functions and discuss the advantages and limitations of thesebasis functions.

5.4.1 Box functionsA set of disjunct box functions is always an orthogonal set of basis functions. This setcan easily be normalized, resulting in an orthonormal set of basis functions. Such aset of box functions can be defined on the hemisphere, making them a candidate forcompact representation.

Page 127: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.4. COMPACT REPRESENTATION OF 2D REFLECTANCE FUNCTIONS 115

As an example, a set of box functions on the hemisphere can be created usinga set of directions, in a zero-order hold manner (i.e. creating an angular Voronoidiagram based on the M directions results in a set of M disjunct Voronoi cells, Ωi). Asignificant disadvantage of reducing the number of coefficients by this method is theloss in smoothness of the reconstructed reflectance function.

Any set of disjunct box functions can be used, but we choose to use box functionsbased on a Voronoi diagram. This has several advantages:

• The complete domain Ω is covered.

• The box functions are convex sections on the hemisphere.

• The M box functions can be easily defined by a set of M directions.

The box functions are completely defined by a set of M directions ω1,ω2, ...,ωMand the corresponding angular Voronoi diagram. In the next paragraphs, we discusstwo different options for the choice of these M directions.

Regularly Sampled Directions

The M directions can be chosen to be regularly sampled in the latitude-longitude pa-rameterization. The resulting approximated reflectance function will be similar to thereflectance function reconstructed with the zero-order hold reconstruction techniquefrom the values rx,y

i sampled at regular directions ωi. Note that the zero-order hold re-construction technique (section 5.2.1) used measured values sωi instead of the averagevalues rx,y

i on the section Ωi.Due to the use of regularly sampled directions, the resulting approximations of

the reflectance functions will contain a lot of noticeable discontinuities and the erroron the approximation will be high. The middle column of figure 5.14 displays theapproximated reflectance function using only 320(= M) regularly sampled directionsand thus 320 coefficients per color channel to represent the reflectance functions. Ingeneral, this compact representation requires M×3 coefficients: 3 color channels foreach of the M basis functions and no information about the directions since these areregularly sampled.

Note that using regularly sampled directions corresponds to down-sampling thereflectance function expressed in a latitude-longitude parameterized texture. Hence,this can be done extremely efficient.

Structured Importance Sampling Directions

Instead of using M regularly sampled directions, one can also sample the directionsbased on the reflectance functions. A variety of techniques exist to sample an envi-ronment map, based on the illumination represented in the environment map. Thesetechniques range from a basic sampling technique, like [10], to methods introducingquadrature rules like the method of Kolig and Keller [29] and hierarchical methodslike structured importance sampling [2]. Recently, a technique was proposed by Os-tromoukhov et al. [43] achieving similar qualitative results as structured importance

Page 128: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

116 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Importance SampledStructured

Originalreflectance function

Regularly sampled

Reflectance function expressedwith 320 box functions

Figure 5.14: Two original reflectance functions (left) are approximated with 320 boxfunctions defined by regularly sampled directions (middle) and structured importancesampled directions (right).

sampling in only a fraction of a second, allowing real time sampling of changing en-vironment maps.

These sampling techniques can be used to sample reflectance functions as well. Inthis case, the sampling is not guided by the illumination intensity but by the reflectanceintensity. Therefore, these sampling techniques applied to reflectance functions resultin ’good’ directions for approximating the reflectance functions. In our work, we usedthe structured importance sampling technique of [2], to sample 320 (= M) directionson the reflectance functions. Note that each reflectance function will have differentsampled directions. Using these directions as sites for the angular Voronoi diagramresults in approximations of the reflectance functions with very low error. Moreover,as can be seen in the right column of figure 5.14, these structured importance sampleddirections lead to significantly better approximations of the reflectance functions thanregularly sampled directions.

A disadvantage to this approach is that these structured importance sampled direc-tions differ for each reflectance function. Therefore, the set of box functions (or setof sampled directions) needs to be redefined and stored for each reflectance function,resulting in a less compact representation. Since each direction requires two parame-ters, the reflectance function is represented with M× (2 + 3) values: 3 coefficients forthe reflectance (i.e. color) value and 2 coefficients for the direction for each of the Mdirections.

Page 129: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.4. COMPACT REPRESENTATION OF 2D REFLECTANCE FUNCTIONS 117

A second and even more significant disadvantage is the need for a recalculationof the coefficients of the incident illumination for each pixel. In order to reduce theinner-product of the incident illumination with the reflectance function to a sum ofproducts of the corresponding coefficients, the incident illumination and reflectancefunction need to be projected onto the same set of basis functions. Since the basisfunctions differ for each reflectance function, the incident illumination needs to beprojected onto a different set of basis functions, for each pixel. This prohibits a fastrelighting process. A solution would be to cluster pixels with the same or similarsampled directions. However, if too many clusters are used, the relighting will not befast, whereas if too few clusters are used, severe discontinuity features will become aproblem, similar as for regularly sampled directions.

5.4.2 Spherical Harmonic Basis FunctionsSpherical harmonics can be used to reconstruct reflectance functions of a pixel, basedon a set of measured values as was discussed in section 5.2.4. However, the sphericalharmonic basis functions can be used to approximate any reconstructed reflectancefunction. This can be accomplished in a similar manner as the reconstruction of a re-flectance function. The main idea to compactly store this approximation is to use onlythe low frequency bands to represent the data, leaving out high frequency “details”.The number of frequency bands can be chosen and is a trade-off between compactnessand range of the reflectance functions:

• The fewer frequency bands are used to more compact the representation will be.The number of coefficients grows quadratically (i.e. for b frequency bands, b2

coefficients need to be stored).

• The more frequency bands are used, the wider the range of reflectance functionsthat can be approximated correctly with the limited number of frequency bandsused. Ramamoorthi and Hanrahan [48] demonstrated that diffuse un-occludedreflectance functions can be approximated with only the first three bands whilethe error of the approximation is less than 1 %. Using more bands allows to addmore detail in the approximation of the reflectance functions.

There is an end to this trade-off in both ways. On the one hand, a reflectancefunction cannot be approximated with only the first frequency band, since this wouldmean that the function would be constant. On the other hand, an infinite numberof frequency bands will not result in a compact representation. However, not all re-flectance functions can be represented with a limited number of frequency bands. Asignificant drawback of spherical harmonics is Gibbs ringing or aliasing which occursaround high frequency features, such as highlights and self-shadowing boundaries,when using few frequency bands.

There have been attempts to find a balance between compactness and range offunctions that can be approximated: Kautz et al. [28] used the first 15 frequencybands to represent synthetic reflection properties without taking into account self-shadowing effects or inter-reflection (i.e. a BRDF). Sloan et al. [52] extended this

Page 130: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

118 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

225 SH Coefficients25 SH CoefficientsOriginal

Figure 5.15: Two original reflectance functions (left) are approximated with a setof spherical harmonic basis functions. In the middle, only 25 spherical harmoniccoefficients are used, while on the right, 225 coefficients are used.

technique to include self-shadowing for synthetic scenes by using two sets of sphericalharmonics, one set for the reflectance properties and one set for the self-occlusionfeatures. However, this second set of spherical harmonics coefficients can only becalculated correctly if the geometry of the object is known. We are working with realworld data and want to approximate a reflectance function in a compact way, hence,no extra information is available. Approximating the reflectance functions with thefirst 15 spherical harmonic bands results in severe Gibbs ringing artefacts, as can beseen on the right of figure 5.15.

The resulting approximations of the reflectance functions are similar as when re-constructing the reflectance functions directly with spherical harmonics and sufferfrom the same artefacts.

5.4.3 Wavelets

Wavelets are well known for their use in image compression and have also been usedfor representing and compressing incident illumination functions. For example, [40]used a cube-parameterization and non-linear Haar wavelet approximation [15] to re-duce the number of basis functions. This significantly speeds-up the computation ofnew relit images (section 2.4.4.3). However, representing the incident illuminationwith a non-linear approximation does not reduce the storage requirements for the re-flectance functions.

In our work, we use non-linear approximation as well, but on the reflectance func-

Page 131: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.4. COMPACT REPRESENTATION OF 2D REFLECTANCE FUNCTIONS 119

tions. This is a more logical choice, since this allows a fast computation of relit imageswhile the reflectance functions are compactly represented. Furthermore, we do not re-strict our work to the Haar wavelet but discuss the use of higher order 5/3 LeGalland 9/7 Daubechies biorthogonal wavelets as well. These wavelets are part of theJPEG2000 standard and have better properties with respect to image compression [56]than the Haar wavelet.

In order to perform a non-linear approximation with the wavelets, we expressedthe reflectance functions in a latitude-longitude parameterized texture, at a resolutionof 256×64. From this texture, wavelet decomposition and composition is straightfor-ward. To maintain the continuity in the azimuthal direction of the reflectance func-tions, we repeated the texture on both sides. To maintain the continuity at the tiltangle, we mirrored the texture vertically. In figure 5.16 the results of a non-linearapproximation with 256 wavelet coefficients are depicted for the three wavelets.

5/3 LeGall Haar waveletOriginalB−Spline 9/7 Daubechies

256 Coefficients

Figure 5.16: On the left two original reflectance functions. On the right the results ofnon-linear approximation with 256 coefficients and using the 5/3 LeGall wavelet, the9/7 Daubechies and the Haar wavelet.

For the comparison between wavelets and the error measurements we use the scenewe have been using all along in our work: the miniature race car (figure 3.2). It isimportant that the non-linear approximation, for both fixed and variable number ofcoefficients maintains the smoothness of the original reflectance functions as muchas possible. Introducing additional discontinuities will result in visually disturbingfeatures when animating the incident illumination. To visualize both spatial as wellas temporal errors, we use the video cubism technique [19] again and render sweeps,similar as in figure 5.13. In figure 5.17, the sweep of the multilevel B-Spline recon-struction technique is repeated for comparison. As can be seen, non-linear approxima-tion with the Haar wavelet, introduces new disturbing temporal discontinuity features,while the other wavelets do not. The reason is the low number of vanishing momentsin the Haar wavelet.

Page 132: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

120 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

Multilevel

B−Spline

Compressed:

Haar wavelet

Compressed:

Compressed:

wavelet

wavelet

5/3 LeGall

9/7 Daubechies

Figure 5.17: Sweeps, similar as in figure 5.13: on top, the sweep using non-compressed reflectance function. The three other sweeps are computed from com-pressed reflectance functions with different wavelets.

Page 133: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.4. COMPACT REPRESENTATION OF 2D REFLECTANCE FUNCTIONS 121

In figure 5.18 the relative Sobolev H1-error is plotted in function of the number ofwavelet coefficients. The Sobolev H1-error is given by [40] as:

‖ f ‖H1=

(Zf 2(x)+ | ∇ f (x) |2 dx

)1/2

(5.17)

This error is the sum of the error on the function and the error on the function in thegradient direction. The relative L2-error resulted in a simular graph and is thereforeomitted. In general, the Haar wavelet needs more coefficients to achieve the samerelative error. For example, 4096 Haar wavelet coefficients are needed for a ±2%error, as opposed to only 256 5/3 LeGall or 9/7 Daubechies wavelet coefficients. The9/7 Daubechies wavelet slightly outperforms the 5/3 LeGall wavelet in terms of error:for 256 coefficients the Haar wavelet results in a 7.05% relative error, the 5/3 LeGallwavelet in a 2.12% relative error and the 9/7 Daubechies wavelet in a 1.88% relativeerror.

201816141210

0

8642

9/7 Daubechies

Number of coefficients32 64 128 256 512 1024 2048 4096

Perc

entH

1 -err

or

Haar5/3 LeGall

Figure 5.18: Sobolev H1-norm versus the number of wavelet coefficients.

Using the same number of coefficients for each reflectance function in the re-flectance field is not optimal. Low frequency reflectance functions can be compressedusing less coefficients and still result in an acceptable error during relighting. Re-flectance functions with high frequency features will require more coefficients in or-der for the approximation to have a similar error. To determine the number of waveletcoefficients to be used for a specific reflectance function, our algorithm selects thelargest wavelet coefficients such that the error on the L2-norm on the compressed re-flectance function itself is bounded by an error-threshold. This L2-norm can be easilycomputed by decomposing the reflectance function into the primal and dual waveletspace and summing the multiplied corresponding coefficients (i.e. compute the selfinner-product). The L2-norm is sufficient for determining the number of wavelet co-efficients, although it is only an indication of the error on the inner product of thereflectance function and the (at processing time) unknown incident illumination.

See figure 5.19 for a comparison of the Sobolev H1-norm versus different error-threshold values and the respective number of wavelet coefficients. The correlationbetween the number of coefficients for a specific wavelet and the error threshold is

Page 134: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

122 CHAPTER 5. 2D REFLECTANCE FUNCTIONS

hard to predict. When using a variable number of coefficients, more wavelet coeffi-cients are assigned to high-detail reflectance functions, whereas low-detail functionsare compressed using less wavelet coefficients. These details contribute little to theerror, hence the small difference in error between compressing reflectance functionsusing a variable number of wavelet coefficients and using a fixed number of waveletcoefficients.

0

7

98

654321

1413121110

316 481 702 992 1346 1766 2285203364 588 908 1356 1923 2620 3460 437888 174 327 582 976 1548 2246 3047

Average number of coefficients per wavelet

9/7 Daubechies5/3 LeGallHaar

Perc

entH

1 -err

or

Inner product error-threshold0.8 0.4 0.2 0.1 0.05 0.025 0.0060.012

Figure 5.19: Sobolev H1-norm versus the error-threshold on the inner product for non-linear approximation using a variable number of coefficients per reflectance function.

In terms of error and smoothness the 9/7 Daubechies wavelet is preferred (fol-lowed by the 5/3 LeGall wavelet), achieving a compression ratio of 1 : 34 for a 0.2%error-threshold and a H1-error of less than 1% on a 256×64 discretization of a mul-tilevel B-Spline reconstructed reflectance functions. Non-linear approximation usingthe Haar wavelet will only be better suited when compressing reflectance functionsthat are non-smoothly reconstructed (e.g. using the zero-order hold technique).

5.5 ConclusionIn this chapter, relighting with directionally dependent illumination, is surveyed morethoroughly. All techniques in this chapter are designed for data resulting from usinga Light Stage or Free-form Light Stage data acquisition and thus, based on a set ofsamples of the reflectance function for a set of directions.

First, several new reconstruction techniques for reflectance functions are presented.These techniques are the generalized zero-order hold method, interpolation betweenmeasured values and a multilevel B-Spline reconstruction technique. For the specialcase of measured values at regularly sampled directions in latitude-longitude parame-terization different wavelet interpolation techniques are presented as well.

Then, these techniques are compared with previously published reconstructiontechniques such as the projection of the samples onto the spherical harmonic basis

Page 135: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

5.5. CONCLUSION 123

functions and fitting the data onto a single biquadric polynomial. This comparison isthree-fold: a numerical comparison in which the error of the reconstructed reflectancefunction for each method is calculated, a visual comparison of images of the relit ob-ject resulting from reflectance functions reconstructed with different techniques anda visual comparison for the temporal coherence of a series of images resulting fromanimated incident illumination for the different reconstruction techniques.

Since most of the reconstruction techniques produce an algebraic expression forthe reconstructed reflectance function, the storage requirement may become a prob-lem. Moreover, evaluating this algebraic form for each direction during relightingmay be a bottleneck for the time to compute an image. Several compact representa-tions such as the use of different box functions, spherical harmonics basis functionsor the use of the wavelet transform are discussed.

As a conclusion we find that the multilevel B-Spline reconstruction techniqueperforms best, while the non-linear wavelet approximation with the 9/7 Daubechieswavelet represents a reflectance function in the most compact way. The 5/3 LeGallwavelet interpolation and representation is a valid alternative, but may incorporateminor errors.

Page 136: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 137: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 6

4D Reflectance Functions

A relighting technique using 4D incident illumination has been surveyed in chapter 4.In that chapter, the discussion of the relighting technique mainly focused on the dataacquisition method, a speedup for this data acquisition method and a straightforwardtechnique to construct reflectance functions from the acquired data. In this chapter,we focus on the reflectance function reconstruction and the data representation of theacquired data. By using more intelligent reconstruction techniques for the reflectancefunctions (like the techniques of chapter 5) or using other illumination patterns duringthe data acquisition, better images of the objects with novel illumination can be pro-duced. Additionally, an alternative storage method for the acquired data is discussed,allowing to save only important information.

6.1 Introduction

The techniques discussed in chapter 4 allow to relight a real object with general 4D il-lumination. However, due to the use of the chosen illumination patterns, the resultingimages contain some artefacts. In section 6.2, we survey the possibilities for improve-ments by discussing two options for the illumination patterns: illumination patternsbased on general basis functions and illumination patterns avoiding the artefacts whilestill allowing to use the data acquisition speedup.

Although these improvements allow to create images of the relit object withoutvisible artefacts, the relighting can be further improved by using more intelligent re-flectance function reconstruction techniques, similar to the techniques discussed inchapter 5: the reflectance function values for positions in between sampled positionscan depend on more than one projector. These techniques are discussed in section 6.3.

Finally, section 6.4 describes an efficient alternative to store the acquired data foreach projector position separately.

125

Page 138: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

126 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

6.2 Different Illumination PatternsAlthough the results of the relit toy monkeys on page 87 are visually pleasing anddemonstrate the need for relighting with 4D illumination, the results of figure 4.8clearly contain some artefacts, especially when considering the wooden surface onwhich the chess pieces are placed. In figure 6.1, the chess pieces are illuminated withtwo spot lights and parts of the image are zoomed in to show these artefacts moreclearly.

Figure 6.1: The chess pieces illuminated with two spot lights. Two parts of the imageare zoomed in to clearly show the artefacts.

The origin of the artefacts lies in the use of the white square patterns and thedirect visualization of these patterns. This can be intuitively understood when thinkingof the relighting as a linear combination of the basis images and just using a singleprojector position: the incident illumination is distributed in the form of weights overthe basis images. A high-frequency angular change in the incident illumination willresult in very different weights for basis images corresponding to neighboring patterns.A spotlight is such a high-frequency angular change in incident illumination, and thus,the resulting relit image will contain artefacts.

By using other illumination patterns, these artefacts can be avoided. In the nextsection (section 6.2.1) a general solution to the problem is discussed. Although thisgeneral solution allows to use a wide range of illumination patterns, each with itsown properties, not all of these illumination patterns allow to use the speedup as dis-cussed in section 4.5. In section 6.2.2, a set of illumination patterns is discussed thateliminates the artefacts to some extent and still allows to record a reduced number ofimages.

6.2.1 General SolutionIn section 4.2, all relevant directions, for a projector position pi, are called Ωpi . Thisset of directions corresponds to all directions in the angular range of the projector at

Page 139: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.2. DIFFERENT ILLUMINATION PATTERNS 127

position pi. An illumination condition Ci, j is then defined as the light originating fromposition pi into the set of directions Ωpi

j , which is a subset of Ωpi and can be writen asa box function BOXpi

j .Instead of using box functions BOXpi

j for the reflectance function reconstruction asin equation 4.5 other sets of basis functions on the domain Ωpi can be used. Supposefor position pi, a set of M basis functions Bpi

j is chosen that return the value 0 fordirections outside Ωpi , then the illumination conditions become:

∀i, j 1≤ i ≤ P, 1≤ j ≤M : Ci, j(p← ω) =

Bpij (ω) with p = pi

0 otherwise.

With these illumination conditions, one can make a similar deduction as for the boxfunctions and obtain the reflectance function reconstruction formula:

Rx,y(p← ω) =M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω))

with p ∈ Spi . (6.1)

The set of basis functions can be any set: box functions, spherical harmonic basisfunctions, wavelet functions defined on the projector image plane or even other basisfunctions, as long as the set of basis functions are an orthonormal basis.

By using basis functions with only non-zero values for directions in Ωpi , the re-sulting illumination conditions can still be recreated in reality, since the angular rangeof the projector is limited to Ωpi . However, in general, basis functions may return neg-ative values for some directions or correspondingly, positions on the projector plane.Since negative radiance values cannot be emitted by any display system, these func-tions could be impractical as illumination conditions. To solve this problem, a basisfunction can be expressed as the subtraction of two positive functions. These two pos-itive functions can be used as illumination conditions and the resulting images can besubtracted, to create the basis image resulting from the original basis function. Notethat the time requirement of the data acquisition process doubles for such basis func-tions.

To reduce the artefacts in the relit image, an appropriate set of basis functionsneeds to be selected. Although any set of basis functions can be used as illuminationconditions for a fixed projector position, the choice of this set of basis functions maybe also be influenced by the kind of material of the object. In the next two sections wediscuss both these factors: the error introduced by the set of basis functions and theinfluence of the material of the object on the artefacts.

The Error Introduced by the Basis Functions

The use of basis functions for illumination conditions is not new, but was alreadymentioned as an elegant theoretical possibility for relighting with angularly varyingincident illumination in section 3.3.2 and was briefly discussed with the practical im-plementation for wavelet environment matting in section 3.4.3.2. However, the use

Page 140: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

128 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

of basis functions as illumination conditions also introduces an error in the relighting.We discuss this error here.

As already noted several times, the relighting with 4D illumination can be sepa-rated into two 2D problems: for each projector position pi, a set of directions Ωpi is ofimportance. If we fix our attention to a single projector position pi, then the incidentillumination conditions are only dependent on the directions in Ωpi and are given as:

∀ j 1≤ j ≤M : Ci, j(pi← ω) = Bpij (ω) .

The resulting M basis images Imgi, j (with 1≤ j ≤M) when using these illuminationconditions, are the projection of the true reflectance function onto the different basisfunctions Bpi

j . The reflectance functions at position pi are approximated as:

Rx,y(pi← ω) ≈M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω)).

These reconstructed reflectance functions at position pi are approximated with thebasis functions Bpi

j . The basis functions Bpij span a subspace V of the function space

H and the reflectance function is approximated in this subspace V . Therfore, in thisapproximation of the reflectance function, an error is made, that partially causes theartefacts in the resulting relit images. Of course, this error accumulates for differentprojector positions in the general reflectance function (equation 6.1). By choosing a“better” set of basis functions, this approximation error can be reduced.

A “better” set of basis functions is a set of basis functions that span a subspace V ofthe function space H so that V is almost equal to H and thus spans the function spaceH. Since such a basis is hard to find, we choose a basis dependent on the reflectancefunction. However, this constraint is not well defined, since this would require toknow the reflectance functions in advance. Therefore, we propose to use a set of basisfunctions that can represent as many reflectance functions as possible. In chapter 5,we already made a survey of sets of basis functions to represent reflectance functions.We review the different options again in this context:

• Box Functions: As already presented, box functions can be chosen. To ap-proximate a function, the error on the approximation to the real function oftendecreases by increasing the number of box functions. This property originatesfrom the fact that the basis consisting of more box functions will span a largersubspace V (in the limit, it will span the complete function space H). Therefore,using more box functions in the data acquisition will allow to reduce the errorsin the resulting images of the relit object. However, increasing the number ofbox functions results in a linear increase in the data acquisition time, when us-ing the speedup. Moreover, box functions may not be the best choice for basisfunctions, since these functions result in a piecewise constant reflectance func-tion. In chapter 5, we noted that piecewise constant reflectance functions mayincur discontinuities.

• Spherical Harmonic Basis Functions: In section 5.4.2, spherical harmonicbasis functions did not prove to be useful to represent reflectance functions,

Page 141: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.2. DIFFERENT ILLUMINATION PATTERNS 129

since they have difficulties representing high-frequency functions. Therefore,these functions are not suitable as illumination conditions either.

• Smooth Wavelets: At first sight, smooth wavelets may seem very promisingas illumination conditions. However, there are no smooth finite wavelets thatare orthonormal. Since the 9/7 Daubechies wavelet is near orthogonal, we canstill pursue this idea. However, these functions may have non-zero values overthe complete domain Ωpi

j . Since the data acquisition speedup assumes verylocal influence, this speedup can no longer be used. Because of the long dataacquisition time (i.e. all wavelet functions need to be projected and all must besplit into two functions and be projected separately), we did not implement thisapproach.

[46] proposes to use a feedback loop to determine the most important waveletsas illumination condition and thus reduce the total amount of illumination con-ditions. This approach would still require to much data acquisition time andwas therefore not implemented.

The Error Introduced by Different Surfaces

As was noted earlier, the artefacts are less visible in the scene with the monkeys thanin the scene with the chess pieces, even though the set of basis functions for the chesspieces (32×32 box functions) was “better” than the set of functions for the toy mon-keys (16× 16 box functions). Therefore, the artefacts in the relit images do not onlydepend on the ability of the basis functions to approximate the reflectance functionwell, but also on the kind of surface in the scene.

To demonstrate this fact clearly, a fairly flat scene was constructed, consisting offour different materials (dark wood, diffuse bumped paper, foam balls and stones), asdepicted on the top of figure 6.2. Dark wood and bumped paper have little scatteringeffects, while stones are opaque as well but distort the flat surface severely. The foamballs distort the surface and may produce scattering effects. For this scene, we selecteda projector position such that the projection plane coincided with the plane of thescene, in order to be able to have a focused projected pattern. Similarly, we positionedthe camera close to the projector position, with the view direction orthogonal to theplane of the scene, so that only the direct reflectance effects and no warping effectswould be noticeable due to the camera.

We projected an artificially constructed pattern (a ring shape) and a natural image(a digital photograph) onto this scene to have a reference of what the scene shouldlook like when illuminated with these patterns from the fixed projector position. Thesepatterns and the images of the resulting illuminated scene are depicted at the middleand bottom of figure 6.2 respectively.

The scene of figure 6.2 was illuminated from the fixed projector position withdifferent sets of box functions and the basis images were recorded. The sets werechosen to be box functions on a grid, with different grid resolutions. These resolutionswere 64× 64, 32× 32, 16× 16 and 8× 8. The reflectance functions of the scenewere reconstructed four times with these four sets and the scene was relit with thetwo illumination patterns of figure 6.2 with each of these four sets of reconstructed

Page 142: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

130 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

A

C

B

D

Figure 6.2: On top, a scene is depicted with neutral illumination. This scene is afairly flat surface consisting of four different materials: dark wood (A), white diffusebumped paper (B), foam balls (C) and stones (D). In the middle, two illuminationpatterns are displayed. The resulting images when projecting these patterns onto thescene is shown at the bottom.

reflectance functions. These results are shown from top to bottom in figure 6.3 withdecreasing grid resolution.

As noted in the previous section, we clearly see that using many box functions(i.e 64× 64) results in better relit images than using only few box functions. This isbecause such a set of box functions can approximate the reflectance functions better.We call the error of the reconstructed ring pattern to the true ring pattern, Epattern.This error Epattern thus expresses the relative difference between the values of thereconstructed pattern and the true pattern. On the top right of figure 6.4, this errorEpattern is displayed for the four quadrants of the pattern for the different sets of basisfunctions. As can clearly be seen, the error Epattern of the reconstructed pattern to thetrue pattern decreases when more basis functions are used. This was to be expected.

Similarly to error Epattern, we call the relative error of the image of the projec-

Page 143: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.2. DIFFERENT ILLUMINATION PATTERNS 131

64 x 64

16 x 16

8 x 8

32 x 32

Figure 6.3: The illumination patterns of figure 6.2 are approximated with different setsof box functions and the resulting illuminated scene is displayed. From top to bottom,64×64, 32×32, 16×16 and 8×8 box functions were used.

tion of reconstructed patterns and the image of the projection of the true pattern, errorEpro jection. This error Epro jection is thus the relative difference between the images offigure 6.3 to the true image in figure 6.2. On the bottom left of figure 6.4, the errorEpro jection is displayed for the different quadrants. As was expected, the error also de-creases when more basis functions are used. Notice that the error for the quadrant withthe foam balls is the smallest, while the error for the upper quadrants is largest. Thiseffect is due to the scattering of the foam balls, which blurs the incident illumination.

We can find the effect of the different materials by computing how the error ofthe images Epro jection relates to the error on the reconstructed patterns Epattern. This isachieved by calculating the effect of the material as:

Page 144: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

132 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

Material coefficient =Epattern

Epro jection.

If this coefficient remains constant for different basis functions, the Epro jection is di-rectly related to the error Epattern and thus the material has no effect on the error of theresulting images. However, if the coefficient increases for different sets of basis func-tions, the error Epro jection decreases faster than the error Epattern and thus, the materialhas an impact on the error. On the right bottom of figure 6.4, this material coefficientis plotted for the four quadrants over the different sets of basis functions. As can beseen, the coefficient increases for the foam balls but remains constant for the threeother materials. Therefore, we can conclude that the material can have an impact onthe error.

From our experiments, materials that produce a lot of scattering effects will resultin a larger reduction of error Epro jection, than the reduction imposed by the differentsets of basis functions. For other materials that induce no scattering effects, this is notthe case. This is why the toy monkeys with a surface made of fur, which has manyscattering effects, had less noticeable artefacts than the wooden plate under the chesspieces.

Depending on the kind of material, the requirements for the set of basis functionsas illumination conditions, can be relaxed.

0.05

0.10

0.15

0.20

0.25

0.35

0.30

0.25

0.20

0.15

0.10

0.05

012345

789

10

6

0

08x8 16x16 32x32 64x64

8x8 16x16 32x32 64x648x8 16x16 32x32 64x64

Epattern

Epro jectionEpro jection

Epattern

Figure 6.4: Errors introduced by the use of basis functions and the projection of thepatterns onto different materials.

Page 145: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.2. DIFFERENT ILLUMINATION PATTERNS 133

Perceptual Error

The discontinuities of the box functions in figure 6.3 is even more accentuated visuallydue to the perceptual Mach band effect [20]. The Mach band effect is the effect that theeye will observe an intensity change in an otherwise uniformly colored surface as moresignificant than the same discrete intensity change in an otherwise high-frequencyintensity changing surface. Even though the focus of our work is physically correctrelight of real objects, we still want to obtain ’nice’ images.

For materials containing high-frequency intensity changes or frequent geometricdiscontinuities, these Mach band effects due to the box functions are therefore less ofa problem, and the resulting images look ’better’. This can be demonstrated in ourexample: the discontinuities due to the box functions are easily perceived on the darkwood or even the bumped diffuse white surface, but are harder to detect on the smallstones.

6.2.2 Fast SolutionAlthough the previous general technique proposed to work with a set of basis func-tions, other sets of functions can be used as well, as long as the values of Bpi

j (ω) forany direction ω sum to 1:

∀ω ∈Ωpi :M

∑j=1

Bpij (ω) = 1.

This requirement originates from the fact that the amount of light (or energy) mustbe balanced. If the sum would be larger than 1, more energy would be reflected thanwas incident, if the sum would be smaller than 1, energy would be lost (while theabsorption is already taken care of by using recorded images). Note that the boxfunctions fulfill this requirement. Examples of other sets of functions, which are notorthonormal, but also have this property are a set of normalized hat-functions andnormalized box functions convolved with a Gaussian function. These sets can thus beused as well. The advantage of these sets of functions is that they still allow to easilyuse the speedup technique.

We used the functions based on the box functions convolved with a Gaussian func-tion, since these smooth functions do not suffer from the Mach Band effect [20]. Theconvolution width was set to be equal to the width of the box functions. In figure 6.5,the two toy monkeys are displayed (from a different viewpoint as in previous exam-ples) with a projector illuminating the monkeys completely. On the left of the figure,a single basis image is displayed (with zoom in) when using a box function. At thebottom left, an ’X’ mark is used as incident illumination. This illumination is heav-ily distorted and severe aliasing artefacts can be seen in the image of the relit object.On the right of figure 6.5, the corresponding images are displayed using Gaussianconvolved patterns. As can be seen, the resulting relit image suffers less from theseartefacts.

These patterns allow to reduce the perceptual artefacts significantly while the dataacquisition speedup can still be used. Since a single Gaussian convolved square covers

Page 146: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

134 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

a larger solid angle, the distance d in the speedup method will need to be increased.However, less functions are required when exploiting the kind of surface of the objector scene. In figure 6.6, the scene of figure 6.2 is re-used and the box functions offigure 6.3 are convolved with the Gaussian function and used as illumination functionson that scene. In figure 6.6, one can notice that for surfaces with a lot of scatteringeffects, like the foam balls, few functions are necessary to obtain reflectance functionsenabling to create faithful relit images.

Notice that this kind of illumination conditions blurs the incident illumination, butnot the image. Therefore, discontinuities in the angularly varying part of the illumina-tion will be blurred as well, and thus the effect of a sharp spotlight cannot be emulated

Figure 6.5: On top, the two toy monkeys are illuminated with a single fixed projector.On the left, a basis image when using a box pattern (with zoom in) and the resultingimage of the toy monkeys when projecting an ’X’ mark. On the right, the same imageswhen using box functions that are convolved with a Gaussian function.

Page 147: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.3. 4D RECONSTRUCTION TECHNIQUES 135

with these functions.

64 x 64

16 x 16

32 x 32

8 x 8

Figure 6.6: The illumination patterns of figure 6.2 are approximated with different setsof Gaussian convolved box functions and the resulting illuminated scene is displayed.From top to bottom, similar as for figure 6.3, 64× 64, 32× 32, 16× 16 and 8× 8functions were used.

6.3 4D Reconstruction TechniquesIn this section we theoretically define the most general reconstruction technique for4D reflectance function. First, we rewrite the 4D reflectance function reconstructiontechnique in a single equation. This equation can then be generalized into the recon-struction technique that is applicable to all relighting techniques.

Page 148: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

136 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

6.3.1 Rewriting the 4D Reconstruction TechniqueThe reflectance functions for relighting with 4D illumination depend on the positionand direction in the 4D illumination space. This 4D illumination space can be seen astwo 2D spaces: positions on surface S and directions in angular space Ω. In chapter 4these two spaces were subdivided into partitions. The surface S was partitioned intoP partitions Spi , based on a Voronoi diagram with Voronoi sites being the P sampledpoints in S.

For a position pi, a set of M functions Bpij can be defined. These functions are

defined for all directions in Ω and can be emitted by the projector during the dataacquisition process. For all positions in Spi this set of functions Bpi

j could be used,resulting in a reflectance function (equation 6.1):

Rx,y(p← ω) ≈M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω))

with p ∈ Spi .

In this definition, the reflectance function depends on the partition Spi in which thepoint p lies. Since this is a Voronoi function, we can define a set of P functions Vi as:

Vi(p) =

1 if p ∈ Spi

0 otherwise.(6.2)

Substituting equation 6.2 into the definition of the reflectance function results in:

Rx,y(p← ω) ≈P

∑i=1

(M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω))×Vi

).

6.3.2 General 4D Reconstruction TechniqueInstead of using a set of box functions Vi based on a Voronoi diagram, other recon-struction functions can be used as well. In chapter 5, different reconstruction tech-niques were discussed for sampled reflectance functions with incident illuminationonly dependent on the direction (i.e. in the angular space Ω). However, the basics ofthese techniques can be converted to the positional space S as well. These techniquescan easily be plugged in the definition of reflectance functions for general incidentillumination.

The general 4D reflectance function reconstruction technique then boils down tothe following procedure:

• A set of P projector positions pi is chosen on the surface S.

For each projector position pi, a subset of Ω, called Ωpi , is defined. This Ωpi isusually the projector cone with the projector at position pi aimed at the object.

For each pi a set of M orthonormal basis functions Bpij (ω) is chosen as illumina-

tion conditions, based on the corresponding Ωpi . (Bpij (ω) = 0 outside ω /∈ Ωpi).

Page 149: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.3. 4D RECONSTRUCTION TECHNIQUES 137

Each basis image Imgi, j is a recording of the real object illuminated with theseincident illumination conditions Ci, j(p← ω), defined as:

∀i, j 1≤ i ≤ P, 1≤ j ≤M : Ci, j(p← ω) =

Bpij (ω) with p = pi

0 otherwise.

• A set of P reconstruction functions Fi(p) on surface S can be chosen, based onthe projector positions pi. Function Fi defines the influence on the reflectanceof the projector at position pi.

The reconstructed reflectance function can then be defined as:

Rx,y(p← ω) ≈P

∑i=1

(M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω))×Fi(p)

). (6.3)

6.3.3 Instantiations of the General 4D Reconstruction TechniqueEquation 6.3 is the most general reconstructed reflectance function. There exist manyinstantiations of this equation. We survey an example of such a 4D reflectance functionand derive the general reflectance functions for the two options of relighting with 2Dillumination.

• In section 5.2.2, interpolation is used to reconstruct reflectance functions withlinear characteristics. The functions Fi can then be a set of P hat functionsaround each position pi, called Hi, with:

∀ p ∈ S :P

∑i=1

Hi(p) = 1.

The reflectance functions can then be defined as:

Rx,y(p← ω) ≈P

∑i=1

(M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω))×Hi(p)

).

• Equation 6.3 can be restricted to a 2D reflectance function to represent the re-flectance function for relighting with angularly dependent incident illumination.If the surface S collapses to a single point, the reflectance functions are indepen-dent of the position and the functions Fi(p) are constants and can be removedfrom the equation, resulting in:

Rx,y(ω) ≈M

∑j=1

(Imgi, j(x,y)×Bpi

j (ω)).

Page 150: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

138 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

• Similarly, if the direction of the reflectance function and the incident illumina-tion is of no importance, than the equation 6.3 can be written as:

Rx,y(p) ≈P

∑i=1

(Imgi, j(x,y)×Fi(p)

),

which are the general reflectance functions for relighting with spatially depen-dent incident illumination.

6.4 Compact Representation of 4D Reflectance Func-tions

In chapter 4, the complete space of directions Ω was reduced to only a subset ofdirections Ωpi for each sampled position pi. This set of directions Ωpi was originallydesigned to contain only directions towards the object. Since the geometry of theobject was not known, Ωpi was chosen to be equal to the angular range of the projector,and the assumption was made that all directions towards the object lie in this projectorcone (i.e. the entire object is enclosed in the projection cone).

Since the angular range of the projector is a superset of the truly desired directions,data being acquired and stored for some directions not directed towards the object, canbe of no use. Although this acquisition can not be prohibited, due to the fact that nogeometric information is at hand during the data acquisition, this data can be discardedafter the data acquisition, to lower the storage requirements.

Moreover, for each particular pixel, not all directions of Ωpi towards the objectmay be of influence. If the influence parameter d is very small (i.e. close-by lightbundles do not interfere), a direction ω may be directed towards the object and still beof no influence to that pixel. This acquired data can also be discarded.

For example, if the box functions are used as illumination patterns, as presentedin chapter 4, the set of directions Ωpi is subdivided into sets Ωpi

j and each set, corre-sponding with a white square pattern is an illumination condition. Some of the setsof directions Ωpi

j will not even illuminate the object directly, other sets will illuminatethe object directly but will probably not influence all pixels representing the object.

Since no information about the object is given, except for the recorded images, wecan only discard the measured values of the reflectance functions for a pixel (x,y) ifthe basis image value Imgi, j(x,y) for an illumination condition Ci, j is below a chosenthreshold. However, this threshold is difficult to define. Another solution is to storeonly a fixed number of values.

Sorting Measured Values

In our solution, the basis images corresponding with a single projector position aregrouped and processed independently. For a single projector position pi, all measuredvalues for each pixel are sorted from large to small and a reference to the basis imagefor each value is maintained. For a pixel (x,y), the largest measured value, for instance

Page 151: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.4. COMPACT REPRESENTATION OF 4D REFLECTANCE FUNCTIONS 139

Imgi, j(x,y), is stored as V1(x,y) and the value j is stored as W1(x,y). The secondlargest value for pixel (x,y) is then stored as V2(x,y) and its reference as W2(x,y),and so on, until the smallest measured value is stored in VM(x,y) and its reference asWM(x,y). This can be performed for each pixel (x,y) resulting in a set of images Vand W .

On the left of figure 6.7, the images V1, V2, V3 and V4 are displayed for a singleprojector position using the box functions for the scene with the toy monkeys of page87. Since the references in W are grid positions, we can visualize these W -values usinga color scheme. References with increasing X-grid position are more red, referenceswith increasing Y-grid position are more green. On the right of figure 6.7, the imagesW1, W2, W3 and W4 are displayed.

Image V1 corresponds with the largest measured value for each individual pixel,and from the references in W1 from which patterns these largest values originate, onecan clearly recognize the regular grid of patterns (partly warped by the monkeys infigure 6.7). For pixels not directly illuminated, W1 gives a reference, based on therecorded noise.

With image V2, we can visualize the second largest measured value for each indi-vidual pixel. Here we can recognize the borders of the regular grid of the projectedpatterns. This is due to the projector that is out-of-focus. Since the object is not flat,the projector can not be in focus for all positions on the object. Projecting a pattern infocus is thus not possible, which results in blurred patterns on the object. Therefore,the borders of the patterns are direct illumination for some positions on the object.

Note that the positions not on the border of patterns are not completely dark. Thisis due to the ’illumination-bleeding’ of projecting a square pattern: pixels not directlyilluminated by the light bundle, but very near positions that are directly illuminatedwill be influenced by this light bundle. W2(x,y) is the second most important patternto pixel (x,y) based on the size of the measured values. As can be seen in figure 6.7,the grid positions of the references in W2 are warped and jittered. As can also be noted,more pixels have a reference based on noise.

The images V3 and V4 still have some direct illumination effects, at the corners ofthe projected patterns. We learn that the measured values are lower values (third andfourth) and the accompanying references W3 and W4, become more random, due tothe noise. W3(x,y) and W4(x,y) are the third and fourth most important light bundle topixel (x,y). Therefore, we clearly see a correspondence between the size of measuredvalues and structure in the references.

In general, V1 will contain only direct illumination effects, the first subsequentimages V2 and V3 will contain mostly direct illumination effects and few indirect illu-mination effects. Then the images V contain only indirect illumination effects, whilethe last images of the series of V images contain only noise.

Discarding Measured Values

Since the images V represent the measured values per pixel in a sorted way and theaccompanying different values in each ’image’ W refer to the patterns illuminatingthe object, the user can decide to include only the set of the first n (< M) images

Page 152: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

140 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

V

V

V

V

1

4

3

2

1

4

3

2

W

W

W

W

Figure 6.7: The monkeys from page 87, captured with white square patterns. On theleft, the measured values and on the right a reference to the basis image to whichthe measured value corresponds per pixel (patterns with increasing X-grid positionare more red, patterns with increasing Y-grid position are more green). From topto bottom, the first to fourth largest measured value per pixel. In the first row, themonkeys are clearly recognizable and there is a clear one-to-one mapping of measuredvalues and the direct illumination. The second row shows direct illumination dueto the projector that is out of focus and the ’illumination-bleeding’ of neighboringpatterns. The reference-to-basis image outside the angular range of the projector ismerely noise.

Page 153: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

6.4. COMPACT REPRESENTATION OF 4D REFLECTANCE FUNCTIONS 141

V1,V2, ...,Vn and the accompanying reference images W1,W2, ...,Wn in the re-flectance function reconstruction. The rest of the measured values can then be dis-carded by changing these values to 0. The choice for n is not clearly defined. However,there are some minimal bounds for n:

• If the user decides to set n equal to 1, and thus to use only a single measuredvalue (i.e. V1) per pixel per projector, artefacts will be noticed in the resultingrelit images. On the top left of figure 6.8 only the largest measured value perpixel was used to relight the toy monkeys with an ’O’-shaped spotlight fromthe projector position used in figure 6.7. Seams can be seen where neighboringsquare patterns meet and no inter-reflection effects appear in the relit images.

• Using only a very small set of values (i.e. n is 3 or 4) results in a visual pleasingimage of the relit object, although a lot of illumination effects such as inter-reflection and scattering are lost (top right of figure 6.8).

When using the box functions in a grid, there is also an upper bound on the numberof measured values n, if the speedup technique was used during the data acquisition. Ifthe distance between two light bundles in a horizontal or vertical patterns was d, thenonly the first (d + 1)2 measured values can be of importance. Using more measuredvalues for the reconstruction of the reflectance functions will only add noise to theresulting reflectance functions. When the object is relit using more measured values,accumulated noise effects can be noted in the relit images. At the bottom right offigure 6.8, 25 measured values were used. As can clearly be seen, the middle of the’O’-shaped spotlight effect lights up, while these parts of the toy monkeys should notbe illuminated. Since we want to incorporate as many illumination effects as possible,while not storing useless information or noise, we propose to store and use only thefirst (d + 1)2 measured values. In the example of the toy monkeys, the distance waschosen to be 2 and thus, only the first 9 measured values are truly of importance:at the bottom left of figure 6.8 all illumination effects are noticeable without muchnoise. However, this is only a guideline based on the influence parameter for the boxfunctions and the user is able to select the number of measured values.

If wavelets would be used as illumination conditions for each projector position,this compact representation can be seen as a non-linear approximation of all measureddata. For a non-linear approximation using wavelets of an ordinary image, the imageis converted into wavelet space by projecting the image onto the wavelet functions andonly the largest coefficients are stored and the small coefficients are set to zero. Anapproximation of the original image can be created by converting these coefficientsback to image space. Selecting only the first n images of V and W is similar: the truereflectance function is projected onto the wavelet functions during the data acquisi-tion. The n largest measured values can then be used while the other values are set tobe zero. Reconstructing the reflectance functions with only these values is an approx-imation of the original captured reflectance function values. Therefore, this techniquecan be seen as a non-linear approximation of the reflectance functions

Page 154: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

142 CHAPTER 6. 4D REFLECTANCE FUNCTIONS

41

9 25

Figure 6.8: The monkeys from page 87, relit with an ’O’-shaped spotlight, usingonly the data from the projector position used in figure 6.7. On the top left only themost important measured value for each pixel is used (V1) for the reflectance functionreconstruction. On the top right, all the values depicted in figure 6.7 are used (V1, V2,V3 and V4). On the bottom, 9 and 25 most important values are used. As can be noticed,using to few measured values results in artefacts, or loss of illumination effects, whileusing almost all measured values results in noise in the reflectance functions.

6.5 ConclusionIn this chapter, relighting with 4D illumination is further improved.

First, two possibilities are presented to remove the artefacts of relighting withthe basic technique presented in chapter 4. One approach is to use any set of basisfunctions instead of only box functions as illumination patterns per projector position.A second approach solves the artefacts while maintaining the possibility to speed upthe data acquisition process as discussed in section 4.5. For either solution, boththe kind of illumination patterns as well as the kind of materials of the scene are ofimportance.

Then, by combining these improvements with the techniques presented in chap-ter 5, a general reflectance function reconstruction formula can be deduced, allowingto use two sets of functions, one set for the angular part of the 4D illumination andone set of functions for the positional part of the 4D illumination.

Finally, this chapter discusses a simple but effective way to reduce the data storageper projector position, based the size of the measured values per pixel, and very similarto a non-linear approximation of the reflectance functions.

Page 155: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Chapter 7

Conclusion

This chapter concludes this dissertation. In section 7.1 a general summary of theresearch work is overviewed. The original contributions to the field of research aresummarized in detail for each chapter separately in section 7.2. Section 7.3 finalizesthe dissertation by over-viewing the possible avenues for future research, based on thiswork.

7.1 SummaryThis work has focused on the relighting of real objects. The visualization of realobjects with novel incident illumination was restricted to a single viewpoint to theobject out of practical considerations.

In the first part of our work, a general relighting framework was developed, basedon physical properties of light: a transport function called a reflectance field can beused for relighting real images. To make the framework practical, an image-based ap-proach was proposed to construct this reflectance field of the real object. In this gen-eral image-based relighting method, the user is required to select a set of illuminationconditions, apply these illumination conditions to the real object and record images ofthe resulting illuminated object. From these images, all necessary information is to beretrieved to construct the reflectance field and relight the real object. Throughout therest of the dissertation, previously published relighting techniques were united in thispractical image-based relighting framework by demonstrating that all these techniquesonly differ in the choice of illumination conditions and construction of the reflectancefield.

In the second part of the dissertation, the practical image-based relighting frame-work was instantiated with different newly designed relighting techniques. Simpletechniques for restricted (2D) relighting and general (4D) relighting were overviewed,by discussing the data acquisition techniques and the straightforward construction ofthe reflectance field. These techniques were refined in subsequent chapters by us-ing the presented data acquisition techniques but using more intelligent constructiontechniques for the reflectance field.

143

Page 156: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

144 CHAPTER 7. CONCLUSION

7.2 Original ContributionsWe overview the original contributions to the relighting research field for each chapterseparately:

Chapter 2• A formal description of relighting of real objects with novel incident illumi-

nation is given, based on the properties of light and the notion of a transportfunction that was previously derived by several authors in a global illuminationcontext.

• A practical image-based relighting framework is developed, which consists ofthree stages. First, the data is acquired by choosing the illumination conditions,applying these illumination conditions to the real object and recording imagesof the illuminated object. Subsequently, these images are used to reconstructthe reflectance field of the object. Finally, using this reflectance field, the objectcan be visualized with arbitrary novel incident illumination.

Chapter 3• The practical relighting framework is expressed for relighting with restricted

2D illumination. This 2D incident illumination is either independent of thedirection or independent of the position. For both cases, a dimensional reductionof the reflectance field can be achieved, allowing a relatively easy reproducibleset of illumination conditions.

• A new, more practical data acquisition technique is presented. A light sourceis moved freely around and over the object while a camera records images con-tinuously. From the shading information of 4 diffuse white spheres positionedaround the object, and visible by the camera during the photo shoot, the illumi-nant direction can still be estimated. This technique circumvents the limitationsimposed by previously used data acquisition methods using a gantry.

• Previously published relighting techniques are fit in the general framework.These techniques include the Light Stage, relighting with steerable functionsand all environment matting techniques, among others.

Chapter 4This chapter introduced the first relighting technique using general 4D incident illumi-nation. By using general 4D illumination, effects such as spot light effects or objectsbeing partially in shadow can be created.

• The data acquisition setup for this relighting technique is discussed, as wellas a reconstruction method for the reflectance functions. When all reflectancefunctions are obtained (i.e. the full reflectance field of the object is known) theobject can be relit.

Page 157: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

7.2. ORIGINAL CONTRIBUTIONS 145

• A speedup for the proposed data acquisition technique is presented as well, al-lowing to reduce the data acquisition time and data storage requirements signif-icantly. This speedup builds on the assumption of limited influence of differentbeams of light, illuminating a single real object.

Chapter 5

In this chapter, relighting with 2D illumination dependent on the direction (i.e. re-lighting with environment maps), is further investigated.

• Three new techniques for the reconstruction of reflectance functions, based ona set of measured samples, are introduced: a generalized zero-order hold tech-nique, data interpolation between the measured values and a multilevel B-Splineapproximation reconstruction technique.

• These new reconstruction techniques are compared with previously publishedreconstruction techniques on different levels: a numerical comparison, a visualcomparison on still relit images and a visual comparison on a series of relitimages resulting from animated incident illumination.

• Different compact representations of the reconstructed reflectance functions arepresented: projecting the reflectance functions on a set of box functions (basedon regular sampled and structured importance sampled directions), expressingthe reflectance functions with spherical harmonic basis functions and convertingthe functions into wavelet space (Haar wavelet, 5/3 LeGall wavelet and 9/7Daubechies wavelet).

Chapter 6

In chapter 6 the relighting method of chapter 4 (i.e. relighting with general 4D illumi-nation) is further improved.

• Two solutions to the visual artefacts in the resulting images of the relit objectsare proposed: the use of general basis functions and use of smoother local pat-terns as illumination conditions while allowing to use the speedup method.

• Previous improvements are combined with the intelligent reconstruction tech-niques of chapter 5, resulting in a general formula for the data acquisition andconstruction of the 4D reflectance functions.

• An efficient data storage technique is presented. This technique allows to storeonly the useful information and discard noise on the acquired information, foreach projector position.

Page 158: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

146 CHAPTER 7. CONCLUSION

7.3 Future DirectionsThe general relighting framework and practical image-based relighting method allowto think about relighting of real objects in a more mathematical way. Besides com-paring previously published techniques fairly, by fitting them in the framework, andpointing out the differences mathematically, the framework opens a gate for a wholenew range of relighting techniques. Future directions of research can exploit thisframework to design these techniques formally.

7.3.1 Better Reflectance Function Reconstruction

With the existing data acquisition devices, the relighting of real objects with 4D illu-mination can be improved. As discussed in section 6.2.1, different combinations ofusing basis functions on the projected image together with reconstruction techniquesin between sampled projector positions are possible. Until now, we only have a vagueidea of the possible results and advantages of some combinations. Implementing andcomparing different combinations of the general reconstruction technique is thereforean immediate possible future direction in the image-based relighting research.

Using 4D basis functions may result in even better reconstructed reflectance func-tions. However, not all sets of 4D basis functions spanning the 4D illumination spacewill prove to be useful for 4D relighting. The sets of 4D basis functions should prob-ably be tailored specifically to the 4D relighting. This problem is to be researched aswell.

7.3.2 Design new Data Acquisition Devices

Using the framework, we can design mathematically elegant new relighting tech-niques, requiring few recorded basis images and allowing to generate visually pleas-ing images of the relit object. However, these techniques often require the use of basisfunctions as illumination conditions that can not be recreated in reality.

Basis functions producing negative values are impossible to reproduce. This prob-lem can be solved by splitting the basis function into two strictly positive functions.These two functions can be used as incident illumination and the resulting images canbe subtracted to obtain a basis image of the original (partly) negative basis function.Also, the basis functions are often continuous, while no illumination devices exist ca-pable of emitting these basis functions. CRT monitors and other display devices cansolve this problem to some extent for relighting with spatially dependent illumination.However, relighting from an enclosing surface is still impractical, if not impossible.Moreover, no illumination devices exist capable of recreating continuous directionalillumination for a hemisphere, let alone devices to create continuous 4D illuminationbasis functions.

If these devices would exist, faster data acquisition methods and better relightingtechniques could be implemented. The general framework already provides the the-ory for such techniques. For example, if a device would be able to emit directionallight continuously over the hemisphere, one could use illumination conditions based

Page 159: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

7.3. FUTURE DIRECTIONS 147

on steering functions, as discussed in section 3.3.1 or based on spherical harmon-ics. Designing such acquisition devices is therefore an important avenue for futureresearch.

7.3.3 Camera Movement

This work focused on visualizing a real object, viewed from a fixed viewpoint, withnovel incident illumination. This restriction to a single viewpoint allowed to reducethe reflectance field to six dimensions, and thus lower dimensional reconstructiontechniques can be used. Therefore, the data acquisition step can be simplified sig-nificantly: no camera calibration was necessary (neither position estimation nor lensdistortion), only a single image was to be recorded for each illumination condition,...However, even though beautiful images can be created using these techniques, the useroften desires to look at the object from a different point of view than the viewpointused during data acquisition.

An option was already mentioned in the introduction: capture the reflectance fieldfor a set of viewpoints [37]. Then the user can select an arbitrary viewpoint for whichthe nearest captured reflectance fields are looked up. These reflectance fields are usedto compute the relit object with the desired incident illumination and the resulting relitimages (from the nearby viewpoints of the desired viewpoint) are combined usingimage-based rendering techniques. In order to obtain good results with this image-based rendering technique, an approximation of the geometry of the object will berequired.

Although this method will allow to visualize the object from an arbitrary view-point with novel incident illumination, better techniques can be thought of. Since the(approximate) geometry of the object would be known, 6D reflectance functions canbe created on the surface of the geometry of the object. These 6D reflectance func-tions can again be sampled with the data acquisition techniques presented in this work,and be reconstructed with techniques similar to the reconstruction techniques also pre-sented in this work. Although these techniques may be based on methods described inthis work, the implementation is still not straightforward and requires some research.

7.3.4 Other Avenues

Of course, there are many other avenues for future research for image-based relight-ing of real objects or scenes. One could argument that dynamic scenes are the nextimportant step in the relighting algorithms, be it movie relighting instead of relight-ing a still image or visualizing animatable objects with arbitrary incident illuminationlike [24]. Other people may urge to research real time relighting algorithms, capableof visualizing an object with real time captured incident illumination like [39]. Evenother people are interested in relighting gigantic scenes such as cities like the workof [36]. All these different applications require custom tailored relighting techniquesand more research.

At the time when I started my own work, image-based relighting was a whole newresearch field of which few people knew about. Now, it has become a familiar term in

Page 160: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

148 CHAPTER 7. CONCLUSION

computer graphics and more and more new applications are proposed, hence, survey-ing all future research possibilities would lead us to far. In fact, all these techniquesare similar and differ in data acquisition approach and assumptions only.

In this work, I tried to fit all these techniques in a single framework, in order to geta good view on what already exists in the relighting research field, and added a fewnew methods. Therefore, I hope this can be the basis for other people to come up witheven better relighting techniques. I’m very curious what is coming next...

Page 161: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Publications

• Vincent Masselus, Philip Dutre, and Frederik Anrys. “The Free-form LightStage” In S. Gibson and P. Debevec, eds., Rendering Techniques ’02 (Pro-ceedings of the 13th EuroGraphics Workshop on Rendering), Pisa, Italy, pages247–256, ACM Press, June 2002.

• Vincent Masselus, Philip Dutre, and Frederik Anrys. “The Free-form LightStage” Technical Sketch, SIGGRAPH2002, San Antonio TX, USA, ACM Press,July 2002.

• Vincent Masselus, Pieter Peers, Philip Dutre and Y. D. Willems. “Relightingwith 4D Incident Light Fields” ACM Transansactions on Graphics (Proceed-ings of SIGGRAPH2003, annual conference), San Diego CA, USA. pages 613–619, ACM Press, July 2003.

• Vincent Masselus. “Visualizing Real Objects with Novel Illumination” 1stFlanders Engineering PhD Symposium, Brussels, Belgium, December 2003

• Vincent Masselus, Pieter Peers, Philip Dutre and Y. D. Willems. “Smooth Re-construction and Compact Representation of Reflectance Functions for Image-based Relighting” In H. W. Jensen and A. Keller, eds., Rendering Techniques’04 (Proceedings of EuroGraphics Symposium on Rendering 2004), Norrkoping,Sweden. pages 287-298, ACM Press, June 2004.

149

Page 162: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer
Page 163: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

Bibliography

[1] Edward H. Adelson and J. R. Bergen. The Plenoptic Function and the elementsof early Vision. In Landy and Movshon, editors, Computational models of visualprocessing. MIT press, 1991.

[2] Sameer Agarwal, Ravi Ramamoorthi, Serge Belongie, and Henrik Wann Jensen.Structured importance sampling of environment maps. ACM Trans. Graph.,22(3):605–612, 2003.

[3] Daniel G. Aliaga and Ingrid Carlbom. Plenoptic Stitching: A Scalable Methodfor Reconstructing 3D Interactive Walkthroughs. In Eugene Fiume, editor, SIG-GRAPH 2001, Computer Graphics Proceedings, Annual Conference Series,pages 443–450. SIGGRAPH, ACM Press / ACM SIGGRAPH, 2001.

[4] Michael Ashikhmin and Peter Shirley. Steerable illumination textures. ACMTrans. Graph., 21(1):1–19, 2002.

[5] R. Azuma. A survey of augmented reality, In Computer Graphics (SIGGRAPH’95 Proceedings, Course Notes: Developing Advanced Virtual Reality Applica-tions) (Aug. 1995), pp. 1–38., 1995.

[6] James F. Blinn and Martin E. Newell. Texture and reflection in computer gener-ated images. Commun. ACM, 19(10):542–547, 1976.

[7] Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and MichaelCohen. Unstructured lumigraph rendering. In Proceedings of the 28th annualconference on Computer graphics and interactive techniques, pages 425–432.ACM Press, 2001.

[8] Wei-Chao Chen, Jean-Yves Bouguet, Michael H. Chu, and Radek Grzeszczuk.Light Field Mapping: Efficient Representation and Hardware Rendering of Sur-face Light Fields. In John Hughes, editor, SIGGRAPH 2002 Conference Pro-ceedings, Annual Conference Series, pages 447–456. SIGGRAPH, ACM Press/ ACM SIGGRAPH, 2002.

[9] Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H.Salesin, and Richard Szeliski. Environment matting extensions: towards higher

151

Page 164: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

152 BIBLIOGRAPHY

accuracy and real-time capture. In Proceedings of the 27th annual confer-ence on Computer graphics and interactive techniques, pages 121–130. ACMPress/Addison-Wesley Publishing Co., 2000.

[10] J. Cohen and P. Debevec. LightGen, HDRShop plugin. 2001.

[11] Paul Debevec. Rendering Synthetic Objects Into Real Scenes: Bridging Tradi-tional and Image-Based Graphics With Global Illumination and High DynamicRange Photography. Computer Graphics, 32(Annual Conference Series):189–198, 1998.

[12] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, WestleySarokin, and Mark Sagar. Acquiring the Reflectance Field of a Human Face.In Kurt Akeley, editor, SIGGRAPH 2000, Computer Graphics Proceedings, An-nual Conference Series, pages 145–156. ACM SIGGRAPH, Addison Wesley,July 2000.

[13] Paul Debevec and Jitendra Malik. Recovering High Dynamic Range RadianceMaps from Photographs. In Turner Whitted, editor, SIGGRAPH 97 ConferenceGraphics Proceedings, Annual Conference Series, pages 369–378. ACM SIG-GRAPH, Addison Wesley, August 1997.

[14] Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese,and Tim Hawkins. A Lighting Reproduction Approach to Live-Action Com-positing. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings,Annual Conference Series, pages 547–556. SIGGRAPH, ACM Press/ACM SIG-GRAPH, 2002.

[15] R. A. DeVore. Nonlinear approximation. Acta Numerica, 7:51–150, 1998.

[16] Philip Dutre. Mathematical Frameworks and Monte Carlo Algorithms for GlobalIllumination in Computer Graphics. PhD thesis, Katholieke Universiteit Leuven,Department of Computer Science, Leuven, Belgium, September 1996.

[17] Philip Dutre, Philippe Bekaert, and Kavita Bala. Advanced Global Illumination.A.K. Peters., 2003.

[18] Philip Dutre and Pieter Peers. Personal Communication. 2004.

[19] Sidney Fels, Eric Lee, and Kenji Mase. Techniques for Interactive Video Cu-bism. In ACM Publication, pages 368–370, 2000.

[20] James A. Ferwerda. Elements of Early Vision for Computer Graphics. IEEEComput. Graph. Appl., 21(5):22–33, 2001.

[21] Pascal Gautron, Jaroslav Krivanek, Sumanta Pattanaik, and Kadi Bouatouch. ANovel Hemispherical Basis for Accurate and Efficient Rendering. In Proceed-ings of the 15th Eurographics workshop on Rendering, pages 321–330. Euro-graphics Association, 2004.

Page 165: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

BIBLIOGRAPHY 153

[22] Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen.The Lumigraph. In Holly Rushmeier, editor, SIGGRAPH 96, Computer Graph-ics Proceedings, Annual Conference Series, pages 43–54. ACM SIGGRAPH,Addison Wesley, August 1996.

[23] Tim Hawkins, J Cohen, and Paul Debevec. A Photometric Approach to Digi-tizing Cultural Artifacts. In In 2nd International Symposium on Virtual Reality,Archaeology, and Cultural Heritage, Glyfada, Greece, November 2001., 2001.

[24] Tim Hawkins, Andreas Wenger, Chris Tchou, Andrew Gardner, Franske Gorans-son, and Paul Debevec. Animatable Facial Reflectance Fields. In Proceedingsof the 15th Eurographics workshop on Rendering, pages 309–319. EurographicsAssociation, 2004.

[25] Wolfgang Heidrich and Hans-Peter Seidel. Realistic, hardware-accelerated shad-ing and lighting. In SIGGRAPH 1999 Conference Proceedings, pages 171–178.ACM Press/Addison-Wesley Publishing Co., 1999.

[26] Insung Ihm, Sanghoon Park, and Rae Kyoung Lee. Rendering of spherical lightfields. In Yeong Gil Shin and James Kwangjune Hahn, editors, 5th Pacific Con-ference on Computer Graphics and Applications (Pacific Graphics ’97), AnnualConference Series, pages 59–69. IEEE Computer Society, 1997.

[27] James T. Kajiya. The rendering equation. In Proceedings of the 13th annualconference on Computer graphics and interactive techniques, pages 143–150.ACM Press, 1986.

[28] Jan Kautz, Peter-Pike Sloan, and John Snyder. Fast, arbitrary BRDF shading forlow-frequency lighting using spherical harmonics. In Proceedings of the 13thEurographics workshop on Rendering, pages 291–296. Eurographics Associa-tion, 2002.

[29] Thomas Kollig and Alexander Keller. Efficient illumination by high dynamicrange images. In Proceedings of the 14th Eurographics workshop on Rendering,pages 45–50. Eurographics Association, 2003.

[30] Eric Lafortune. Mathematical Models and Monte Carlo Algorithms for Physi-cally Based Rendering. PhD thesis, Katholieke Universiteit Leuven, Departmentof Computer Science, Leuven, Belgium, February 1996.

[31] S. Lee, G. Wolberg, and S. Y. Shin. Scattered Data Interpolation with Multi-level B-Splines. IEEE Transactions on Visualization and Computer Graphics,3(3):228–244, 1997.

[32] Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, andHans-Peter Seidel. Image-based reconstruction of spatial appearance and geo-metric detail. ACM Trans. Graph., 22(2):234–257, 2003.

Page 166: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

154 BIBLIOGRAPHY

[33] Marc Levoy and Pat Hanrahan. Light Field Rendering. In Holly Rushmeier, ed-itor, SIGGRAPH 96 Conference Proceedings, Annual Conference Series, pages31–42. ACM SIGGRAPH, Addison Wesley, August 1996.

[34] Zhoulin Lin, Tien-Tsin Wong, and Heung-Yeung Shum. Relighting with theReflected Irradiance field: Representation, Sampling and Reconstruction. IEEEComputer Vision and Pattern Recognition, 1:561–567, December 2001.

[35] Tom Malzbender, Dan Gelb, and Hans Wolters. Polynomial Texture Maps. InEugene Fiume, editor, SIGGRAPH 2001, Computer Graphics Proceedings, An-nual Conference Series, pages 519–528. ACM Press / ACM SIGGRAPH, 2001.

[36] Wojciech Matusik, Matthew Loper, and Hanspeter Pfister. Progressively-RefinedReflectance functions from Natural Illumination. In Proceedings of the 15th Eu-rographics workshop on Rendering, pages 299–308. Eurographics Association,2004.

[37] Wojciech Matusik, Hanspeter Pfister, Addy Ngan, Paul Beardsley, Remo Ziegler,and Leonard McMillan. Image-based 3D photography using opacity hulls. InProceedings of the 29th annual conference on Computer graphics and interac-tive techniques, pages 427–437. ACM Press, 2002.

[38] Sebastian Magda Melissa L Koudelka, Peter N Belhumeur and David J. Krieg-man. Image-based Modeling and Rendering of Surfaces with Arbitrary BRDFs.In Computer Vision and Pattern Recognition ( CVPR’01), Computer GraphicsProceedings, Annual Conference Series, 2001.

[39] Shree Nayar, Peter Belhumeur, and Terry Boult. Lighting Sensitive Display.conditional accepted to ACM Transactions on Graphics, ??(?):??–??, 2003.

[40] Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. All-Frequency Shadows UsingNon-Linear Wavelet Lighting Approximation. ACM Trans. Graph., 22(3):376–381, 2003.

[41] Jeffry S. Nimeroff, Eero Simoncelli, and Julie Dorsey. Efficient Re-rendering ofNaturally Illuminated Environments. In Fifth Eurographics Workshop on Ren-dering, pages 359–373, Darmstadt, Germany, 1994.

[42] Ko Nishino and Shree Nayar. The World in an Eye. In Proceedings of ComputerVision and Pattern Recognition CVPR ’04, pages 444–451. IEEE Press, 2004.

[43] Victor Ostromoukhov, Charles Donohue, and Pierre-Marc Jodoin. Fast Hierar-chical Importance Sampling with Blue Noise Properties. ACM Transactions onGraphics, 23(3), August 2004. Proc. SIGGRAPH 2004.

[44] Wai-Man Pang, Tien-Tsin Wong, and Pheng-Ann Heng. Estimating Light Vec-tors in Real Time. IEEE Computer Graphics & Applications, 24(3):36–43, 2004.

[45] Pieter Peers and Philip Dutre. Accurate Image Based Relighting through Opti-mization,. Technical Sketch, SIGGRAPH2002, 2002.

Page 167: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

BIBLIOGRAPHY 155

[46] Pieter Peers and Philip Dutre;. Wavelet environment matting. In Proceedingsof the 14th Eurographics workshop on Rendering, pages 157–166. EurographicsAssociation, 2003.

[47] Thomas Porter and Tom Duff. Compositing Digital Images. Computer Graphics,18(3):253–259, July 1984.

[48] Ravi Ramamoorthi and Pat Hanrahan. An Efficient Representation for IrradianceEnvironment Maps. In SIGGRAPH 2001, Computer Graphics Proceedings, An-nual Conference Series, pages 497–500, 2001.

[49] Peter Schroder and Wim Sweldens. Spherical Wavelets: Efficiently representingfunctions of the sphere. In SIGGRAPH 95, Computer Graphics Proceedings,Annual Conference Series, pages 161–172. Addison Wesley, August 1995.

[50] Peter Schroder, Wim Sweldens, Michael Cohen, Tony DeRose, and DavidSalesin. Wavelets in Computer Graphics, SIGGRAPH 96 Course Notes, 1996.

[51] Peter Shirley and Kenneth Chiu. A low distortion map between disk and square.J. Graph. Tools, 2(3):45–52, 1997.

[52] Peter-Pike Sloan, Jan Kautz, and John Snyder. Precomputed radiance transferfor real-time rendering in dynamic, low-frequency lighting environments. ACMTransactions on Graphics, 21(3):527–536, July 2002.

[53] SpheronVR. www.spheron.com. 2004.

[54] Eric J. Stollnitz, Tony D. DeRose, and David H. Salesin. Wavelets for computergraphics: theory and applications. Morgan Kaufmann Publishers, Inc., 1996.

[55] J. Unger, A. Wenger, T. Hawkins, A. Gardner, and P. Debevec. Capturing andrendering with incident light fields. In Proceedings of the 13th Eurographicsworkshop on Rendering, pages 141–149. Eurographics Association, 2003.

[56] Michael Unser and Thierry Blu. Mathematical Properties of the JPEG2000Wavelet Filters. IEEE Transactions on Image Processing, 12(9):1080–1090,September 2003.

[57] Eric Veach. Robust Monte Carlo methods for light transport simulation. PhDthesis, Stanford university, Department of Computer Science, Stanford, CA, De-cember 1997.

[58] Greg Ward, Paul Debevec, Rod Bogaert, Dan Lemmon, and Frank Vitz. HDRIand Image-based lighting, SIGGRAPH 2003 Course Notes, 2003.

[59] Tien-Tsin Wong, Pheng-Ann Heng, Siu-Hang Or, and Wai-Yin Ng. Image-basedRendering with Controllable Illumination. In Eurographics Rendering Workshop1997, pages 13–22. Springer Wien, June 1997.

Page 168: A PRACTICAL FRAMEWORK FOR FIXED VIEWPOINT IMAGE … · Viewpoint Image-Based Relighting Vincent Masselus Department of Computer Science, Katholieke Universiteit Leuven ABSTRACT Computer

156 BIBLIOGRAPHY

[60] Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp,David H. Salesin, and Werner Stuetzle. Surface Light Fields for 3D Photogra-phy. In Kurt Akeley, editor, SIGGRAPH 2000, Computer Graphics Proceedings,Annual Conference Series, pages 287–296. SIGGRAPH, ACM Press / ACMSIGGRAPH, 2000.

[61] Gnther Wyszecki and W. S. Stiles. Color Science : Concepts and Methods,Quantitative Data and Formulae. Wiley Series, 1982.

[62] Douglas E. Zongker, Dawn M. Werner, Brian Curless, and David H. Salesin.Environment matting and compositing. In Proceedings of the 26th annual con-ference on Computer graphics and interactive techniques, pages 205–214. ACMPress/Addison-Wesley Publishing Co., 1999.