CS6630 Realistic Image Synthesis

Cornell University, Fall 2015

T/Th 10:10am, Phillips 407

Instructor: Steve Marschner (Office hours: Tuesday 3–4, Thursday 1:15–2:15 in Gates 313)

PhD TA: Pramook Khungurn (Office hour: Monday 3–4pm at Gates G11)

Schedule

date		topic	reading	assignments
25	Aug	intro slides
27	Aug	radiometry notes	Preisendorfer 1976
1	Sep	radiometry & BRDF
3	Sep	direct illumination notes		hw1 out
8	Sep	monte carlo integration	Feller 1968, Feller 1971
10	Sep	monte carlo | sampling PDFs demo		hw1 due, pa1 out
15	Sep	sampling from PDFs
17	Sep	sampling from PDFs
22	Sep	direct illumination slides
24	Sep	direct illumination		pa1 due, pa2 out
29	Sep	PA2 discussion (optional) notes notes
1	Oct	multiple importance sampling | rendering equation slides	Kajiya 1986, Veach & Guibas 1995
6	Oct	path tracing | microfacet models slides notes slides	Walter et al. 2007
8	Oct	microfacet		pa2 due
13	Oct	—Fall Break—
15	Oct	radiative transfer notes		pa3 out
20	Oct	Zechen Zhang Ning Wang	Lehtinen et al. 2011 Oren & Nayar 1995
22	Oct	radiative transfer
27	Oct	James Briggs Bryce Evans	Sadeghi et al. 2012 Ngan et al. 2005
29	Oct	volume path tracing notes
3	Nov	Suren Jayasuriya Eric Gao	Debevec et al. 2000 Hawkins et al. 2005
5	Nov	Eric Lee Eston Schweickart	Peers et al. 2009 Walter et al. 2005
10	Nov	volume path tracing		proposals due
12	Nov	Olufemi Adegunloye Ryan Hall	Crassin et al. 2009 d'Eon et al. 2007	pa3 due
17	Nov	path integral formulation	Veach 1997
19	Nov	Kristen Crasto Andrew Wolfers	Xu et al. 2001 Zhao et al. 2011
24	Nov	bidirectional path tracing	Veach 1997
26	Nov	—Thanksgiving—
1	Dec	Stephen McDowell Rahul Chatterjee	Jensen 1996 Novak et al. 2012
3	Dec	Rundong Wu Fujun Luan	Křivánek et al. 2014 Georgiev et al. 2012

Coursework

There will be 3 programming assignments during the semester, based on a new renderer called Wakame, a port of Wenzel Jakob's Nori 2 instructional renderer. Some have associated written assignments. Programming assignments may be done alone or in pairs.

The last part of the semester will be devoted to a final project, in which students implement a substantial state-of-the-art extension to Wakame.

During the course of the semester each student will present a paper from the rendering research literature. This will be singly or in pairs depending on the course population.

About CS6630

CS6630 is an introduction to physics-based rendering at the graduate level. Starting from the fundamentals of light transport we will look at formulations of the Rendering Equation, and a series of Monte Carlo methods, from sequential sampling to multiple importance sampling to Markov Chains, for solving the equation to make pictures. We'll look at light reflection from surfaces and scattering in volumes, illumination from luminaires and environments, and diffusion models for translucent materials. We will build working implementations of many of the algorithms we study, and learn how to make sure they are actually working correctly. It's fun to watch integrals and probability distributions transform into photographs of a slightly too perfect synthetic world.

Questions, help, discussion: The instructor and TA are available to answer questions, advise on projects, or just to discuss interesting topics related to the class at office hours and by appointment as needed. For electronic communication we are using Piazza (handy link also at the top of this page).

Academic integrity: We assume the work you hand in is your own, and the results you hand in are generated by your program. You're welcome to read whatever you want to learn what you need to do the work, but we do expect you to build your own implementations of the methods we are studying. If you're ever in doubt, just include a citation in your code or report indicating where some idea came from, whether it be a classmate, a web site, another piece of software, or anything—this always maintains your honesty, whether the source was used in a good way or not. The principle is that an assignment is an academic document, like a journal article. When you turn it in, you are claiming that everything in it is your original idea (or is original to you and your partner, if you're handing in as a pair) unless you cite a source for it.

School can be stressful, and your coursework and other factors can put you under a lot of pressure, but that is never a reason for dishonesty. If you feel you can't complete the work on your own, come talk to the professor or the TAs, or your advisor, and we can help you figure out what to do. Think before you hand in!

Clear-cut cases of dishonesty will result in failing the course.

For more information see Cornell's Code of Academic Integrity.

Collaboration: You are welcome (encouraged, even) to discuss projects among yourselves in general terms. But when it comes to writing up the homeworks or implementing the projects, you need to be doing the work yourself (or only with your partner if you are doing a project as a pair). In particular, it's never OK to directly or indirectly copy parts of one person's or team's writeup, code, or results into another's, even if the general solution was worked out together.

Reading

This is a collection of the readings for the class, many of which are assigned as reading for a particular lecture or as a paper for student presentation. Those that are assigned readings sometimes come with remarks about what part you need to read; don't forget to check that before you start!

[Agarwala 2007] Agarwala, A. 2007. “Efficient Gradient-Domain Compositing Using Quadtrees.” ACM Trans. Graph. 26, 3, Article
94 (July 2007), 5 pages.

[Agarwala et al. 2004] Agarwala, A. et al. “Interactive digital photomontage.” ACM Trans. on Graphics, 23(3):294--302, Aug. 2004

[Arbree 2009] Arbree, A. “Scalable and Heterogeneous Rendering of Subsurface Scattering Materials.” PhD Thesis, Cornell University, August 2009

This thesis on rendering translucent materials contains a nice explanation of various boundary conditions, including a derivation of the widely used dipole approximation.

[Arvo 1995] Arvo, James. “Analytic Methods for Simulated Light Transport.” PhD Thesis, Yale University, December 1995

This thesis includes a number of mathematical methods applied to rendering; for this course the relevant parts are the excellent measure-theoretic treatment of radiometry and the operator formulation of global illumination.

[Ashikhmin & Shirley 2000] Michael Ashikhmin and Peter Shirley. “A microfacet-based BRDF generator.” SIGGRAPH 2000.

[Ben-Artzi et al. 2008] Ben-Artzi, A., Egan, K., Ramamoorthi, R., & Durand, F. “A precomputed polynomial representation for interactive BRDF editing with global illumination.” ACM Transactions on Graphics, 27(2) (2008).

[Bouthors et al. 2008] Bouthors, A., F. Neyret, N. Max, E. Bruneton, C. Crassin. “Interactive multiple anisotropic scattering in clouds.” In Proceedings of I3D 2008.

[Blinn 1977] James F. Blinn. “Models of Light Reflection for Computer Synthesized Pictures.” In Proceedings of SIGGRAPH 1977.

[Brown 1980] Gary S. Brown. “Shadowing by Non-Gaussian Random Surfaces.” IEEE Transactions on Antennas and Propagaion 28:6 (1980).

[Cerezo et al. 2005] Cerezo, Eva and Perez-Cazorla, Frederic and Pueyo, Xavier and Seron, Francisco and Sillion, Francois. "A Survey on Participating Media Rendering Techniques" in the Visual Computer. 2005

[Cline et al. 2005] Cline, D., Talbot, J., & Egbert, P. “Energy redistribution path tracing.” ACM Transactions on Graphics (TOG), 24(3) (2005).

[Cook & Torrance 1981] Robert L. Cook and Kenneth E. Torrence. “A Reflectance Model for Computer Graphics.” Computer Graphics. 14(3), pp. 307-316. 1981.

[Crassin et al. 2009] Crassin, C., F. Neyret, S. Lefebvre, and E. Eisenmann. “GigaVoxels: ray-guided streaming for efficient and detailed voxel rendering.” In I3D 2009.

[Debevec & Malik 1997] Debevec, P. E., AND Malik, J. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 (August 1997), pp. 369--378.

[Debevec et al. 2000] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. “Acquiring the reflectance field of a human face” SIGGRAPH 2000.

[Drebin 1988] (acm) Drebin, R., L. Carpenter, and P. Hanrahan. “Volume rendering.” In SIGGRAPH 1988.

[Durand & Dorsey 2002]Durand, F. and Dorsey, J. 2002. "Fast bilateral filtering for the display of high-dynamic-range images." in Proceedings of SIGGRAPH 2002, Annual Conference Series, ACM, 257-265.

[d'Eon et al. 2007] d'Eon, E., D. Luebke, E. Enderton. “Efficient Rendering of Human Skin” In Eurographics Symposium on Rendering 2007.

[Fattal et al. 2002]Fattal, Raanan, Lischinski, Dani, and Werman, Micheal . Gradient domain high dynamic range compression. In Proceedings of ACM SIGGRAPH 2002, Computer Graphics Proceedings, Annual Conference Series. ACM Press / ACM SIGGRAPH, July 2002. 10, 13, 15

[Feller 1968] William Feller. “Chapter IX: Random Variables; Expectation.” An Introduction to Probability Theory and Its Applications, Vol.I. 3rd Edition. John Wiley & Sons, Inc.: New York, 1968.

[Feller 1971] William Feller. “Chapter I: The Exponential and the Uniform Densities” and “Chapter III: Densities in Higher Dimensions; Normal Densities and Processes.” An Introduction to Probability Theory and Its Applications, Vol.II. 2nd Edition. John Wiley & Sons, Inc.: New York, 1971.

[Georgiev et al. 2012] Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. “Light transport simulation with vertex connection and merging.” SIGGRAPH 2012.

[van Ginneken et al. 1998] Bram van Ginneken, Marigo Stavridi, and Jan J. Koenderink. “Diffuse and specular reflectance from rough surfaces.” Applied Optics 37:1 (1998)

[Glassner 1995a] Andrew S. Glassner. “Chapter 12: Energy Transport.” Principles of Digital Image Synthesis. Morgan-Kaufman, 1995.

Glassner explains the volume rendering equation from the ground up, at a leisurely pace and in language familiar to graphics people.

[Glassner 1995b] Andrew S. Glassner. “Chapter 15: Shading.” Principles of Digital Image Synthesis. Morgan-Kaufman, 1995.

Glassner covers the basics of BRDFs and shading, and discusses many shading models in a fair amount of detail.

[Goesele et al. 2004] Michael Goesele, Hendrik P. A. Lensch, Jochen Lang, Christian Fuchs, Hans-Peter Seidel. “DISCO: acquisition of translucent objects.” ACM Transactions on Graphics. 23(3), pp. 835-844, 2004.

[Gortler et al. 2006]Gortler, S.,  Grzeszczuk R., Szeliski, R, and Cohen,M.,  “The Lumigraph,” Proc. ACM SIGGRAPH ’96 (1996), pp. 43–54.

[Hanrahan & Krueger 1993] Pat Hanrahan, Wolfgang Krueger. “Reflection from Layered Surfaces Due to Subsurface Scattering.” (Proceedings of SIGGRAPH 93). pp. 165-174, 1993.

[Hawkins et al. 2005] Tim Hawkins, Per Einarsson, and Paul Debevec. A Dual Light Stage. Proceedings of EGSR 2005.

[Ikits et al. 2004] Milan Ikits, Joe Kniss, Aaron Lefohn, and Charles Hansen. “Volume Rendering Techniques” In Randima Fernando, ed., GPU Gems, Pearson Education, 2004.

[Ishimaru 1978a] Akira Ishimaru. “Transport Theory of Wave Propagation in Random Particles,” Chapter 7 in Wave Propagation and Scattering in Random Media, Volume 1. Academic Press, 1978.

[Ishimaru 1978b] Akira Ishimaru. “Approximate Solutions for Tenuous Medium,” Chapter 8 in Wave Propagation and Scattering in Random Media, Volume 1. Academic Press, 1978.

[Ishimaru 1978c] Akira Ishimaru. “Diffusion Approximation,” Chapter 9 in Wave Propagation and Scattering in Random Media, Volume 1. Academic Press, 1978.

[Jakob et al. 2010] Wenzel Jakob, Adam Arbree, Jonathan T. Moon, Kavita Bala, and Steve Marschner. “A Radiative Transfer Framework for Rendering Materials with Anisotropic Structure” SIGGRAPH 2010.

This paper's main topic is light transport in anisotropic materials, but the accompanying tech report contains nice derivations of the standard case as well, for the diffusion approximation and the dipole BSSRDF.

[Jensen 1996] Jensen, H. W. Global illumination using photon maps. In Rendering Techniques '96, pages 21--30, 1996

[Jensen & Christensen 1998] Henrik W. Jensen and Per H. Christensen. “Efficient Simulation of Light Transport in Scene with Participating Media using Photon Maps.”(Proceedings of SIGGRAPH 98). pp. 311-320, 1998.

[Jensen et all. 2001] Henrik Wann Jensen, Stephen R. Marschner, Marc Levoy, Pat Hanrahan. “A Practical Model for Subsurface Light Transport.” (Proceedings of ACM SIGGRAPH 2001). pp. 511-518, 2001.

[Jensen & Buhler 2002] Henrik Wann Jensen and Juan Buhler. “A Rapid Hierarchical Rendering Technique for Translucent Materials.” In SIGGRAPH 2002.

[Kajiya 1986] James T. Kajiya. “The Rendering Equation.” Computer Graphics (Proceedings of SIGGRAPH 86). 20(4), pp. 143-150, 1986.

[Kajiya & von Herzen] J. Kajiya and B. Von Herzen, “Ray tracing volume densities”, SIGGRAPH 1984, July 1984, pp. 165-174.

This seminal work on light transport for graphics formalizes rendering using the Rendering Equation, which we now take as the theoretical basis for rendering of surfaces.

[Keller 1997] Keller, A. “Instant radiosity.” SIGGRAPH 1990.

[Kindlmann & Durkin 1998] Kindlmann, G., and J. W. Durkin. “Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering.” In IEEE Symposium on Volume Visualization 1998.

[Koenderink et al. 1999] Koenderink, J.J., A.J. van Doorn, K.J. Dana, and S. Nayar. “Bidirectional Reflection Distribution Function of Thoroughly Pitted Surfaces.” International Journal of Computer Vision 31 (1999).

[Kniss et al. 2002] Kniss, J., G. Kindlmann, and C. Hansen. “Multidimensional Transfer Functions for Interactive Volume Rendering.” IEEE TVCG 8:3 (2002).

[Kniss et al. 2003] Kniss, J., S. Premoze, C. Hansen, and P. Shirley. “A model for volume lighting and modeling.” IEEE TVCG 9:2 (2003).

[Křivánek et al. 2014] Jaroslav Křivánek, Iliyan Georgiev, Toshiya Hachisuka, Petr Vévoda, Martin Šik, Derek Nowrouzezahrai, and Wojciech Jarosz. “Unifying points, beams, and paths in volumetric light transport simulation.” SIGGRAPH 2014.

[Lacroute & Levoy 1994] Lacroute, P. and M. Levoy. “Fast Volume Rendering Using a Shear-Warp Factorization of the Viewing Transformation.” In SIGGRAPH 1994.

[Lafortune 1996] Lafortune, Eric. “Mathematical Models and Monte Carlo Algorithms for Physically Based Rendering.” PhD Thesis, Katholieke Universiteit Leuven, February 1996.

This thesis introduces bidirectional path tracing (see also Veach & Guibas), and also contains nice explanations of all the radiometric and probabilistic ideas leading up to it.

[Lehtinen et al. 2011] Lehtinen, J., Aila, T., Chen, J., Laine, S., Durand, F., Lehtinen, J., et al. “Temporal light field reconstruction for rendering distribution effects.” SIGGRAPH 2011.

[Lehtinen et al. 2012] Lehtinen, J., Aila, T., Laine, S., & Durand, F. “Reconstructing the indirect light field for global illumination.” SIGGRAPH 2012.

[Lehtinen et al. 2013] Lehtinen, J., Karras, T., Laine, S., Aittala, M., Durand, F., & Aila, T. “Gradient-domain metropolis light transport.” ACM Transactions on Graphics, 32(4) (2013).

[Lensch et all. 2003] Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, Hans-Peter Seidel. “Image-based reconstruction of spatial appearance and geometric detail.” ACM Transactions on Graphics. 22(2), pp. 234-257, 2003.

[Levoy 1988] Levoy, M. “Display of surfaces from Volume Data.” IEEE Computer Graphics & Applications 8:3 (1988).

[Levoy et al. 2006] Levoy, M., Ng, R., Adams, A., Footer, M., Horowitz, M., “Light field microscopy,” ACM Transactions on Graphics (Proc. SIGGRAPH 2006), Vol. 25, No. 3, 2006.

[Levoy & Hanrahan 1996] Levoy, Marc and Hanrahan, Pat. Light field rendering. In Proceedings of SIGGRAPH '96, Annual Conference Series, ACM, 31-42. 1996

[Matusik et al. 2003] Wociech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan, “A data-driven reflectance model,” SIGGRAPH 2003. [author page]

[Mitsunaga & Nayar 1999] Mitsunaga and S. K. Nayar. Radiometric self calibration. In Proc CVPR, volume 2, pages 374--380, June 1999

[Nicodemus et al. 1977] F.E. Nicodemus, J.C. Richmond, J.J. Hsia, I.W. Ginsberg, and T. Limperis. Geometrical Considerations and Nomenclature for Reflectance. National Bureau of Standards (U.S.) monograph, issued October 1977.

This monograph is the origin of the terminology and (nearly) the notation that we use for surface reflection in graphics, including the BRDF and the BSSRDF. Sections I and II are the most relevant for rendering; Section IV is required reading for anyone who is going to measure light reflection.

[Ngan et al. 2005] Ngan, A., Durand, F., & Matusik, W. “Experimental analysis of BRDF models.” Proceedings of the 2005 Eurographics Symposium on Rendering. [author page]

[Novak et al. 2012] Jan Novák, Derek Nowrouzezahrai, Carsten Dachsbacher, and Wojciech Jarosz. “Virtual ray lights for rendering scenes with participating media,” SIGGRAPH 2012.

[Oren & Nayar 1995] Michael Oren and Shree K. Nayar. “Generalization of the Lambertian Model and Implications for Machine Vision” International Journal of Computer Vision 14:3 (1995).

[Peers et al. 2009] Pieter Peers, Dhruv K Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik, Ravi Ramamoorthi, and Paul Debevec. “Compressive Light Transport Sensing,” ACM Transactions on Graphics 28:1 (2009).

[Poulin & Fournier 1990] Pierre Poulin and Alain Fournier. “A model for anisotropic reflection,” SIGGRAPH 1990.

[Preisendorfer 1976] Rudolph W. Preisendorfer. Hydrologic Optics, Vol. I. Section 1.1.

This section provides clear, operational definitions of all the important radiometric quantities. When you read it for class, feel free to skim over the parts that are specific to the hydrologic context.

The rest of Preisendorfer's book contains similarly clear explanations of many of the classical radiative transfer results we have borrowed for graphics.

[Premoze et al. 2004] Simon Premoze, Michael Ashikhmin, Jerry Tessendorf, Ravi Ramamoorthi, Shree Nayar. “Practical Rendering of Multiple Scattering Effects in Participating Media.” Rendering Techniques 2004: 15th Eurographics Workshop on Rendering. pp. 363-374, 2004.

[Sadeghi et al. 2012] Sadeghi, I., Munoz, A., Laven, P., Jarosz, W., Seron, F., Gutierrez, D., & Jensen, H. W. “Physically-based simulation of rainbows.” ACM Transactions on Graphics, 31(1), (2012).

[Shirley et al. 1996] Peter Shirley, Changyaw Wang, Kurt Zimmerman. “Monte Carlo Techniques for Direct Lighting Calculations.” ACM Transactions on Graphics. 15(1), pp. 1-36, 1996.

This paper discusses the details of choosing random shadow rays to sample planar or spherical luminaires.

[Smith 1967] Bruce G. Smith. “Geometrical Shadowing of a Random Rough Surface.” IEEE Transactions on Antennas and Propagation 15:5 (1967)

[Szeliski & Shum 1997] R. Szeliski and H. Shum. Creating full view panoramic image mosaics and environment maps. In Proc. of SIGGRAPH, pages 251--258, 1997

[Torrance & Sparrow 1967] K.E. Torrance and E.M. Sparrow. “Theory for Off-specular Reflection from Roughened Surfaces.” Journal of the Optical Society of America 57:9 (1967).

[Veach & Guibas 1994] Eric Veach and Leonidas J. Guibas. “Bidirectional Estimators for Light Transport” (Proceedings of EGRW 1994). 1995.

[Veach & Guibas 1995] Eric Veach and Leonidas J. Guibas. “Optimally Combining Sampling Techniques for Monte Carlo Rendering” (Proceedings of SIGGRAPH 95). pp. 419-428, 1995.

[Veach & Guibas 1997] Veach, E., & Guibas, L. “Metropolis light transport.” SIGGRAPH 1997.

[Veach 1997] Eric Veach. “Robust Monte Carlo Methods for Light Transport Simulation.” PhD Thesis, Stanford University, December 1997.

This thesis introduces a number of important Monte Carlo rendering methods, including multiple importance sampling, bidirectional path tracing (see also Lafortune & Willems), and Metropolis Light Transport.

[Walter et al. 2005] Walter, B., Fernandez, S., Arbree, A., Bala, K., Donikian, M., Greenberg, D. P., et al. “Lightcuts: a scalable approach to illumination.” ACM Transactions on Graphics, 24(3) (2005).

[Walter et al. 2007] Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. “Microfacet Models for Refraction through Rough Surfaces.” Eurographics Symposium on Rendering 2007.

The slides I presented in class came originally from the presentation of this paper. The notation and presentation of the microfacet framework is similar, though you won't see the delta functions that appear in the paper in class.

[Walter et al. 2009] Walter, B., Zhao, S., Holzschuch, N., & Bala, K. “Single scattering in refractive media with triangle mesh boundaries.” ACM Transactions on Graphics, 28(3) (2009).

[Wyman et al. 2006] Chris Wyman, Steven Parker, Peter Shirley, and Charles Hansen. “Interactive display of isosurfaces with global illumination.” IEEE TVCG 12:2 (2006).

[Xu et al. 2001] Xu, Y-Q., Chen, Y., Lin, S., Zhong, H., Wu, E., Guo, B., and Shum, H-Y. “Photorealistic rendering of knitwear using the Lumislice.” In SIGGRAPH 2001.

[Zhao et al. 2011] Shuang Zhao, Wenzel Jakob, Steve Marschner, and Kavita Bala. “Building Volumetric Appearance Models of Fabric using Micro CT Imaging,” ACM Transactions on Graphics (SIGGRAPH 2011), 30(4), July 2011.

[Zhao et al. 2012] Shuang Zhao, Wenzel Jakob, Steve Marschner, and Kavita Bala. “Structure-aware Synthesis for Predictive Woven Fabric Appearance,” ACM Transactions on Graphics (SIGGRAPH 2012), 31(4), July 2012