Neural Rendering in Action: Build Modern Rendering Systems with Differentiable Rendering, NeRFs, and Gaussian Splatting

Learn to build neural rendering systems from scratch, turning 2D images into interactive 3D scenes with differentiable rendering, NeRFs, 3D Gaussian Splatting, and diffusion models

This book  teaches readers how to move between theory and code, between image formation and optimization, and between classical structure and learned flexibility. For a field defined by that convergence, this is exactly the right kind of guide. -  Alexey Medvedev, AR Tech Lead, Meta

Key Features

Build complete implementations of differentiable renderers, NeRFs, and Gaussian Splatting Learn inverse rendering, neural scene representations, and generative 3D models from text prompts Master Slang shaders and neural compression with practical recipes for modern graphics pipelines Book Description

As AI transforms 3D graphics, neural rendering has become essential for graphics engineers and ML developers. This practical guide takes you from computer vision fundamentals to modern neural rendering systems with complete PyTorch and Slang implementations.

The book begins with inverse rendering, teaching you to reconstruct 3D scenes from 2D images using Structure from Motion. You'll build a strong foundation in camera geometry, epipolar constraints, light transport, and BRDFs before setting up Slang and SlangPy to write differentiable shaders and implement neural texture compression.

Next, you'll explore neural scene representations, progressing from voxel grids and point clouds to neural fields. You'll implement Neural Radiance Fields (NeRFs), covering differentiable volume rendering, positional encoding, and training pipelines for photorealistic novel view synthesis. You'll then learn 3D Gaussian Splatting, representing scenes as millions of transparent Gaussians with differentiable rasterization for real-time rendering.

The final chapter explores generative 3D using diffusion models such as Stable Diffusion for single-image reconstruction and texture and material generation. By the end, you'll be able to build complete neural rendering systems from scratch, guided by the authors' decades of real-world rendering experience.

What you will learn

Master the mathematical foundations of inverse graphics Understand neural scene reconstruction and novel view synthesis Build differentiable rendering systems from scratch Implement Neural Radiance Fields (NeRFs) Optimize textures and mipmaps using neural compression Create real-time neural graphics with Gaussian Splatting Integrate neural rendering techniques into existing graphics pipelines Generate images and materials using diffusion models Who this book is for

This book is for graphics engineers, technical artists, and VFX developers with a background in traditional rendering who want to bring deep learning into real-time graphics. It's equally suited to machine learning and research engineers looking to apply their skills to 3D content generation. A working knowledge of computer graphics and basic machine learning is recommended.

Table of Contents

Introduction to Inverse Rendering 3D Scene Understanding Neural Rendering Toolkit Implementing Texture Compression using Neural Networks Implementing a Differentiable Renderer Representing Geometry and Appearance with Neural Networks Implementing a Simple Neural Radiance Cache Building and Training a NeRF Model Real-time rendering with 3D Gaussian Splatting Learning the Fundamentals of Diffusion Models