On March 13th, Epic Games the developers of the Unreal Engine showcased a technology demo developed using Unreal Engine 5 running on the next generation PlayStation 5 console. During the short video, Technical Director of Graphics Brian Karis and Special Projects Art Director Jerome Platteaux tout several new features or technologies slated to ship with Unreal Engine 5 (of which only two would really have to do with 3D computer graphics directly) but only very lightly touch on what they actually do, never going into detail on that, and are even more tight fisted with any information at all regarding the actual implementation of the new features, how they do what they do.
Thus, we’re mainly left to speculate on what the new features are, really, and what’s driving them.
The first feature the presentation touches on is ‘Dynamic Global Illumination’. This is dubbed ‘Lumen’ and is described as being ‘beautiful bounce lighting’ that’s instantaneous, which presumably means it’s sufficiently computationally light so as to be a real time effect rather than having to be pre-computed beforehand and remain static during gameplay (such as the scenery shadows, particularly from the sun, which are actually static level textures in older games, such as games based on Valve’s Source and GoldSource engines).
Could this be dynamic volumetric lighting with extra steps to create the impression of ray or path traced traced illumination? Or maybe even just dynamic volumetric lighting with a post-processing bloom effect? Or is it actual but limited and rationed path tracing with a very limited number of bounces? Even as low as 6 bounces might go a long way towards lighting realism without being exorbitantly taxing on hardware, if used discerningly and sparingly.
Before the viewer has even had a chance to begin to ponder what Lumen might be or how it might work, Brian and Jerome quickly switch to talking about what they later dub ‘Nanite’, which they initially refer to as ‘truly virtualized geometry’, as if that’s supposed to mean anything to someone watching the presentation without any technical background or context provided, and is not just marketing gobbledygook. The presenters do mention that this feature would allow for very or extremely high polygon count assets to be imported into Unreal Engine 5, without requiring the usual and industry standard preparatory steps of having to create normal maps from the original, high polygon count version of a given asset which are then projected unto and applied to a significantly lower polygon count version of the same asset, which is what normally goes into and is actually used in the game itself, rather than the original artwork.
Brian explains that the main benefit of the ‘Nanite’ feature is that artists would no longer need to limit their artistic freedom over concern with keeping the polygon count of their models (and especially the peak cumulative number of triangles which might be inside of the virtual camera’s view frustum at any given moment during gameplay) in check so as to not cause an excessive number of draw calls (to the GPU) by the CPU to be required (which can cause the CPU to become a bottleneck for the GPU and thus degrade performance and the gameplay experience, especially if either the CPU or GPU is not a current generation AMD part or not an AMD part at all – as only these are capable of communicating with each other using PCI Express 4 if fitted to a PCI Express 4 capable mainboard – and/or if the game engine does not have its draw calls properly multithreaded or is using an outdated API, such as Direct3D 11 or earlier or OpenGL, which handle draw calls in a single thread or make it difficult for developers to use several threads for handling draw calls to the GPU, rather than just one) or the large memory footprint which high poly count game assets can reach. Of course, beyond a certain number of polygons on screen at the same time, the GPU itself might start to take long enough to render each frame to where the game becomes unplayable. If system memory or video memory doesn’t become exhausted first.
Beyond, obviously, the game’s ability to deal with far more geometry on screen at the same time without degrading performance to where the game becomes unplayable, the Nanite feature is explained to have several different positive implications. One being reduced workload for 3D artists, who would no longer need to manually create several different versions, of varying degree of detail and complexity (as measured in the number of polygons the asset is comprised of), of the same game asset, so as to be able to selectively display one or another of these different versions of the same asset depending on the current distance of an instance of the asset from the player’s position, their position of their virtual camera inside of the game world or level.
This obviously incurs artists a sizeable amount of work duplication or redundant work and wasted time which would otherwise be entirely unnecessary if not for the need to cope with the limitations of the hardware the game they’re contributing art to will run on. Which is done by using lower detail, lower polygon count versions of a given art asset for a particular instance of said asset, if or when its distance from the player, in the game world or level, is sufficiently large such that the asset only takes a minute amount of space (and thus pixels) on the player’s screen and the player would be hard pressed to tell that anything was actually different in the complexity or about the quality of that particular instance of that particular game asset, when it’s at a considerable virtual distance from them, inside of the game level.
The other important implication of the Nanite feature is mentioned to be the ability to use CAD models as game assets (presumably after they’ve been tessellated into polygon meshes).
With current game industry asset generation practices and production practices and workflows, CAD model derived polygonal models, exported from CAD software, cannot be used as game assets themselves and actually need to be re-created from scratch by artists, who might only be able to use the CAD software exported polygon meshes as references or as input for automated, retopologizing software, which is able to take a fully triangulated mesh and convert it to a quadrilateral polygon based or dominant mesh, which is suitable for use in the rest of the industry standard and established game art asset production pipeline.
However, current automated solutions for converting a triangle only or triangle dominant polygon mesh into a quadrilateral only or quad dominant mesh, the latter which is suitable for use in the industry established game art asset production pipeline that includes such steps as creating a high polygon count and a low polygon count version of each individual asset (possibly along with several intermediary complexity versions in between as well), generating detailed normal maps from the high polygon version of the asset which are afterwards applied to the lower polygon count versions of the same so as to grant them a veneer of increased complexity or smoother edges (which would otherwise require additional geometry that isn’t actually there with the lower polygon count models), are not full proof and cannot successfully deal with all possible models and mode topology they may come across, thus still requiring a sizeable amount of manual post processing and correction of their output by a 3D artist.
If the Nanite feature of Unreal Engine 5 can truly deliver on the promise of doing away with the need to fake non-existent geometric detail in game art by way of generating normal maps from high poly count versions of assets to apply to lower poly count versions of the same assets, that would be a huge selling point for game studios as it would do away with a lot of tedious, repetitive and time consuming work their artists currently need to do to tone down insanely high polygon, ‘master copy’ original artwork and photogrammetry derived assets (assets derived from photographing or 3D scanning real world objects, scenery or pieces of scenery and inferring the geometric shape and the colour information, texture and pattern which a 3D asset would need to have to be able to mimic that shape in a virtual environment, such as a game) to where they’re of a manageable size and complexity for the game end consumer’s hardware to be able to deal with at a suitably high frame rate and resolution, as required for an enjoyable gameplay experience.
After a very brief and superficial mention of the promises of the ‘Nanite’ feature, which I tried to explain in more depth above, the presentation switches to purported in-engine footage from what would appear to be a cross between the Legend of Zelda and Rise of the Tomb Raider’s Prophet Tomb level.
The presenters go on to narrate the game footage, blithely making only passing mention of such things as ‘virtual textures’ without the slightest inkling of awareness of the need to qualify or explain the term to the audience, but do deign to add that the Nanite feature enables the rendering of an extremely high number of triangles (all 3D objects, at least when rendered in any games or 3D or CAD software interface viewport, are represented as fully triangulated 3D meshes; quads aren’t used in the actual rendering) very quickly, adding that there are over a billion triangles of ‘source geometry’ (again, not even a hint of an intent to explain or qualify the term) in each frame of the footage being shown, which Nanite ‘crunches down’ (whatever that means, we’ll speculate lower down) ‘losslessly’ (again, no hint of even an intent to explain this) to ‘around 20 million drawn triangles’.
Now that we’re done going over the scant, meagre information in the clip, which seems to be an infomercial without any real info, we have no choice but to turn to educated guesses and speculation as to its meaning and the implementation of the features of UE 5 being touted.
Let’s start with Lumen. I personally think it might be bidirectional path tracing, possibly biasing reflections towards directions and along paths which have already yielded routes successfully connecting light source and virtual camera in prior samplings, with a limited number of light bounces, per-pixel samples and employed discerningly and sparingly. However, that might just be giving UE 5 developers too much credit. Acceleration structure biased path tracing is an advanced light path tracing technique which is at the forefront of photorealistic rendering research and not yet a fully or effectively resolved problem. To presume that game engine developers have cracked problems which keep the brightest minds in 3D computer graphics scientific research awake at night is probably not a prudent assumption. It might also have a post-processing Bloom effect incorporated as well, the likes of which can be seen in the LightTracer path tracing software, for example.
At first, I thought the Nanite feature might actually be automated retopologizing of game assets (down to a far lower triangle count which is workable for realtime 3D graphics in a game setting), by way of UE5 included tools, along with automated normal map generation and application – to the lower triangle count version of the asset derived from the original or source version of it – as well.
However, upon further consideration, and in light of the use of the ‘lossless’ qualifier Brian used when describing Nanite’s ‘crunching down’, I think it’s just marketing gobbledygook for very high poly counts throughout the level and all assets in the level which are efficiently and effectively automatically sectioned up into rooms (or separated into containers, in the case of assets) and are only loaded into memory and rendered when the player can at least theoretically see them, in the case of assets, or is inside them or can see inside them from an adjacent room or section, in the case of rooms.
Basically overselling the concept of Object Container Streaming already used by the Star Citizen game engine, maybe combined with fast and effective hidden surface or hidden polygon early discard by way of primitive shaders, likely already provided at the hardware level by AMD’s next iteration of RDNA graphics, and available on the next generation XBOX console as well.
In fact, I even wonder if Unreal Engine developers are not actually trying to imply that even backface culling, which simply means that the GPU doesn’t even attempt to render any triangles in the scene which are facing away from the virtual camera and is a technique as old as rasterized 3D graphics or even older, is part of what makes Nanite an innovative feature.
In the last two decades, a new approach to marketing of technological products has started trending and appears to have become the new norm with tech companies where they seem to rely on the audience or general public’s ignorance about already existing technology which is branded and packaged or re-packaged as if it is a new technology being offered by a particular company for the first time, rather than just another proprietary implementation or bundling of already existing technologies, maybe even with little to no changes at all.
I believe the trend was started by Apple but nVidia took notes and followed suit, intentionally or short of deliberately conveying, through their marketing, the flawed perception that they actually invented physics engines with PhysX, ray tracing or ray tracing ASICs with their RTX lineup of cards or upscaling with DLSS (which I believe to be just checkerboard rendering with a possibly better post-processing filter for handling artifacting between adjacent pixels caused by temporal discrepancy of their respective rendering; checkerboard rendering itself is only a development or offshoot of the concept of interlacing, applied to 3D computer graphics). Consumers who don’t know any better will just assume these are all technologies which debuted with nVidia products. Maybe Epic Games took notes and are now attempting the same approach to marketing with Unreal Engine 5, trying to ride the coattails of Sony PlayStation 5 hype and (I believe rightly) assuming that console devoted fans (and nVidia fans for that matter as well) aren’t exactly well versed in the technology behind 3D computer graphics or the history of its development?
One has to wonder why the presentation is so scant on technical or implementation details. Protecting technical implementation details from industrial espionage or even just the broad concept of how the features work, under the hood, from being copied by other budding game engine developers? Or only providing just enough details to make the touted new features appear more impressive than they actually are, conveying the impression that they are qualitative shifts rather than just incremental or quantitative improvements? Who knows, they must even be just new implementations of already existing technologies or technical approaches, like efficient, fast, early hidden surface removal (for instance, by way of primitive shaders), Object Container Streaming and breaking levels or areas of the gameworld up in to discrete sections and only loading into memory and dealing with one, two or a few of the sections at any one given time rather than bogging the machine down with invisible, redundant details in other sections of the level or the game world. Only time will probably tell.
In any case, unlike has been bandied about, neither the Nanite nor the Lumen features of Unreal Engine 5 are likely to be exclusive to the PlayStation 5. Or, if they are, they won’t remain so for long. For instance, if the Nanite feature requires a very fast SSD to be feasible, which the PlayStation 5 is said to be equipped with, then that will just mean that, on PC, you’ll need an AMD X570 series board or newer and an AMD Zen2 based CPU or newer, along with a good PCIe4 NVMe SSD. Or, alternatively, a RAID matrix of 2 or more PCIe3 NVMe SSD. Along with a NAVI GPU, so that the CPU can benefit from PCIe4 bandwidth when having geometry dynamically loaded and unloaded from its VRAM.
And if the Lumen feature requires a GPU with hardware level support for AMD’s implementation of real time ray tracing, that’s surely coming to desktop with AMD’s big NAVI GPU, by the end of the year. In any case, real time (or near realtime) raytracing is not arcane magic and is certainly not exclusive to either next generation console hardware, Nvidia RTX GPUs or AMD’s big NAVI GPU many of us are waiting for. It is, for example, possible to do ray tracing using OpenGL, as is the case with the LighTracer photorealistic rendering online tool as well as standalone desktop application which I routinely use with my Vega 64 GPU to perform renders of .OBJ 3D files exported from my CAD models, even though the OpenGL does not favour AMD hardware:
With the above in mind, I’d be wary of assuming that the Nanite and Lumen features of UE5 are revolutionary rather than evolutionary improvements, or that they are exclusive to either it or the PS5.