DLSS 4’s New Transformer Model Could Enable Frame Generation on RTX 30-Series GPUs

gamingbolt.com:

Nvidia has offered up more details on the new features coming to its GPUs thanks to the release of DLSS 4, as well as how some of its technologies, such as its new transformer models and enhanced frame generation work. In an interview with Digital Foundry, vice president of applied deep learning research at Nvidia Bryan Catanzaro spoke about the technology.

Speaking about frame generation, Catanzaro explained how Nvidia’s Optical Flow accelerator was used. The technology, originally used for Nvidia’s automotive division’s research into self-driving cars, is an evolution of the company’s video encoding algorithms. However, Optical Flow was difficult to improve, so Nvidia has decided to switch things up with a fully AI-based solution for DLSS 4’s take on frame generation.

“When we built Nvidia DLSS 3 Frame Generation, we absolutely needed hardware acceleration to compute Optical Flow,” explained Catanzaro. “We didn’t have enough Tensor Cores and we didn’t have an Optical Flow algorithm that was good enough. We hadn’t developed a real-time Optical Flow algorithm that ran on Tensor Cores that could fit our compute budget. We had the Optical Flow accelerator, which Nvidia had been building for years as an evolution of our video encoder technology, and it’s also been a part of our automotive computer vision acceleration for self-driving cars.”

“It made sense for us to use that for Nvidia DLSS 3 Frame Generation. But the difficult part about any sort of hardware implementation of an algorithm like Optical Flow is that it’s really difficult to improve it. It is kind of what it is and the failures that arose from that hardware Optical Flow, we couldn’t undo them with a smarter neural network until we decided to just replace it and go with a fully AI-based solution, so that’s what we’ve done for Frame Generation in DLSS 4.”

With the new transformer model, the AI used for frame generation can learn things much quicker, while simultaneously also being less heavy on the GPU’s