# Neural upscaling and restoration

> Real-ESRGAN, Real-CUGAN and NAFNet, and NVIDIA's DLSS and RTX VSR models through the MLX-DLSS port, running on Apple Silicon Macs with MLX and Metal.

Web page: https://procrastinine.github.io/maclauncher/graphics/neural-restoration/

Neural upscaling and restoration use trained models, running on a Mac's GPU, to enlarge or repair a game's image. Ordinary upscalers fill in new pixels by fixed rules. A neural model, by contrast, is trained on many images and learns what images tend to look like, such as how a line in an anime drawing continues, what grass looks like up close, or which fine patterns are noise rather than detail. Given a small or damaged image, the model redraws it from what it learned.

MacLauncher runs these models on your Mac's GPU through [MLX](https://github.com/ml-explore/mlx) and [Metal](https://developer.apple.com/metal/), inside the same graphics chain as every other effect. They need an Apple Silicon Mac with macOS 14 or later. Intel games running translated under [Rosetta](https://support.apple.com/en-us/102527) or [CrossOver](https://www.codeweavers.com/crossover/) can use them as well, because a small Apple Silicon helper runs the model on their behalf.

## Upscaling drawn art

Neural upscalers do especially well on visual novels and anime-style games. This scene from [CLANNAD](https://store.steampowered.com/app/324160/CLANNAD/) was reduced to 640×480 and then doubled:

*Image comparison, Bilinear 2× and Real-CUGAN 2×: Nagisa under the cherry blossoms in CLANNAD, upscaled 2× from 640×480.* Real-CUGAN redraws line work and flat color cleanly where bilinear filtering only blurs.

The lighter shader upscalers from the [shader collections](https://procrastinine.github.io/maclauncher/graphics/shaders.md) aim at the same kind of image and cost far less:

**Real-CUGAN**

*Image: The CLANNAD scene upscaled with Real-CUGAN 2× Conservative.* Real-CUGAN 2× (Conservative), a neural model.

**Anime4K**

*Image: The CLANNAD scene upscaled with Anime4K Upscale CNN x2 M.* Anime4K Upscale CNN x2 M, a GPU shader.

**ArtCNN**

*Image: The CLANNAD scene upscaled with ArtCNN C4F32.* ArtCNN C4F32, a GPU shader.

**Bilinear**

*Image: The CLANNAD scene upscaled with bilinear filtering.* Plain bilinear filtering, for reference.

At 4× the differences grow. This is the same scene, starting from only 320×240:

**Real-ESRGAN Anime**

*Image: The CLANNAD scene upscaled 4× with Real-ESRGAN Anime.* Real-ESRGAN Anime 4×.

**Real-CUGAN 4×**

*Image: The CLANNAD scene upscaled 4× with Real-CUGAN.* Real-CUGAN 4× (Conservative).

**Bilinear**

*Image: The CLANNAD scene upscaled 4× with bilinear filtering.* Bilinear 4×.

## Upscaling everything else

[Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)'s **General** model handles 3D games and other rendered or photographic images rather than line art:

*Image comparison, Bilinear 4× and Real-ESRGAN General 4×: Geralt from The Witcher 3, upscaled 4× from 480×270.* Geralt in The Witcher 3: Wild Hunt, reduced to 480×270 and enlarged 4×. Real-ESRGAN rebuilds hair, beard and chain mail where bilinear filtering only blurs them.

On pixel art, the same model behaves very differently. It treats the pixels as detail to smooth over and redraws the scene with soft, outlined shapes, which is why pixel-art games usually stay with Nearest or a [pixel-art scaler](https://procrastinine.github.io/maclauncher/graphics/upscaling.md):

*Image comparison, Nearest 4× and Real-ESRGAN General 4×: Madeline among wind-blown flags in Celeste, enlarged 4× from 320×180.* Celeste at its native 320×180, enlarged 4×.

## Cleaning up an image

Not every model changes the size. [NAFNet](https://github.com/megvii-research/NAFNet) restores an image at its original resolution, with one model for noise and another for blur:

*Image comparison, Noisy and NAFNet SIDD: A close-up of Geralt from The Witcher 3 with heavy added noise.* NAFNet's SIDD model removing heavy added noise.

*Image comparison, Motion blur and NAFNet GoPro: A close-up of Ciri from The Witcher 3 with added motion blur.* NAFNet's GoPro model reversing added motion blur.

## The models

| Model | What it does | Output size |
| --- | --- | --- |
| Real-ESRGAN | Anime and General upscalers. General has a paired denoising model, and when both are installed, the **Denoising** control blends between them. | 4× |
| [Real-CUGAN](https://github.com/bilibili/ailab/tree/main/Real-CUGAN) | Upscaler trained on anime, in Conservative, No denoising and Denoise variants. | 2×, 3× or 4× |
| RTX Video Super Resolution | NVIDIA's video upscaler, through [MLX-DLSS](https://github.com/iamwavecut/MLX-DLSS). | 2× |
| DLSS neural rendering | The DLSS 5 neural rendering model through MLX-DLSS. It redraws the image toward a photoreal look, blended in with **Intensity**, and can keep image history with **Temporal reconstruction**. | Same size |
| DLSS SR (image) | DLSS super resolution working from a sequence of finished images. | 2× |
| NAFNet | SIDD (denoising) and GoPro (deblurring) restoration models. | Same size |

MacLauncher's other neural models are [RIFE](https://github.com/hzwer/Practical-RIFE) and MLX interpolation for [frame generation](https://procrastinine.github.io/maclauncher/graphics/frame-generation.md), and [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), which estimates depth from a flat image for the [estimated depth fog](https://procrastinine.github.io/maclauncher/graphics/color-and-light.md#depth-without-a-depth-buffer) effect.

Real-ESRGAN and NAFNet build on the existing [Real-ESRGAN MLX](https://github.com/xocialize/mlx-realesrgan-swift) and [NAFNet MLX](https://github.com/xocialize/mlx-nafnet-swift) ports; Real-CUGAN runs its original models on MLX.

## Getting models

MacLauncher doesn't include model files. **Runtimes → Graphics → Neural models** lists the supported downloads: **Install** downloads one, and **Import model…** adds a model you already have. Installed models work offline and are shared by every game.

Each neural row in the chain has a **Model** menu. **Automatic** chooses an installed model for that row's operation, starting with the default you set for the operation in **Neural models**. You can also pick a specific model for one row. If a row's model is missing, the row shows **Model needed** and the rest of the chain keeps running.

## Keeping it fast

A neural model's cost grows with the number of pixels it processes, so the usual approach is to give it a smaller image. Put a **Downsample** row before the model, or lower the **Game resolution**, and fill the window afterward with a fast upscaler such as [MetalFX](https://developer.apple.com/documentation/metalfx). [Building a chain](https://procrastinine.github.io/maclauncher/graphics/composing-effects.md) shows a complete example. If the model still can't keep up with the game, frame generation can fill in the missing frames.

The graphics guide's [neural models section](https://procrastinine.github.io/maclauncher/reference/graphics/README.md#neural-models) lists every download and setting, and [Neural models and packages](https://procrastinine.github.io/maclauncher/reference/graphics/NEURAL_RESTORATION.md) describes the model architectures and import formats.

## Further reading

- [Super-resolution imaging](https://en.wikipedia.org/wiki/Super-resolution_imaging) on Wikipedia, for the broader field these models come from.
- The [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN), [Real-CUGAN](https://github.com/bilibili/ailab/tree/main/Real-CUGAN) and [NAFNet](https://github.com/megvii-research/NAFNet) project pages, each with its own comparisons and papers.