Skip to content

Temporal upscaling and frame generation — frame analysis and plan

Native TAA and missing FSR 4 assets (2026-10-01)

upscaler=taa / BB_UPSCALER=taa is a separate native-resolution temporal AA mode, available live in the launcher and overlay. It uses the existing jitter and camera/object motion vectors, reprojection, depth rejection and 3×3 neighborhood clipping in one compute pass with two ping-pong RGBA16F histories. History stores depth in alpha; the scene's original alpha is preserved by the HDR merge. HUD is composed afterwards and never enters history. Output changes and AA/provider changes reset history. TAA uses Native AA dimensions while preserving the saved FSR preset, and does not allocate an FSR context or execute an FSR model/RCAS. It still adds GPU work compared with disabling temporal AA; native 4K raster cost also remains. The explicit BB_RENDER_RES compatibility path must be unset.

FSR Native AA is not a fast pass-through: it renders at the output resolution and also executes the temporal reconstruction (including the model for FSR 4) and optional sharpening. It can therefore be slower than no AA at the same output.

FSR 4.1.1 assets have separate 1080p/2160p tiers and standard/Ultra models: t1080_m0, t1080_m1, t2160_m0, t2160_m1. Outputs above 1080p need the 2160 tier, irrespective of the preset. A missing t2160_m0/spd.spv means missing assets, not a GPU feature failure. These assets are not bundled: build the full set with tools/fsr4cap/build_assets.sh and put it in BB_FSR411_DIR or the packaged data directory's fsr4_411. Fatal asset failures now select FSR 3.1 in the live settings, so the menu shows the active provider, retains the error, and allows retrying FSR 4 after changing the output/installing assets.

taa-shader-test checks real SPIR-V output for accumulation, clipping, depth disocclusion, invalid history and out-of-frame motion on Vulkan (Lavapipe works). out/taa-appimage-build.log records these checks, settings round trips and 64 Python tests. out/taa-live-validation.log and out/taa-appimage-validation.log check TAA at 1080p/720p/1440p/4K, TAA ↔ FSR 3, and a missing 2160-tier FSR 4.1.1 model in one gameplay process, including output captures and camera movement. out/taa-launcher-validation.log checks actual GTK controls and saved settings. TAA has not yet been tested on the tester's GTX 1060.

Startup patch is the default again for outputs other than 1080p (2026-10-02)

The live path below made the Steam Deck and a GTX 1060 + 4-core Haswell drop to 7–8 FPS (release 0.1 AppImage: 40–50 FPS on the Deck). With guest allocations at 1920×1080 the game's post-processing stays at 1080p even for 720p output, guest compute passes resolve scene proxies back to 1080p every frame (RX 7800 XT, 720p FSR 3 Performance on 4 cores: 826 vs 576 draws/frame, 185 vs 210 FPS), and on GPUs without shader stencil export every depth/stencil copy took nine draws.

  • run.sh again patches the game's render size for outputs other than 1080p (as in 0.1) and marks it with BB_AUTO_RENDER_RES=1, so an in-game restart recomputes it. Preset and output changes then need a restart (the menu offers it). BB_LIVE_RES=1 selects the live path. 1080p output and TAA keep the live path.
  • Depth/stencil scene proxies need VK_EXT_shader_stencil_export again; without it these targets stay native (as in 0.1). BB_SCENE_STENCIL_BITS=1 still forces the portable resampler for tests.

Live resolution changes (2026-10-01)

The in-game output selector now resizes 720p/1080p/1440p/2160p host targets at a display-pass boundary. Presets use output / scale, including raster sizes above 1080p (4K Quality: 2560×1440; Native AA: 3840×2160). Guest allocations stay at 1920×1080; run.sh no longer inserts a resolution patch for normal output choices. The explicit BB_RENDER_RES compatibility override is retained.

SceneTargets resolves and retires old proxies before changing size. FSR resources and history are rebuilt for the new input/output, with completion waits for both FSR 3 and FSR 4. The 1080p path keeps HDR reconstruction before game post; other outputs reconstruct tonemapped scene proxies at the first Scaleform pass, then draw HUD/menu geometry into the output-size target. The display buffer is resized too. Movie shader identification prevents a native-size fullscreen post pass from being mistaken for UI.

Guest compute passes retain their native dispatch sizes and can resolve proxies back to 1080p. This change does not scale every post-processing pass; performance and intermediate detail can differ from the old startup-patched render sizes.

Verification: out/dev/live-resolution-summary.log switches output, preset and FSR off/on in one PID. BB_PRESENT_DUMP_TRIGGER captures the completed host display buffer (including HUD) into BB_DUMP_DIR. Captures confirm actual 720p/1440p/2160p buffers. scene-resolution-test covers color/depth/stencil round trips through proxies both smaller and larger than the guest allocation. BB_PRESET_FILE accepts preset [output-index [upscaler-index]] for scripted menu-equivalent changes.

Frame analyzer: BB_CAPTURE_TRIGGER=<file> BB_CAPTURE_DIR=<dir>; creating the file records the next frame (boundary: the pass writing a display buffer) — passes, targets, shaders, sampled textures, and the first 1 KiB of bound constants for small passes and first draws.

Bloodborne's frame (1920x1080, Hunter's Dream / Nightmare)

Passes What
first pass of a frame copy of the previous UI target into the display buffer (sRGB)
shadow 4096x4096 D32 depth
G-buffer 6 targets 1920x1080 (RGBA8 x3, sRGB albedo, B10G11R11, RGBA16F) + D32S8 depth
lighting light volumes into two B10G11R11 targets
scene color RGBA16F 1920x1080, also forward/transparent draws and effects
volumetric fog compute, reads linear depth (R32F 1920x1080) and composites into scene color
half-res effects RGBA8 960x540 with half-res depth
post combine/bloom pyramid in RGBA16F / B10G11R11
tonemap into RGBA8 1920x1080 (the UI target)
game AA ping-pong RGBA8 1920x1080 after tonemap (to be skipped when upscaling)
UI stencil-masked draws over the same RGBA8 target

There is no velocity buffer, also with the camera moving: motion vectors are computed.

Scene constants (864 bytes, bound by most passes)

Signature: [0]=3000 (far) [1]=1/3000 [4]=1920 [5]=1080 [6]=1/1920 [7]=1/1080.

Floats Meaning
8–19 view matrix, 3x4 rows (rotation + translation); the only block that changes with the camera
36–51 inverse projection (0.700285 = 1/1.42799, 0.39391 = 1/2.53865)
52, 57, 62, 63 projection (rows 52–67), D3D depth 0..1: x 1.42799, y 2.53865, z 1.00002 / -0.0500679 (near 0.05, far 3000)
112–175 shadow cascade matrices
180–191 inverse view (camera to world), 3x4 (176–179: 2, 8, 15, 0)

The previous frame's matrices are not there; the port keeps them itself.

Plan

  1. Find the scene constants every frame (signature), keep the previous view/projection.
  2. Camera motion vectors: compute pass from depth and current/previous matrices into an RG16F target; debug view to check them. Object motion later (vertex shader replay with the previous frame's constants through the recompiler).
  3. Jitter: sub-pixel offset of clip-space positions in the scene passes (recompiler).
  4. Upscaler at the scene color stage (before post and UI): FSR 3.1 first (open, native Vulkan, also frame generation), then DLSS (native on Linux), XeSS/XeFG and OptiScaler through a loader for their Windows DLLs.

Status

  • Camera motion vectors: verified by reprojecting the previous frame (BB_DEBUG_MOTION=1, toggles 1<<22 reprojected frame, 1<<23 error map): static geometry matches.
  • FSR 3.1 (BB_UPSCALER=fsr3, FireBurn/FSR-Vulkan submodule): scene color before the post-processing combine (compute shader 9a9cf8a9), 1:1, RGB written back (the game keeps data in the alpha). Toggle 1<<24 switches it off at run time.
  • Jitter: viewport offset of scene geometry (drawn with the scene depth, not full-screen quads), Halton(2,3) 8 phases; sign checked by sharpness (correct 255, flipped 208, off 271 — 1:1 jitter trades high-frequency aliasing for a slightly softer image). Toggle 1<<25 off.
  • Frame-to-frame difference while standing still: ~33% lower with FSR.
  • Missing: motion of animated objects (characters, cloth, foliage), reactive/transparency masks for particles and fog, render-resolution scaling, frame generation.

2026-10-01: FSR 4 did not work at 1440p/2160p output (flicker and shimmer)

Video video_2026-10-01_03-58-59.mp4 (3840x2160 output, FSR 4 Performance): flicker and shimmer on all objects. Cause: the upscaler did not run at all in this mode, while jitter stayed enabled. The screen showed a stretched scene frame, each frame shifted by its own Halton phase.

  1. The UI pass was identified by the exact BB_RENDER_RES target size (1916x1078), but the game allocates targets with an aligned height (1916x1080; the scene constants also say 1916x1080). RunScaled was never called (the log had no UI: native composition). Alignment up to 8 pixels is now accepted (RenderTarget), and the scene size for FSR comes from the scene constants (CameraMotion::RenderSize, SceneSize).
  2. After that FSR 4 failed with external image registration failed (-1000069000): RunScaled created new image views every frame, and the FSR 4 registry holds eight. It now uses the same CachedView as the Native AA path.

Dump check (BB_DUMP_TRIGGER=<file> BB_DUMP_DIR=<dir>, BB_DUMP_FRAMES, default 8: FSR input, motion vectors and output, raw): with a static camera the PSNR of adjacent output frames is ~45 dB against ~28 dB for the jittered input; no trails when turning the camera or walking. FPS in this mode is ~107 instead of ~220; before, FSR 4 simply did not run.

Remaining: sprites (4 indices each) drawn into scene color with scene depth (glows, small lights) are still not shifted by jitter (the rule "≤ 6 indices = full-screen pass").

2026-10-01: Faster FSR 4 — the post pass (2.9 → 0.8 ms at 4K)

Measurements: BB_FSR4_PROFILE=1 (time of each FSR 4 pass, a provider patch in the submodule, see gpu/patches/fsr-vulkan), the out-of-game benchmark out/gpu/fsr4-bench (ninja -C out/gpu fsr4-bench; --stats prints registers and instructions from RADV). 4K Balanced (2260x1272 → 3840x2160), RX 7800 XT: all of FSR 4 takes 5.8 ms, of which post is 2.5–2.9 ms (not the neural network: the last layers, 2x2 pixel shuffle and blending with history), the 12 model passes 2.4 ms, pre 0.55.

Cause: each thread computes a 2x2 block of output pixels and writes them one by one into three images (recurrent state, history, output), so each wave store instruction writes every other pixel. Removing any one of the three stores made the pass 1.3–4 times faster with the same code.

Fix: tools/fsr4_optimize.sh decompiles post (spirv-cross), tools/fsr4_post_lds.pl gathers the workgroup's 16x16 block in shared memory and writes contiguous rows, and glslang compiles it back into fsr4_shaders/opt/; vk_fsr4.cpp loads it from there (BB_FSR4_OPT=0 selects the original). Two subtleties, found by comparing outputs: - spirv-cross translates the signed int8 unpack (OpBitcast to i8vec4) as unpack8(uint) (unsigned); without a fix the result is completely different (PSNR 17 dB); - values in shared memory must stay float: the compiler turns a half from shared memory into a 16-bit store, which rounds differently to unorm8.

tools/fsr4_verify.sh compares the original and the optimized post for all presets at 1080p/1440p/4K output on pseudo-random inputs: 1440p and 4K are bit-identical; post at 4K 2.1–3.5 → 0.81–0.87 ms, 1440p 1.2–1.5 → 0.36–0.38 ms, 1080p 0.37–0.69 → 0.20–0.22 ms. In game (4K Balanced): FSR 4 5.8 → 4.0 ms, GPU frame 13.1 → ~11.6 ms; beyond that FPS is CPU-bound (the GPU command thread waits for guest memory copies, "host copies" ~20%).

Found along the way: the original FSR 4 at the 1080 tier (1920x1080 output) is nondeterministic: from run to run on identical inputs a strip at the left edge changes (columns 0–132), so there is a race or a read of uninitialized memory somewhere in the model/provider. At 1080p output this can cause flicker at the left edge of the frame. Not investigated.

The reactive mask is no longer computed with FSR 4 (FSR 4 does not accept it).

What else was checked (same setup, 4K Balanced)

  • A/B in one process (ab.sh, bit 24 turns FSR off, UI copy without upscaling): 86.5 FPS with FSR 4 against 87.7 without it. After the post optimization the frame is CPU-bound (GPU command thread, waiting for guest memory copies), not GPU-bound. Async compute for FSR (overlapping it with the start of the next frame) would gain no more than this ~1%; postponed until the CPU side is faster. It also requires reordering the UI and present commands of frame N after the work of frame N+1.
  • WMMA (VK_KHR_cooperative_matrix, supported by RADV on RDNA3): a prototype of model pass 1 (16-channel residual block: 3x3 16→16, 1x1 16→32 ReLU, 1x1 32→16): 0.85 ms in the first version, 0.49 ms with a word layout in shared memory, against 0.30 ms for the original dot4. The arithmetic on WMMA would take ~0.1 ms, but with 16 channels the tile layout, epilogues through shared memory and occupancy (6 waves/SIMD, limited by LDS) eat the gain. The ceiling is about 1 ms for all 12 passes with careful manual rework of each; not started.
  • RADV_PERFTEST=cswave32 (wave32 for compute): FSR 4 is slower, 4.2 → 5.4 ms.
  • The pre pass (0.54 ms) is not limited by stores (0.50 ms without stores).

2026-10-01: FSR 4 flicker at the left edge at 1080p output — a race in model pass 11

The 1080-tier nondeterminism (see above) is a bug in the v07 model shader, pass 11 (decoder, 1/4 → 1/2 resolution). Each thread is one pixel of the 1/4-resolution input and writes a 2x2 block of the 1/2-resolution output. The dispatch rounds the width up to 64 threads, and threads beyond the input width do not exit: at the 1080 tier (input 480 wide) threads 480..511 write output pixels 960..1023, which are the first pixels of the next row (a row of any tensor is 15360 bytes), and race with their real writers. At 4K output the width 960 is divisible by 64, so there is no bug.

tools/fsr4_pass11_guard.pl adds an early exit for out-of-bounds threads (the size comes from the pass's own neighbor check); tools/fsr4_optimize.sh builds it into fsr4_shaders/opt for all presets together with post; vk_fsr4.cpp and the benchmark load any pass from opt/. tools/fsr4_verify.sh: 1440p/4K are bit-identical to the original (the guard does not trigger there); 1080p gives the same result from run to run, and after the first frame it differs from the original only in the former strip at the left edge.

Found in the process: spirv-cross translates all signed int8 unpacks (OpBitcast to v4char) as unsigned unpack8(uint); pass 11 has nine different forms of them. The fix is shared by the scripts: tools/Fsr4SpirvCrossFixes.pm (all unpack8 go through a signed helper).

2026-10-01: FSR 4.1.1 — what is inside and how to run it in Vulkan (investigation)

Source: amd_fidelityfx_upscaler_dx12.dll 4.1.1.2740 from OptiScaler (FSR4_LATEST). - The DLL contains the provider ffxProvider_FSR4_Int8 / Fsr4Int8UpscalerModel (version "FSR4-i8 4.1.1"). 1028 DXIL containers (25.7 MB): model passes fsr4_model_v07_fp8_no_scale_pass1..12 (+ _post), 270 prepass variants, 144 postpass, fsr_rcas_pass and the FSR 3.1/2 fallback. There is no compressed data. - Despite "fp8" in the name, the model passes are plain DXIL with dot4AddPacked (int8 dot4), without AMD extensions (WMMA). The architecture is the same v07 (12 passes), but weights and biases are not baked into the shader; they are read from InitializerBuffer (t18, typed buffer), tensor sizes from the cbuffer CsTensorSizes, scratch is u11. Each pass has 9 variants: the grid is baked in (1080 / 4K / 8K tiers: 480x270, 960x540, 1920x1080 for pass 5) and, apparently, the presets. - amdxcffx64.dll (loader 2.3.0.2913, loaded by Proton) has the same FP8 FSR 4 passes and ML frame generation (mlfi_*), with no separate INT8 model. FSR4_INT8 4.0.2 is v07_i8, that is, what the port already uses. - dxil-spirv (HansKristian-Work/dxil-spirv, built from source) translates the passes into correct SPIR-V: dxil-spirv <pass> --enable-shader-i8-dot --ssbo-uav --use-reflection-names.

What is missing for an own 4.1.1 provider: the container mapping (preset, tier, pass), the contents of InitializerBuffer, the CsTensorSizes values, the prepass/postpass permutations, constants and dispatch sizes. The most reliable way is not to reverse the provider's x86 code but to record one frame: a small FFX API (DX12, MinGW) test program under Wine/Proton with vkd3d-proton, explicit selection of the "FSR4-i8" provider through a version override, and a Vulkan layer that writes the pipelines (SPIR-V from dxil-spirv), buffer contents and dispatch sizes. Then replay with an own provider through the same benchmark (fsr4-bench), checked against the recorded output.

2026-10-01: FSR 4.1.1 in the port (upscaler=fsr411)

The AMD 4.1.1 (INT8) model runs natively in Vulkan and matches the DLL bit for bit.

How it was obtained. tools/fsr4cap/fsr4cap.exe (C, MinGW) calls the FidelityFX API 2.3 from amd_fidelityfx_loader_dx12.dll with an explicit version 4.1.1 and, by replacing D3D12 method tables, records everything the DLL does under vkd3d-proton: pipelines (DXIL), root signatures, descriptors, uploads, constants, dispatches. (Since then fsr4cap.exe loads AMD's upscaler DLL directly, which exports the API itself; the loader is only needed for a DLL that does not.) A frame is 29 dispatches: auto-exposure SPD, prepass, pass0_post, model passes 1..12 (each followed by _post, which clears the tensor border), postpass, RCAS. - There are two models: m0 (Native..Performance) and m1 (Ultra Performance, ratio 3.0); weights are 128 KB (InitializerBuffer). Tiers: t1080 (output up to 1920x1080), t2160 (larger, incl. 2560x1080). - Tensors are computed from the output aligned to 8; the CsTensorSizes table (17 sizes), the group counts and the constants (MLSR as in v07, but width/height are not aligned and inv_scale = 1 / scale; SPD; RCAS) are checked by extract.py on 20 recordings. - dxil-spirv translates DXIL the same way as vkd3d-proton: --mixed-float-dot-product (VK_VALVE_shader_mixed_float_dot_product, half dot2 with float accumulation); without it PSNR is 40 dB instead of an exact match. --class-bindings (a patch) lays out SRV/UAV/ CBV/sampler registers at bindings +0/+32/+64/+96; the runtime binds resources by name. - fsr411/fsr411.cpp is an independent Vulkan core (the game and fsr4-bench --fsr411). - verify.sh: DLL and replay on identical inputs, 8 frames, bit-identical for 1080p (both models), 1440p, 2560x1080, 4K (both models). 1600x900 differs only at the right edge, where the DLL itself is nondeterministic (a race in its shaders when the tensor width is not a multiple of 64).

Speedup. postpass is rewritten to use workgroup memory at the SPIR-V assembly level (postpass_lds.py; going through GLSL is inexact: DenormPreserve and the exact placement of RelaxedPrecision are lost, and something else changes the result too): 2.18 → 0.95 ms at 4K, bit-identical to the DLL. All of FSR 4.1.1: 1.09 ms at 1080p (1280x720), 4.2 ms at 4K Balanced. In game (4K Balanced) the upscaler takes 4.12 ms against 4.32 ms for the accelerated v07.

Building the assets (by the user, from their own DLLs; the repository contains nothing from AMD):

bash tools/fsr4cap/build_assets.sh <amd_fidelityfx_upscaler_dx12.dll 4.1.x> [<amd_fidelityfx_loader_dx12.dll 2.3.x>]

Needs nix-shell (or MinGW, cmake, ninja, python3, SPIRV-Tools, git), network access for dxil-spirv, and Proton (PROTONPATH, or GE-Proton 10+ / Proton-CachyOS / Proton Experimental; umu-run or Steam's runtime). In the AppImage the tools are prebuilt (BB_FSR4CAP_TOOLS). The DLLs are available, for example, from OptiScaler (FSR4_LATEST/) and in games with FSR 3.1/4. A build from scratch produces the same files; VERIFY=1 at the end compares against the DLL.

Correction to the A/B above ("the frame is CPU-bound"). Those measurements ran while 12 hung shadPS4 processes each occupied a core. On an idle system: "host copies" 1.6%, waiting for the GPU ~41%; the frame is GPU-bound, so async compute for FSR makes sense again.