Add engine-debug-tools and fix critter animation rendering bugs
build / build (push) Successful in 2m52s
build / build (push) Successful in 2m52s
- engine-debug-tools/: a debug dump hotkey + dump2png to inspect decoded critter frames independent of the renderer (see root README.md's "Debugging" section). - engine-patches/04,05: fix critter sprite scale and an OpenGL texture cache collision, found using that tooling. - mdlp: fix encodeFrames RLE-diffing upscaled frames against a previous-frame reference the real RSD8 decoder never implements (verified against Night Dive's original Mac source down to the 68K assembly) - was rendering as scattered static in-game. - Upscale pipeline: better color quantization (dithering, pure-ramp exclusion) and per-resource model selection.
This commit is contained in:
@@ -192,6 +192,8 @@ mods/
|
||||
messages/
|
||||
<label>.json # one message's overrides
|
||||
<label>.txt # optional: a text field's body, referenced by ...File
|
||||
bitmaps/
|
||||
<label>.json # an AI-upscaled set of critter/object art resources
|
||||
```
|
||||
|
||||
Each `messages/*.json` overrides one message (email, log, or fragment) by
|
||||
@@ -213,6 +215,26 @@ label:
|
||||
`mods/mymod/` is a worked example matching section 3's Rebecca's-mail edit,
|
||||
expressed as MDL instead of a `hacked` session.
|
||||
|
||||
Each `bitmaps/*.json` names a shared upscale configuration and the
|
||||
resources to apply it to - `gen_animgallery.sh`'s own checkbox-driven MDL
|
||||
generator (its rendered gallery page has a selection UI that assembles this
|
||||
exact JSON shape from whatever's checked) produces this format directly:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `model` | Which baked-in ONNX model to upscale with - see the table below |
|
||||
| `scale` | Output size as a multiple of the original (for example `2`) |
|
||||
| `resources` | Each entry is `"0x<ID>"` (language `Any`) or `"0x<ID>_<Language>"` (for example `"0x25_German"`), matching `gen_animgallery.sh`'s own id format |
|
||||
|
||||
Available `model` values (baked into `mdlp-image`, see
|
||||
`mdlp-image/upscale_batch.py`):
|
||||
|
||||
| `model` | Checkpoint | Best for |
|
||||
|---|---|---|
|
||||
| `realesrgan-x4plus-anime` | Real-ESRGAN's 6-block anime/illustration RRDBNet (16.7M params) | Flat-color sprite art - keeps hard edges instead of smoothing them the way a photo/video model would. Default choice. |
|
||||
| `realesrgan-x4plus` | Real-ESRGAN's general-purpose photo RRDBNet (16.7M params) | A specific resource where the anime model genuinely mis-renders a region (seen on some high-contrast color boundaries) - less edge-sharp overall, much slower, and can mis-render its own different regions, so check the result before keeping it |
|
||||
| `realesr-general-x4v3` | Real-ESRGAN's compact general-purpose SRVGGNet (1.2M params) | A specific resource neither RRDBNet model renders cleanly - noticeably softer than either, but the most robust against both of their failure modes. Not a blanket replacement, just a last resort for one stubborn sprite |
|
||||
|
||||
### 4.2. `mdlp`
|
||||
|
||||
`src/mdlp/` is first-party Go source (not vendored - edit it directly, no
|
||||
@@ -281,3 +303,83 @@ Same as `hacked-build-image/` (section 2.1): `./mdlp-upload-image.sh` reads
|
||||
`mdlp-image/VERSION` just because the Dockerfile changed - only once a build
|
||||
with the current version has actually succeeded, same policy as the root
|
||||
project's own `build-image/VERSION`.
|
||||
|
||||
## 5. Testing
|
||||
|
||||
Verifying a mod's built output (particularly an AI-upscaled one, section
|
||||
4.1) means comparing its rendered result against the unmodified base game,
|
||||
not just eyeballing the result in isolation - a subtly wrong color or a
|
||||
corrupted frame can look plausible on its own. `gen_animgallery.sh` renders
|
||||
straight from a built mod's own resource data (`dist/mods/<name>/mod`), the
|
||||
same bytes the engine loads, so a problem visible there is a problem in the
|
||||
mod, not an artifact of some other preview path. Regenerate it with `-data
|
||||
res/assets/ss_ee/data -data dist/mods/<name>/mod` layered over the base
|
||||
data to see the mod's effect, or with just the base `-data` argument to
|
||||
check what the unmodified resource looks like for comparison.
|
||||
|
||||
`gen_animgallery.sh` is a from-scratch Go decoder, not the game engine -
|
||||
it never exercises the engine's own texture upload/caching code at all.
|
||||
A clean gallery render rules out a bug in the mod's resource *data*, not a
|
||||
bug in how the engine *renders* that data - the two are independent, and a
|
||||
problem can exist in only one of them. Actually running the built mod
|
||||
(`../dist/run-mods.sh` or `run.sh` with a single mod folder, section 4.3)
|
||||
is the only way to check the second. This in-engine verification has so
|
||||
far only been through the desktop build, not the Quest headset - both
|
||||
read the same resource data and the engine-side bugs found here
|
||||
(`engine-patches/`, root [README.md](../README.md) section 4's patch
|
||||
reference) were resource-data-adjacent, not OpenXR-specific, so a fix at
|
||||
this level should carry over, but it isn't headset-confirmed until tested
|
||||
there.
|
||||
|
||||
The AI-upscale pipeline (`src/mdlp/upscale.go`) and the engine's own
|
||||
rendering of its output have a failure mode worth specifically checking
|
||||
for, found by comparing upscaled output against the base game across many
|
||||
resources rather than trusting a handful of spot-checks:
|
||||
|
||||
- **Wrong-hue speckling in dark or muted regions.** SS1's palette has a
|
||||
handful of fully-saturated "pure color ramp" entries (pure cyan, red,
|
||||
green, ...) that sit deceptively close, in color-distance terms, to
|
||||
common dark/muted shading colors - a plain nearest-color search, with or
|
||||
without dithering, can pick one of these for a near-black pixel the AI
|
||||
upscaler produced, since color distance near black barely penalizes an
|
||||
unrelated hue. `isPureRampColor` excludes these entries from
|
||||
re-quantization entirely rather than relying on a brightness or
|
||||
saturation threshold, which proved too easy for a genuinely
|
||||
dark-but-muted pixel to slip past.
|
||||
|
||||
`encodeFrames` compresses every frame against an all-zero reference
|
||||
instead of diffing against the previous frame's real pixels, because
|
||||
that's the only convention the game's actual RSD8 decoder implements - so
|
||||
an animation's frames never depend on each other, and a size change
|
||||
between poses isn't a concern to test for. This is confirmed directly
|
||||
against Night Dive's officially published original Mac source
|
||||
(`ShockMac/Libraries/2D/Source/RSD/{RSDCVT.C,RSDUnpack.c}`), not just the
|
||||
vendored Shockolate snapshot in `engine/`: both files are logic-identical
|
||||
to their vendored counterparts (same 1994 RCS revision comments, same
|
||||
unconditional `LG_memset(p_dst, kSkipColor, ...)` zero-fill in every skip
|
||||
branch), and `RSDUnpack.c` additionally carries the original hand-written
|
||||
68K assembly decoder alongside its later PowerPC/C port - its own skip
|
||||
handler also zeroes an output register rather than reading from a
|
||||
reference. Two independently-authored implementations, spanning different
|
||||
CPU architectures and years apart, agreeing on "skip means zero" down to
|
||||
matching source comments ("skip (zero) bytes") rules this out as an
|
||||
overlooked bug rather than deliberate, original design.
|
||||
|
||||
The RLE format's own opcode grammar does permit the alternative reading -
|
||||
"skip" meaning "retain the reference image's value" rather than "write
|
||||
zero" (`ss-specs` documents this as relevant to compressing "video
|
||||
clips", which is what led `hacked`'s own GIF-import path, and therefore
|
||||
earlier versions of `encodeFrames`, to use it). Searching the original
|
||||
source for anywhere that convention is actually exercised turned up
|
||||
nothing: SS1's two "video clip" features - cutscenes (`MoviePlay.c`) and
|
||||
"video mail" (`VMAIL.C`, itself built directly on `MoviePlay.h`) - both
|
||||
use an entirely separate movie codec, not RSD8, and neither references
|
||||
the RSD8 decoder at all.
|
||||
|
||||
`gen_animgallery.sh` and the real engine therefore always agree on an
|
||||
upscaled animation's frames; a difference between them points at a
|
||||
genuine engine rendering bug (root [README.md](../README.md) section 4's
|
||||
patch reference) rather than this pipeline's own output, and
|
||||
`engine-debug-tools/`'s debug dump hotkey (root README.md section 4.5
|
||||
"Debugging") dumps the engine's own decoded frame data independent of its
|
||||
renderer to tell the two apart.
|
||||
|
||||
@@ -21,23 +21,43 @@ RUN apt-get update \
|
||||
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||
ca-certificates curl gosu patch python3 python3-pip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN pip3 install --no-cache-dir onnxruntime pillow numpy
|
||||
RUN pip3 install --no-cache-dir onnxruntime pillow numpy scipy
|
||||
|
||||
# The AI upscaler model for mdlp's upscale mod type (see ../README.md
|
||||
# section 4): RealESRGAN_x4plus_anime_6B, a 6-block RRDBNet ONNX-exported
|
||||
# from xinntao/Real-ESRGAN's anime-illustration checkpoint (BSD-3-Clause).
|
||||
# It's trained on flat-color line art, so it keeps hard sprite edges rather
|
||||
# than smoothing them the way a photo- or video-denoising model would. A
|
||||
# mod's own "model" field (upscaleDef in ../src/mdlp/upscale.go) is not
|
||||
# consulted; every resource goes through this one model. upscale_batch.py
|
||||
# resizes its fixed native 4x output down to whatever scale a mod actually
|
||||
# asks for.
|
||||
ARG UPSCALE_MODEL_URL=https://huggingface.co/skillsafe-ai/realesrgan-x4plus-anime-6b/resolve/main/model.onnx
|
||||
RUN mkdir -p /opt/upscale-model \
|
||||
&& curl -sSL -o /opt/upscale-model/model.onnx "$UPSCALE_MODEL_URL"
|
||||
# The AI upscaler models for mdlp's upscale mod type (see ../README.md
|
||||
# section 4), selected per mod by a bitmaps/*.json file's own "model" field
|
||||
# (upscaleDef in ../src/mdlp/upscale.go) - upscale_batch.py resolves a name
|
||||
# straight to $UPSCALE_MODELS_DIR/<name>.onnx, and picks tiled or single-pass
|
||||
# inference based on each model's own declared ONNX input shape (fixed vs
|
||||
# dynamic height/width), so adding another model here needs no code change.
|
||||
# Each model's RGB output is re-quantized to SS1's fixed 256-color palette
|
||||
# with Floyd-Steinberg dithering (imageToBitmap in ../src/mdlp/upscale.go),
|
||||
# not a plain nearest-color map - without it, any model's smooth gradients
|
||||
# can land on a visibly wrong, much darker palette entry in a solid patch
|
||||
# rather than unnoticeable dithering noise.
|
||||
# All three are BSD-3-Clause ONNX exports of xinntao/Real-ESRGAN checkpoints:
|
||||
# - realesrgan-x4plus-anime: a 6-block RRDBNet trained on flat-color
|
||||
# illustration, keeping hard sprite edges rather than smoothing them the
|
||||
# way a photo- or video-denoising model would. Dynamic input shape. Default
|
||||
# choice - sharpest result on most sprites.
|
||||
# - realesrgan-x4plus: the general-purpose photo checkpoint (16.7M-param
|
||||
# RRDBNet). Fixed 128x128 input, needs tiling for anything larger, and
|
||||
# much slower than the other two. Worth trying when the anime model
|
||||
# mis-renders a specific sprite, but can itself mis-render different ones.
|
||||
# - realesr-general-x4v3: a compact 1.2M-param SRVGGNet, dynamic input
|
||||
# shape, ~100x faster than realesrgan-x4plus. Noticeably softer than
|
||||
# either RRDBNet model, but the most robust against both models' failure
|
||||
# modes - the model to reach for on a specific sprite neither RRDBNet
|
||||
# model renders cleanly, not as a blanket replacement.
|
||||
ARG UPSCALE_MODEL_ANIME_URL=https://huggingface.co/skillsafe-ai/realesrgan-x4plus-anime-6b/resolve/main/model.onnx
|
||||
ARG UPSCALE_MODEL_GENERAL_URL=https://huggingface.co/qualcomm/Real-ESRGAN-x4plus/resolve/6f5749d5adff39de0127e42bc45c8ec8e6f76a66/Real-ESRGAN-x4plus.onnx
|
||||
ARG UPSCALE_MODEL_COMPACT_URL=https://huggingface.co/jonathanst29/tinier-upscale-models/resolve/main/realesr-general-x4v3.onnx
|
||||
RUN mkdir -p /opt/upscale-models \
|
||||
&& curl -sSL -o /opt/upscale-models/realesrgan-x4plus-anime.onnx "$UPSCALE_MODEL_ANIME_URL" \
|
||||
&& curl -sSL -o /opt/upscale-models/realesrgan-x4plus.onnx "$UPSCALE_MODEL_GENERAL_URL" \
|
||||
&& curl -sSL -o /opt/upscale-models/realesr-general-x4v3.onnx "$UPSCALE_MODEL_COMPACT_URL"
|
||||
COPY mdlp-image/upscale_batch.py /usr/local/bin/upscale_batch.py
|
||||
RUN chmod 755 /usr/local/bin/upscale_batch.py
|
||||
ENV UPSCALE_MODEL_PATH=/opt/upscale-model/model.onnx
|
||||
ENV UPSCALE_MODELS_DIR=/opt/upscale-models
|
||||
|
||||
RUN curl -sSLO "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" \
|
||||
&& tar -C /usr/local -xzf "go${GO_VERSION}.linux-amd64.tar.gz" \
|
||||
|
||||
@@ -8,7 +8,13 @@ Runs on CPU only, not Vulkan: this Docker image has no real GPU, and
|
||||
Vulkan software emulation (Mesa lavapipe) gives wrong colors for ML
|
||||
inference workloads.
|
||||
|
||||
Usage: upscale_batch.py <model.onnx> <in-dir> <out-dir> <scale>
|
||||
Resolves <model-name> to $UPSCALE_MODELS_DIR/<model-name>.onnx and picks
|
||||
single-pass or tiled inference from that model's own declared ONNX input
|
||||
shape - a model with a fixed height/width (e.g. 128x128) needs tiling for
|
||||
any image bigger than that; one with symbolic dimensions runs the whole
|
||||
image through in one pass, whatever its size.
|
||||
|
||||
Usage: upscale_batch.py <model-name> <in-dir> <out-dir> <scale>
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
@@ -16,31 +22,118 @@ import os
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from PIL import Image
|
||||
from scipy import ndimage
|
||||
|
||||
|
||||
def extend_opaque_color(rgb, alpha, threshold=128):
|
||||
"""Replaces every below-threshold-alpha pixel's RGB with its nearest
|
||||
opaque neighbor's, so the model (RGB-only, see run_tile()) sees a
|
||||
smooth color field instead of a hard edge against flat black -
|
||||
otherwise its convolutions blend genuine edge colors with that black
|
||||
background, leaving dark fringing on the final (correctly re-masked)
|
||||
output."""
|
||||
transparent = np.asarray(alpha) < threshold
|
||||
if not transparent.any() or transparent.all():
|
||||
return rgb
|
||||
_, indices = ndimage.distance_transform_edt(transparent, return_indices=True)
|
||||
arr = np.asarray(rgb)
|
||||
filled = arr[indices[0], indices[1]]
|
||||
return Image.fromarray(np.where(transparent[..., None], filled, arr))
|
||||
|
||||
|
||||
def run_tile(session, input_name, tile_rgb):
|
||||
"""Runs one tile (already at the model's required size) and returns its
|
||||
upscaled RGB as a uint8 HxWx3 array - handles both the with-batch-dim
|
||||
([1,3,H,W]) and without ([3,H,W]) output conventions different ONNX
|
||||
exports use."""
|
||||
arr = tile_rgb.astype(np.float32) / 255.0
|
||||
arr = np.transpose(arr, (2, 0, 1))[np.newaxis, ...]
|
||||
out = session.run(None, {input_name: arr})[0]
|
||||
if out.ndim == 4:
|
||||
out = out[0]
|
||||
out = np.clip(out, 0.0, 1.0) * 255.0
|
||||
return np.transpose(out, (1, 2, 0)).astype(np.uint8)
|
||||
|
||||
|
||||
def upscale_dynamic(session, input_name, rgb_image):
|
||||
return Image.fromarray(run_tile(session, input_name, np.asarray(rgb_image)))
|
||||
|
||||
|
||||
def upscale_tiled(session, input_name, rgb_image, tile_size, native_scale):
|
||||
"""Splits an image larger than the model's fixed input size into
|
||||
overlapping tile_size x tile_size tiles (edge-padding any tile that
|
||||
would otherwise run past the image bounds), runs each individually, and
|
||||
stitches the results back into one canvas - later tiles simply
|
||||
overwrite the small overlap region of earlier ones, which is invisible
|
||||
at sprite scale."""
|
||||
src = np.asarray(rgb_image)
|
||||
h, w = src.shape[:2]
|
||||
canvas = np.zeros((h * native_scale, w * native_scale, 3), dtype=np.uint8)
|
||||
overlap = 16
|
||||
stride = tile_size - overlap
|
||||
|
||||
y = 0
|
||||
while True:
|
||||
ty = min(y, max(0, h - tile_size))
|
||||
x = 0
|
||||
while True:
|
||||
tx = min(x, max(0, w - tile_size))
|
||||
tile = src[ty:ty + tile_size, tx:tx + tile_size]
|
||||
th, tw = tile.shape[:2]
|
||||
if th < tile_size or tw < tile_size:
|
||||
tile = np.pad(tile, ((0, tile_size - th), (0, tile_size - tw), (0, 0)), mode="edge")
|
||||
out = run_tile(session, input_name, tile)[: th * native_scale, : tw * native_scale]
|
||||
oy, ox = ty * native_scale, tx * native_scale
|
||||
canvas[oy:oy + th * native_scale, ox:ox + tw * native_scale] = out
|
||||
if tx + tile_size >= w:
|
||||
break
|
||||
x += stride
|
||||
if ty + tile_size >= h:
|
||||
break
|
||||
y += stride
|
||||
return Image.fromarray(canvas)
|
||||
|
||||
|
||||
def main():
|
||||
model_path, in_dir, out_dir, scale_arg = sys.argv[1:5]
|
||||
model_name, in_dir, out_dir, scale_arg = sys.argv[1:5]
|
||||
scale = int(scale_arg)
|
||||
|
||||
models_dir = os.environ["UPSCALE_MODELS_DIR"]
|
||||
model_path = os.path.join(models_dir, f"{model_name}.onnx")
|
||||
if not os.path.isfile(model_path):
|
||||
available = sorted(f[:-5] for f in os.listdir(models_dir) if f.endswith(".onnx"))
|
||||
sys.exit(f"unknown upscale model {model_name!r} - available: {', '.join(available)}")
|
||||
|
||||
session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
|
||||
input_name = session.get_inputs()[0].name
|
||||
input_info = session.get_inputs()[0]
|
||||
input_name = input_info.name
|
||||
# A fixed (integer) height/width means this model only ever accepts
|
||||
# that exact input size - a dynamic one (reported as a symbolic name
|
||||
# such as "height") accepts any size in one pass. For a fixed-size
|
||||
# model, its own declared output shape gives the native scale factor
|
||||
# directly, with no need to run it just to find out.
|
||||
fixed_size = input_info.shape[2] if isinstance(input_info.shape[2], int) else None
|
||||
native_scale = session.get_outputs()[0].shape[-1] // fixed_size if fixed_size else None
|
||||
|
||||
names = sorted(n for n in os.listdir(in_dir) if n.lower().endswith(".png"))
|
||||
for name in names:
|
||||
img = Image.open(os.path.join(in_dir, name)).convert("RGBA")
|
||||
rgb = img.convert("RGB")
|
||||
alpha = img.split()[3]
|
||||
rgb = extend_opaque_color(rgb, alpha)
|
||||
|
||||
# The model has no concept of transparency (3-channel RGB in/out) -
|
||||
# run it on the flattened RGB only, and scale the alpha mask
|
||||
# separately with a plain resize. Alpha edges don't need AI
|
||||
# enhancement, just to end up the right size.
|
||||
arr = np.asarray(rgb, dtype=np.float32) / 255.0
|
||||
arr = np.transpose(arr, (2, 0, 1))[np.newaxis, ...]
|
||||
output = session.run(None, {input_name: arr})[0]
|
||||
output = np.clip(output[0], 0.0, 1.0) * 255.0
|
||||
output = np.transpose(output, (1, 2, 0)).astype(np.uint8)
|
||||
upscaled_rgb = Image.fromarray(output, "RGB")
|
||||
if fixed_size is not None and (img.width > fixed_size or img.height > fixed_size):
|
||||
upscaled_rgb = upscale_tiled(session, input_name, rgb, fixed_size, native_scale)
|
||||
elif fixed_size is not None:
|
||||
padded = np.pad(np.asarray(rgb), ((0, fixed_size - img.height), (0, fixed_size - img.width), (0, 0)), mode="edge")
|
||||
out = run_tile(session, input_name, padded)
|
||||
upscaled_rgb = Image.fromarray(out[: img.height * native_scale, : img.width * native_scale])
|
||||
else:
|
||||
upscaled_rgb = upscale_dynamic(session, input_name, rgb)
|
||||
|
||||
target_size = (img.width * scale, img.height * scale)
|
||||
if upscaled_rgb.size != target_size:
|
||||
|
||||
@@ -470,13 +470,13 @@ func writeIndex(path string, entries []galleryEntry) error {
|
||||
`<label>Model: <select id="model">` +
|
||||
`<option>realesrgan-x4plus-anime</option>` +
|
||||
`<option>realesrgan-x4plus</option>` +
|
||||
`<option>realesr-animevideov3</option>` +
|
||||
`<option>realesr-general-x4v3</option>` +
|
||||
`</select></label>` +
|
||||
`<label>Scale: <input id="scale" type="number" value="2" min="1" max="4" style="width:4em"></label>` +
|
||||
`<button id="clear-sel" type="button">Clear selection</button>` +
|
||||
`<br><label>Filter: <input type="text" id="search" placeholder="id, language, file, transparent..." style="width:22em"></label>` +
|
||||
`<span id="shown-count"></span>` +
|
||||
`<textarea id="mdl-output" rows="12" readonly placeholder="Select animations below to generate an MDL upscale definition here - copy it into modding/mods/<name>/upscale/<label>.json"></textarea>` +
|
||||
`<textarea id="mdl-output" rows="12" readonly placeholder="Select animations below to generate an MDL upscale definition here - copy it into modding/mods/<name>/bitmaps/<label>.json"></textarea>` +
|
||||
`</div>`)
|
||||
|
||||
b.WriteString(`<h2>Selected (<span id="basket-count">0</span>)</h2><div id="basket"></div>`)
|
||||
|
||||
+133
-17
@@ -14,6 +14,7 @@ import (
|
||||
"image"
|
||||
"image/color"
|
||||
"image/png"
|
||||
"math"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
@@ -46,8 +47,8 @@ var languageByName = map[string]resource.Language{
|
||||
|
||||
// upscaleDef is one modding/mods/<name>/bitmaps/<label>.json file: a shared
|
||||
// set of upscale parameters and every resource to apply them to. Model
|
||||
// is not consulted yet - mdlp-image bakes in a single ONNX model, applied
|
||||
// regardless of this field's value.
|
||||
// names a .onnx file baked into mdlp-image - see upscale_batch.py for the
|
||||
// set currently available.
|
||||
type upscaleDef struct {
|
||||
Model string `json:"model"`
|
||||
Scale int `json:"scale"`
|
||||
@@ -282,11 +283,95 @@ func bitmapToImage(bmp *bitmap.Bitmap, pal bitmap.Palette) *image.RGBA {
|
||||
// sprite renders at the right size but visibly offset from where it should
|
||||
// sit. Writing {0,0} here (as if there were no anchor) makes the renderer
|
||||
// fall back to a path that ignores that patch's size override entirely.
|
||||
func clamp255(v float64) byte {
|
||||
return byte(math.Max(0, math.Min(255, v)))
|
||||
}
|
||||
|
||||
// isPureRampColor flags a palette entry from one of SS1's synthetic "pure
|
||||
// color ramp" sections (used for UI/effect colors) rather than naturalistic
|
||||
// sprite shading: either two of its three channels are exactly zero (a pure
|
||||
// red/green/blue ramp), or one channel is exactly zero and the other two
|
||||
// are exactly equal (a pure cyan/magenta/yellow ramp). A muted shading
|
||||
// color, even a dark or desaturated one, never has two channels in an exact
|
||||
// relationship like this - skin tones and dyed cloth have no reason to - so
|
||||
// these entries are excluded from upscale re-quantization entirely rather
|
||||
// than relying on a saturation/distance threshold to rule them out case by
|
||||
// case: a target this dark reads as "saturated" under any max/min-based
|
||||
// metric even when it's visually indistinguishable from a neutral dark
|
||||
// gray, which is exactly what let these entries win a plain nearest-color
|
||||
// search in the first place.
|
||||
func isPureRampColor(c color.Color) bool {
|
||||
r, g, b, _ := c.RGBA()
|
||||
r, g, b = r>>8, g>>8, b>>8
|
||||
zeros := 0
|
||||
if r == 0 {
|
||||
zeros++
|
||||
}
|
||||
if g == 0 {
|
||||
zeros++
|
||||
}
|
||||
if b == 0 {
|
||||
zeros++
|
||||
}
|
||||
if zeros >= 2 {
|
||||
return true
|
||||
}
|
||||
return zeros == 1 && ((r == 0 && g == b) || (g == 0 && r == b) || (b == 0 && r == g))
|
||||
}
|
||||
|
||||
// nearestSpriteColor finds the closest palette entry to target by plain
|
||||
// squared-RGB distance, skipping Bitmapper's own reserved indices (outside
|
||||
// 0x01-0x02, 0x08-0x0A, 0x20-0xFF) and the pure-ramp entries isPureRampColor
|
||||
// flags. Squared-RGB rather than bitmap.Bitmapper's Lab-space search: close
|
||||
// enough for re-quantizing an already-upscaled sprite, and avoids
|
||||
// duplicating that package's unexported Lab-distance machinery just to add
|
||||
// the ramp exclusion.
|
||||
func nearestSpriteColor(palColors []color.Color, target color.RGBA) byte {
|
||||
tr, tg, tb, _ := target.RGBA()
|
||||
var best byte
|
||||
bestDist := math.Inf(1)
|
||||
for i, pc := range palColors {
|
||||
isRegularColor := (i >= 0x01 && i <= 0x02) || (i >= 0x08 && i <= 0x0A) || i >= 0x20
|
||||
if !isRegularColor || isPureRampColor(pc) {
|
||||
continue
|
||||
}
|
||||
pr, pg, pb, _ := pc.RGBA()
|
||||
dr := float64(tr>>8) - float64(pr>>8)
|
||||
dg := float64(tg>>8) - float64(pg>>8)
|
||||
db := float64(tb>>8) - float64(pb>>8)
|
||||
if dist := dr*dr + dg*dg + db*db; dist < bestDist {
|
||||
bestDist = dist
|
||||
best = byte(i)
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
func imageToBitmap(img *image.RGBA, pal bitmap.Palette, transparent bool, origAnchor bitmap.Area) *bitmap.Bitmap {
|
||||
bounds := img.Bounds()
|
||||
w, h := bounds.Dx(), bounds.Dy()
|
||||
bitmapper := bitmap.NewBitmapper(&pal)
|
||||
palColors := pal.ColorPalette(false)
|
||||
pixels := make([]byte, w*h)
|
||||
|
||||
// Floyd-Steinberg error diffusion: a straight nearest-palette-color map
|
||||
// (SS1's fixed 256-color palette, no per-sprite palette) can push an AI
|
||||
// upscaler's smooth color gradients onto a visibly wrong palette entry
|
||||
// wherever the true color sits in a gap between two distant entries -
|
||||
// diffusing each pixel's quantization error into its unprocessed
|
||||
// neighbors spreads that mismatch into unnoticeable dithering noise
|
||||
// instead of a solid wrong-colored patch.
|
||||
errR := make([]float64, w*h)
|
||||
errG := make([]float64, w*h)
|
||||
errB := make([]float64, w*h)
|
||||
addErr := func(x, y int, er, eg, eb, weight float64) {
|
||||
if x < 0 || x >= w || y < 0 || y >= h {
|
||||
return
|
||||
}
|
||||
i := y*w + x
|
||||
errR[i] += er * weight
|
||||
errG[i] += eg * weight
|
||||
errB[i] += eb * weight
|
||||
}
|
||||
for y := 0; y < h; y++ {
|
||||
for x := 0; x < w; x++ {
|
||||
c := img.RGBAAt(bounds.Min.X+x, bounds.Min.Y+y)
|
||||
@@ -294,7 +379,32 @@ func imageToBitmap(img *image.RGBA, pal bitmap.Palette, transparent bool, origAn
|
||||
pixels[y*w+x] = 0
|
||||
continue
|
||||
}
|
||||
pixels[y*w+x] = bitmapper.MapColor(color.RGBA{R: c.R, G: c.G, B: c.B, A: 0xff})
|
||||
i := y*w + x
|
||||
r, g, b := float64(c.R)+errR[i], float64(c.G)+errG[i], float64(c.B)+errB[i]
|
||||
idx := nearestSpriteColor(palColors, color.RGBA{R: clamp255(r), G: clamp255(g), B: clamp255(b), A: 0xff})
|
||||
pixels[i] = idx
|
||||
pr, pg, pb, _ := palColors[idx].RGBA()
|
||||
er := r - float64(pr>>8)
|
||||
eg := g - float64(pg>>8)
|
||||
eb := b - float64(pb>>8)
|
||||
// Skip diffusion into transparent neighbors so dithering noise
|
||||
// never bleeds past the sprite's own silhouette.
|
||||
rightOK := !(transparent && x+1 < w && img.RGBAAt(bounds.Min.X+x+1, bounds.Min.Y+y).A < 0x80)
|
||||
belowOK := y+1 < h && !(transparent && img.RGBAAt(bounds.Min.X+x, bounds.Min.Y+y+1).A < 0x80)
|
||||
belowLeftOK := y+1 < h && x > 0 && !(transparent && img.RGBAAt(bounds.Min.X+x-1, bounds.Min.Y+y+1).A < 0x80)
|
||||
belowRightOK := y+1 < h && x+1 < w && !(transparent && img.RGBAAt(bounds.Min.X+x+1, bounds.Min.Y+y+1).A < 0x80)
|
||||
if rightOK {
|
||||
addErr(x+1, y, er, eg, eb, 7.0/16.0)
|
||||
}
|
||||
if belowLeftOK {
|
||||
addErr(x-1, y+1, er, eg, eb, 3.0/16.0)
|
||||
}
|
||||
if belowOK {
|
||||
addErr(x, y+1, er, eg, eb, 5.0/16.0)
|
||||
}
|
||||
if belowRightOK {
|
||||
addErr(x+1, y+1, er, eg, eb, 1.0/16.0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -370,13 +480,10 @@ func upscaleBatch(all []resourceFrames, pal bitmap.Palette, model string, scale
|
||||
fmt.Printf(" upscaling %d resource(s), %d frame(s) total (model %s, scale %d):\n",
|
||||
len(all), totalFrames, model, scale)
|
||||
|
||||
// UPSCALE_MODEL_PATH (set by mdlp-image's Dockerfile) points at the ONNX
|
||||
// model baked into the image.
|
||||
modelPath := os.Getenv("UPSCALE_MODEL_PATH")
|
||||
if modelPath == "" {
|
||||
return nil, fmt.Errorf("UPSCALE_MODEL_PATH is not set - see mdlp-image/Dockerfile")
|
||||
}
|
||||
cmd := exec.Command("python3", "/usr/local/bin/upscale_batch.py", modelPath, inDir, outDir, strconv.Itoa(scale))
|
||||
// upscale_batch.py resolves model (this def's own "model" field) against
|
||||
// $UPSCALE_MODELS_DIR (set by mdlp-image's Dockerfile) itself, and fails
|
||||
// with the available model names if it doesn't recognize this one.
|
||||
cmd := exec.Command("python3", "/usr/local/bin/upscale_batch.py", model, inDir, outDir, strconv.Itoa(scale))
|
||||
// Streamed live, not buffered via CombinedOutput: a single directory
|
||||
// batch can run for a while, and silence until the whole thing exits
|
||||
// gives no sense that it's still making progress.
|
||||
@@ -426,17 +533,26 @@ func toRGBA(img image.Image) *image.RGBA {
|
||||
return converted
|
||||
}
|
||||
|
||||
// encodeFrames RLE-diffs each frame against the previous, matching the
|
||||
// convention hacked's own editor/animations/View.go uses when importing a
|
||||
// GIF (the first frame diffs against an implicit all-zero reference).
|
||||
// encodeFrames RLE-compresses each frame against an all-zero reference (the
|
||||
// "still image" convention - see ss-specs media/Bitmaps.md). The RLE format
|
||||
// also grammatically allows a "skip" run to mean "copy from a reference
|
||||
// image" rather than "write zero" (the "video clip" convention, which
|
||||
// hacked's own GIF-import path uses) - but the game's actual RSD8 decoder
|
||||
// (engine/src/Libraries/2D/Source/RSD/{rsdcvt.c,RSDUnpack.c}, confirmed
|
||||
// against Night Dive's published original source down to the hand-written
|
||||
// 68K assembly) only ever implements the zero-reference case, for every
|
||||
// resource type checked including critter sprites: it writes literal zero
|
||||
// for every skip run unconditionally, with no mechanism to carry a
|
||||
// reference image's pixels forward at all. Compressing against the real
|
||||
// previous frame here produced RLE streams only hacked's own reader could
|
||||
// decode correctly - the game rendered scattered noise wherever a frame
|
||||
// carried content unchanged from the one before it.
|
||||
func encodeFrames(frames []*bitmap.Bitmap) [][]byte {
|
||||
blocks := make([][]byte, 0, len(frames))
|
||||
var prevPixels []byte
|
||||
for _, bmp := range frames {
|
||||
buf := new(bytes.Buffer)
|
||||
_ = binary.Write(buf, binary.LittleEndian, &bmp.Header)
|
||||
_ = rle.Compress(buf, bmp.Pixels, prevPixels)
|
||||
prevPixels = bmp.Pixels
|
||||
_ = rle.Compress(buf, bmp.Pixels, nil)
|
||||
blocks = append(blocks, buf.Bytes())
|
||||
}
|
||||
return blocks
|
||||
|
||||
Reference in New Issue
Block a user