This project is a consolidation of various compiled open-source AI image/video
upscaling product for a working CLI-friendly image and video upscaling program.
For these reasons:
- Low-cost image/video AI upscaling software - run locally in your laptop with an AI solution.
- Programmable - when you upscale an album or a video, the AI program has to be programmable and not restricted by any GUI's design.
- Reliabily working for big subject - video files are usually large and require streaming algorithm approach to prevent uncontrollable resources consumption in a simple OS system (e.g. disk space, RAM, and vRAM).
- I urgently need a video upscaling technologies to work locally - for both image and video without any GUI overheads.
This repository was made possible by the following contributors:
- Joly0
- Windows support via PowerShell.
- Cory Galyna - Repository & CI management.
- Jean Shuralyov - Documentations.
- (Holloway) Chew, Kean Ho
- FFMPEG & UNIX Support via POSIX Shell and Polygot Script.
Here are the tested hardware and operating system:
System | Results | Usable Processing Units |
---|---|---|
debian-amd64 (linux) |
PASS |
NVIDIA GeForce MX150 , Intel(R) UHD Graphics 620 (KBL GT2) |
darwin-amd64 (macOS) |
FAILED |
Binary failed to use Intel Iris Graphics iGPU and CPU. |
windows-amd64 (windows) |
PASS |
Nvidia Quadro T600 , Intel Iris Xe Graphics |
IMPORTANT NOTES
(1)
You seriously need a compatible GPU to drastically speed up the upscaling efforts from hours to seconds for image. I tested mine against
NVIDIA GeForce MX150
vs.Intel(R) UHD Graphics 620 (KBL GT2)
built-in graphic hardwares in my laptop. It did a huge difference.(2)
At the moment, the algorithm only works on constant bit rate video. Variable bit rate (VBR) video are considered but may be done in the future with proper programming language. Most (as in 99% of video) are constant bit rate video so VBR support is at least concern.
(3)
Not all images can be upscaled (e.g. some AI generated images from Stable Diffussion) due to the binary's internal image decoder bug. Tracking issue: xinntao/Real-ESRGAN#595
(4)
Re-packaging efforts are unlikely because currently it's a bad investment. It's either I spend the time study NCNN and write the whole XinTao's RealESRGAN-Vulcan program as my own codes from scratch or glued some existing programs together. At the moment, I do not have the resources to rewrite or study the NCNN (yet).
NOTE TO MacOS USERS
The binary
bin/mac-amd64
is currently unsigned. Hence, you need to explictly grant the use permission in yourSettings > Security & Privacy
section.Please be informed that my test result is in accordance with the Upscayl team: many CPU and iGPUs are not working and supported yet.
If you're working on video, you need ffmpeg
and ffprobe
for dissecting and
reassembling a video file.
You can proceed to install them in 1 go at: https://ffmpeg.org/
You need to install Microsoft Visual C++ Redistributable Package from their official website if you haven't do so. Usually other software may automatically include a version installed already. Please check your "Add/Remove Program" control panel to verify.
Here are the basic user manuals:
There are different ways of installing this program depending on your interested versions:
You can download the latest version of the upscaler-[VERSION].zip
package from https://github.com/hollowaykeanho/Upscaler/releases
and then unzip it to an appropriate location. Note that the models are already
included in this package.
For those who wants to just update the models, the
upscaler-models-[VERSION].zip
is made available for you. Simply overwrite
the models/
directory's contents in the software.
For those who wants to run test and benchmarks for the repository, the
upscaler-tests-[VERSION].zip
is made available for you. Simply integrate
the tests/
directory into your existing software program (the unpacked
upscaler-[VERSION].zip
).
You need to git clone
the repository into an appropriate location.
$ git clone https://github.com/hollowaykeanho/Upscaler.git
We advise you to symlink the start.cmd
into $PATH
or %PATH%
directory
depending on your operating system. Example, on Debian Linux
:
$ ln -s /path/to/Upscaler/start.cmd /path/to/bin/upscaler
Alternatively, on UNIX (Linux & Mac) systems, you can create a shell script that
pass all arguments into the start.cmd
. Example:
#!/bin/sh
/path/to/Upscaler/start.cmd "$@"
TIP: if had you decided to use the shell script approach, you can also design the command to use your default model and scaling for your programming efficiencies. Recommend you use
$HOME/bin
directory if it is set visible in your$PATH
value.
This repository was unified using
Holloway's Polygot Script to
keep user instruction extremely simple. Hence, in any operating system (UNIX or
WINDOWS), simply interact with the repository's start.cmd
script will do.
Example for requesting a help instruction:
$ ./Upscaler/start.cmd --help
In the help display, you generally want to take notice of the AVAILABLE MODELS
list. The upscaling algorithms are solely based on the available models in the
models directory.
The scripts are written in a way to dynamically index each of them and present
it in the help display without re-writing itself.
To upscale an image simple run start.cmd
against the image:
$ ./start.cmd \
--model ultrasharp \
--scale 4 \
--format webp \
--input my-image.jpg
To determine the available models, their respective scale limits, and their
output formats, simple execute the --help
and look for: AVAILABLE MODELS
and
AVAILABLE FORMATS
respectively.
If done correctly, an image based on orginal filename with a suffix -upscaled
is created. If we follow the example above, it should be
my-image-upscaled.webp
.
Unless you're working on 8 seconds 8MB sized video, you would want to follow the instructions below to make sure your project are always in-tact and resumeable.
Please keep in mind that you need at least 3x video size storage for the job depending on the video frame rate, frame size, color schemes and etc. The minimum 3x is due to:
- 1 set is your original video.
- 1 set is for all the upscaled images (can be a lot bigger since we're doing it frame by frame; losing the video compression effect).
- 1 set is your output video (bigger than original of course).
Hence, please plan out your storage budget before starting a video upscaling project.
IMPORTANT
Know your hardware limitations before determining the scaling factor. A scale of 4x on a 1090p for a 12GB memory laptop can crash the entire OS (I'm referring the very stable Debian OS) during the video re-assembly phase with FFMPEG due to memory starvation.
You're advised to create a project directory for upscaling video project due to its large sized data.
Instead of executing the start.cmd
straight away, please script it inside and
place it in a project directory.
A simple UNIX example would be a shell script (e.g. run.sh
) as follows:
#!/bin/sh
/path/to/Upscaler/start.cmd --model ultrasharp \
--scale 4 \
--input ./sample-vid.mp4 \
--video
A simple WINDOWS example would be a batch script (e.g. run.bat
) as follows:
@echo off
/path/to/Upscaler/start.cmd --model ultrasharp ^
--scale 4 ^
--input ./sample-vid.mp4 ^
--video
This is for resuming the upscaling project in case of crashes or long hours work. A good directory looks something as follows:
/path/to/my-upscaled-project/
├── run.sh
└── sample-vid.mp4
To initiate or to resume project, simply run your initiated project:
# cd /path/to/my-upscaled-project
$ ./run.sh
The Upscaler will create a workspace to house its output frames and control values. You can inspect the frame images while Upscaler is at work as long as you're viewing the frame that it is working on.
I recommend you inpect the frames first for determining whether the AI model is suitable or otherwise. Otherwise, please stop the process and execute the next step.
Just in case if you bump into something odd that requires to restart the project
from start, you can delete the [FILENAME]-workspace
directory inside the
project directory. This will force the program to restart everything all over
again.
The project is set to intentionally leave the workspace as it is in case you need to use the frames for other purposes (e.g. thumbnail).
Star, Watch, or the best: sponsor the contributors.
In case you need to contribute back:
- Raise an issue ticket.
- Fork the repository and work against the
main
branch. - Develop your contributions and ensure your commit are GPG-signed.
- Once done, DO NOT raise a pull request unless instructed. Simply notify
me via your ticket and I will clone your forked repo locally and
cherry-pick
them (I need those GPG signatures preserved where GitHub Pull Request cannot fulfill). An exception would be the you have a need to earn the GitHub Pair Extraordinaire Badge. For doing that, please add me into your forked repo and I will make it happen for you. - Delete your fork repo once I notify you that the merging is completed.
- Thank you.
Submit results of benchmarks running tests/benchmark.cmd
at the root of the
repository. These data serves few purposes:
- To test the repository's programs are running properly for both video and image upscaling.
- To identify what platform, OS, and hardware capable of running this project. (good for determining usability before procurement).
- To know about its statistical performances.
IMPORTANT NOTE
The
benchmark.cmd
captures wall-clock timing. Hence, please leave the system dedicated to only running the benchmark and not doing something else for maintaining results consistencies. We're capturing wall-clock timing for real time use (e.g. subjected to OS and background processes interferences).Our recommendation is to leave it run overnight before you sleep (~400 frames) so it takes some time.
Version | Sample 1 (Video) |
---|---|
v0.7.0 |
9587 seconds |
v0.6.0 |
9260 seconds |
v0.5.0 |
9192 seconds |
v0.4.0 |
10316 seconds |
Version | Sample 1 (Video) |
---|---|
v0.6.0 |
3649 seconds |
Version | Sample 1 (Video) |
---|---|
v0.7.0 |
11633 seconds |
In case you can't access to the help details found from the --help
command,
here's a copy from the UNIX side:
u0:Upscaler$ ./start.cmd --help
HOLLOWAY'S UPSCALER
-------------------
COMMAND:
$ ./start.cmd \
--model MODEL_NAME \
--scale SCALE_FACTOR \
--format FORMAT \
--parallel TOTAL_WORKING_THREADS # only for video upscaling (coming soon) \
--video # only for video upscaling \
--input PATH_TO_FILE \
--output PATH_TO_FILE_OR_DIR # optional
EXAMPLES
$ ./start.cmd \
--model ultrasharp \
--scale 4 \
--format webp \
--input my-image.jpg
$ ./start.cmd \
--model ultrasharp \
--scale 4 \
--format webp \
--input my-image.jpg \
--output my-image-upscaled.webp
$ ./start.cmd \
--model ultrasharp \
--scale 4 \
--format png \
--parallel 1 \
--video \
--input my-video.mp4 \
--output my-video-upscaled.mp4
$ ./start.cmd \
--model ultrasharp \
--scale 4 \
--format png \
--parallel 1 \
--input video/frames/input \
--output video/frames/output
AVAILABLE FORMATS:
(1) PNG
(2) JPG
(3) ...
AVAILABLE MODELS:
...
This is a binaries assembled repository. You may find the source codes from the original contributors here:
Test sample video in the tests/
directory was supplied by
Igrid North
from
Pexels. Original 4k sized video is also available at
origin for upscaling comparison.
This project is aligned to its upstream sources and is licensed under BSD-3-Clause "New or "Revised" License.