General Roadmap and Plans

[MatthewL246]

My plans for this project

This issue will serve as my plans/roadmap/to-do list for this project.

Small tasks

• Create backup and restore scripts for volumes: definitely MongoDB, Postgres, and MinIO; maybe mitmproxy and Redis
• Ensure NNCS traffic is forwarded to the official Pretendo servers
• Set up a compose.ci.yml to override container restart and ports configuration in CI instead of yq eval
• Migrate from server.local.env to a global .env file
• Fix some infinite loops in the setup scripts, improve error handling in that case
• Create a small update script that resets submodules, pulls, and re-runs the setup script
• Create a full reset script that removes all changes to the repo and backs up and deletes the Docker volumes
• Separate out internal scripts into scripts/internal vs ones that can be run by themselves
• Stop recommending creating a nintendo slot with the GM9 script, Nimbus with the pretendo_official slot is fine
• Make backing up before restoring with the friends account switcher optional
• Recommend the Wii U account swap homebrew for switching users
• Get Wii U account settings to work
• More script refactoring: load environment variables function, move more things from setup.sh to individual scripts, check for other stuff that is copy/pasted
• Fix the Node.js servers taking 10 seconds to restart every time
• Look into creating a proxy autoconfig file, which might be useful for connecting with browsers
• Use Pretendo logger inside run-in-container scripts
• Switch from MailDev to MailPit
• Clearly show when a script fails so people can't accidentally ignore it
• Create a script for generating account.dat and Cemu online files, try using GabIsAwesome/accountfile-generator
• Switch from Redis to Valkey, as it is being used in prod
• Test the scripts in CI
• Switch to https://github.com/PretendoNetwork/BetterKeyDumper for BOSS keys
• Add an option to the scripts to dump the output to a temporary file for issue-reporting purposes. Also include general system info like /etc/os-release, uname -a, lscpu, docker version, repo & submodule commit hashes, git status, and contents of the .env file. Maybe also include container logs?
• Detect if the user incorrectly runs the script in Git Bash and show an error

Larger tasks

• Set up the Docusaurus documentation website: Create a Docusaurus documentation website #66
  • Rewrite the readme to be more of a readme than documentation
  • Generate the documentation automatically on push
• Improve script option parsing (check this)
  • Add a global --help option, manage options centrally and generate help text using add_option() in function-lib
  • Add a global --verbose option to run commands with more output, add run_verbose() to function-lib
  • Add a --force option to bypass prompts
  • Basically, rewrite all the scripts to use this and allow bashisms
• Add more options to the "frontend" networking
  • mitmproxy in proxy mode, useful for consoles and emulators, don't forward nginx ports for convenience (current)
  • mitmproxy in reverse proxy mode, paired with a DNS server (coredns), useful for exposing a service publicly
  • SSSL mode, using something like the SSSL-DNS Dockerfile to let the console connect directly to nginx with an SSSL certificate: Add Pretendo DNS support for hackless client deployment #74
  • The most difficult part is trying to get everything running at the same time. Maybe nginx-sssl => mitmproxy-reverse => nginx => servers? Where does mitmproxy-pretendo come into the chain? Does the 3DS with SSL patches connect directly to nginx-sssl? Should inter-server communication go through mitmproxy-reverse?
• Reconsider emulator redirection strategies: proxy server is not reliable for Cemu, see When i try to connect co mario maker DNS error. #56
  • Might make sense to switch to mitmproxy reverse + DNS strategy
  • Docs will need instructions for switching system DNS server to local or using hosts file as backup. If the user is already using a PiHole this could be even easier!
• Add Citra setup guide - there was one on Discord already
  • Use the Internet Archived setup guide, does the latest download version work?
  • Importing the friends and account system module save data seemed to work
  • The Pretendo patches do work on Citra
  • Using a local Pretendo server (127.0.0.1 nasc.pretendo.cc and account.pretendo.cc in hosts file) works fine
  • Remember for guide: import the mii maker save data too
• Check for nex-go-rewrite branches and switch to them
• Sometimes, the servers change in ways that require database migrations for existing setups. For example, database: Update database defaults and add country PretendoNetwork/friends#15. This is currently not handled and needs to be done manually or it will cause errors after existing setups update. Add some sort of functionality for this to the update script for migration logic.
  • Search repo for "migrate" like https://github.com/PretendoNetwork/friends/pulls?q=migrate
• Some changes that require modifications outside the server, like an update to console CFW or patches. There is no way to communicate that currently, and keeping some sort of state seems complicated.
• Make the server setup significantly more configurable
  • Allow configuring the IP address of internal/admin services to not just be 127.0.0.1 (use a .env file, Docker Compose supports this)
  • Separate different servers into different Compose files and allow only running some services.
  • Support setups using custom domains instead of a proxy server
  • Only build images when necessary, otherwise use the container registry from the container registry

More servers

• Internal Pretendo admin panel (disabled by default, for internal developers only due to being closed-source)
• Eshop stuff? (Grove Eshop UI and SOAP NUS). It would be cool to run this since Pretendo will probably never run it themselves publicly, but what would you do with it? No real point in serving homebrew apps.
• Basically all of the public game servers
  • Remember https://github.com/OatmealDome/Rotationator for Splatoon
• Alternate Miiverse servers: Rverse
• Potentially other third-party Wii U/3DS servers with no relationship to Pretendo. I'm unsure about this because these are outside the scope of this project, even though they might be useful.
  • ImageShare as a 3DS Image Share replacement
  • NetPass server (would probably be unnecessary with the built-in StreetPass Relay server in BOSS)
  • SatellaZone (but the server doesn't seem to be open source?)

Long-term goals

• In general, making this a usable development environment instead of (or as well as) a self-hosting setup
  • Don't require a devcontainer workflow - set up host.docker.internal
  • Port-forward everything necessary for locally running another server and write the secrets
  • Contribute the patches upstream to the Pretendo servers
  • Document development using the 2 workflows - devcontainers or running the service on the host
  • Make all of the host-based routing in the individual servers optional, use ports for subdomains instead
• Get an infrastructure developer to review the configuration and scripts for issues
• Discuss with the Pretendo developers about making this the official Pretendo Network selfhosting/development environment
• Potentially, migrate from Bash scripts to a full server management CLI application