Auto Translate system

[DiscipleOfEris]

Hey, we've been doing research on how Auto Translate codes work in the game. We believe we know enough now to have a feature-complete system for LSB. Just need to figure out how it should look.

I'm imagining the finished system might look something like this, in broad strokes:

A script/utility or extracting AT info from the DATs or other DAT-derived sources into a SQL file. I'd like to put this into LSB's UpdateExtractor, but the MassExtractor.exe dependency doesn't seem to get the AT strings, to my knowledge. Windower's ResourceExtractor does grab these.

The SQL file will setup a table that has columns (names open to change): ID (4 byte code that the game's AT codes use), English (English plain text), Japanese?, type (enum representing string, item, or key_item), and ID2 (2 byte uint that matches the actual ID of the item or key_item, since the AT code doesn't always map correctly).

A cpp util that loads the AT table on init. It should have methods for converting from plain text to AT code, from AT code to plain text (choice of English or Japanese?), from AT code to AT type (string, item, key item), and finally from AT code to item id or key item id.

A system that filters chat messages and strips all AT codes that are not valid AT codes.

A Lua bridge for accessing AT features from Lua?

Bert and I have been working on this for another project (with Bert doing the lion's share of the work so far). Bert wrote this utility that takes output from Windower's ResourceExtractor and creates a SQL file: https://github.com/bsips/AutoTParser

[DiscipleOfEris]

This is the work LSB has so far: https://github.com/LandSandBoat/server/blob/04ed9c7ed256aa8c1ed64d86b249bcb2acd21644/src/map/autotranslate.cpp

[bsips]

#3626 opened for version 30230204_0 for additional reference

[bsips]

I like the idea of a db table and building on init. But I'm unsure of how much that adds to init time/space + what LSB folks feel would be best going forward.

Bottom line is we have at least some primitive methodology for deriving values for future updates (currently a tiny bit tedious due to all the extra escape characters needed).

[TeoTwawki]

Startup time is almost a non issue in my opinion. The current PR adds so much at once that is actually beyond github's "mechanical limits" to show me. Usually in such cases we'd break in in multiple commits adding to the same file (where possible) but if this can be automated and just become part of our client update process we won't have to worry aboput manual verification so much. Being in a db table would certainly go a long way towards simplifying automation - not that we can't automate it if its purely in src, just that its much easier to tool for sql. But Zach and Clay would know more about that process of adding this in.

[claywar]

My view is that I would not want to generate on init if it depends on parsing client files. I don't like the idea of having dependencies on having the client data accessible on the machine that is running the server (if I understand correctly).

Now as a part of the monthly update process, I am absolutely onboard with finding a way to maintain without having to touch core.

Even if it weren't for inconsistency, new column is more work than new table.

[bsips]

I don't like the idea of having dependencies on having the client data accessible on the machine that is running the server (if I understand correctly).

I understand the concern (i think).

Currently, the way we are making this work is by using the Windower teams ResourceExtractor to pull some xml files. They look like this:

From auto_translates.xml (Questions, Zone names, Greetings, Weapon skill/Job ability names, etc)

<o id="257" en="Nice to meet you." ja="初めまして。" />

From items.xml

<o id="637" flags="4" stack="12" type="1" targets="0" category="General" en="Slime Oil" enl="vial of slime oil" ja="スライムオイル" jal="スライムオイル" />

From key_items.xml

<o id="9" en="airship pass for Kazham" category="Permanent Key Items" ja="カザム行き飛空艇パス" />

We're simply pulling the ids and ens from all of these, and then transforming the ids into 32bit values that the autotranslate system uses/wants.

Currently, we would either need to get those XML's one time on a client update, from any updated client, and have the UpdateExtractor handle the transformations -> .sql script, or find another way to get the complete en+ids (looks like UpdateExtractor already works with items and keyitems, so the real hurdle is the remaining "auto_translates").