Changing Filenames per their Formats

[BenjamenMeyer]

NOTE: The goal of this conversation is to identify tasks and then assign them to the Assets and Engine accordingly, after which we can figure out the priorities against release targets.

Starting a discussion around #35 based on findings from @P-D-E who did some research posted to our Gitter chat:

About #35 I took a peek at the data files and found some info hopefully useful for the suggested tool to come, e.g. what it must do (update references to renamed files in txt/html files) and what it can't do (e.g. break hardcoded references in source files, unless code is updated accordingly).
Here's what I've found so far, sorry for the wall of text:

animations
*.ani, txt referring images

animations/blackclear.ani/blackclear.ani -> blackclear.png, no path but it's in textures/blackclear.png; is textures a default fallback?
blackclear.png is a png

animations/blink.ani/blink.ani -> blink00??.texture, same folder
blink00??.texture are dds

animations/whitelight.ani/whitelight.ani -> black.texture, blink00??.texture, same folder
black.texture is a png
blink00??.texture are dds



cockpits
*.sprite, txt referring images
*.xmesh, xml referring images

cockpits/bomber-cockpit.cpt/armorf.sprite -> armorf.image armorf.image
armorf.image is a dds

cockpits/bomber-cockpit.cpt/cockpit-bomber.xmesh
<Mesh texture="metal47.png" scale="60"> no path but it's in textures/
metal47.png is a dds



meshes
*.xmesh, xml referring images

meshes/aeraburn.xmesh
<Mesh texture="thrust2.png" forcetexture="true" blend="ONE ONE"> no path but it's in textures/
thrust2.png is a dds

meshes/mounts/ion.xmesh
<Mesh texture="ion.png">no path but it's in textures/mounts/ [1]
ion.png is a dds
[1] recursive scan at work, or a `/mounts/` subfolder match? would explicit paths speed up anything?

meshes/supernova.bmp is a bfxm; is it actually used? supernova.bmp.bfxm seems to be the referenced one



sprites
*.sprite, *.spr, txt referring images
*.alp, identified by `file` as PC bitmap, Windows 3.x format (5 here, 1 in textures); rename as bmp?

sprites/bases/aera/aera_planet.sprite -> bases/aera/aera_planet.image true (explicit path despite same folder)
aera_planet.image is a dds



techniques
*.technique, xml referring images

techniques/fireglass.technique
        <texture_unit src="decal:2" default="file:bubble_damg.texture" name="damageMap"/>
        <texture_unit src="file:grey_metal.png" name="detail0Map"/>
bubble_damg.texture is a png
grey_metal.png is a dds



textures
*.bmp, some are bmp, some are dds
*.png, few are png, most are dds
*.texture, few are png, most are dds
combine.jpg is a dds
backgrounds/*.cube are dds
9x12.alp doesn't identify as bmp like the ones in sprites/ and 9x12.bmp is there; is 9x12.alp useless?

...ok, this is funny; according to`file`,
textures/sol/miranda-cel.texture:         DOS/MBR boot sector; partition 1 2 : ID=0xd2, active 0xff, start-CHS (0x1de,0,56), end-CHS (0xaa,172,0), startsector 3732406442, 2852170962 sectors; partition 2 3 : ID=0xf6, active 0xaa, start-CHS (0x1de,0,56), end-CHS (0x55,205,0), startsector 3730352725, 1426116054 sectors; partition 3 4 : ID=0xf6, active 0xff, start-CHS (0x3ee,2,26), end-CHS (0x255,205,42), startsector 3732471637, 1426116086 sectors
...of course it's a regular DDS, GIMP reads it just fine; there are a few others whose data cause this ID mistake (`file -k` shows all the matches, DDS getting the lowest score)



units
*.sprite, *.spr, txt referring images
All bmp are bmp (just 5)
All jpg are dds
All png are dds EXCEPT this one:
units/vessels/Taizong/taizongGLO.png:	PNG image data, 1 x 1, 8-bit grayscale, non-interlaced



========= hardcoded references, NOT guaranteed as exhaustive =========

engine/objconv/basemaker/base_maker.cpp : .spr, .png, .jpg
Input( "Add a sprite (ESC=cancel, blank=Ship) (No extension. file must be png or jpg format.) ",
might explain why the massive lie about most jpg and png being dds

engine/src/gui/text_area.h : textures/gui/*.png

engine/objconv/mesher/to_OgreMesh.cpp
584:                "\t\t\t\ttexture $(TEXTURE_DIF_FILE=white.png) $(TEXTURE_DIF_TYPE=2d)\n"

engine/objconv/starsystemgen/starsysgen.cpp
612:                names ).c_str(), entitytype == JUMP ? "jump.png" : s.c_str() );

engine/src/galaxy_gen.cpp
378:        entity.push_back( "white_star.png" );
730:    static bool usePNGFilename = (VSFileSystem::LookForFile( "jump.png", VSFileSystem::TextureFile ) <= VSFileSystem::Ok);
731:    static const char *jumpFilename = usePNGFilename ? "jump.png" : "jump.texture";

engine/src/cmd/basecomputer.cpp
748:        picture->setTexture( "blackclear.png" );
883:        picture->setTexture( "blackclear.png" );
1284:        picture->setTexture( "blackclear.png" );
1394:    window()->setTexture( "basecomputer.png" );
*   :            window()->setTexture( "basecomputer_*.png" );

engine/src/cmd/planet.cpp
123:    string     nam   = f.meshname+XMLSupport::tostring( count )+".png";

engine/src/gui/modaldialog.cpp
77:    window->setTexture( "basecomputer.png" );
113:    window->setTexture( "basecomputer.png" );
163:    window->setTexture( "basecomputer.png" );

engine/src/gfx/nav/navcomputer.cpp
204:    window()->setTexture( "basecomputer.png" );
1060:    window->setTexture( "basecomputer.png" );

engine/src/gfx/mesh_gfx.cpp
1358:                static Texture *white = new Texture( "white.png" );
1633:    static Texture *black = new Texture( "blackclear.png" );

engine/src/options.cpp
314:    DefaultAtmosphereTexture = vs_config->getVariable( "galaxy", "DefaultAtmosphereTexture", "sol/earthcloudmaptrans.png" );

engine/src/gamemenu.cpp
299:    window()->setTexture( "basecomputer.png" );
510:    window->setTexture( "basecomputer.png" );

engine/src/faction_util.cpp
63:            fac->logo = FactionUtil::createTexture( "white.png", true );

engine/src/system_factory.cpp
229:    string myfile = getStringAttribute(object, "file", "planets/ring.png");

engine/objconv/asteroidgen.cpp
190:char  texture[100] = "Asteroid.bmp";

engine/objconv/dattoxml.cpp
914:        StrWrite( "<Light file=\"supernova.bmp\" x=" );

engine/src/cmd/weapon_xml.h
103:            file = std::string( "beamtexture.bmp" );

engine/src/cmd/unit_csv.cpp
1368:        static std::string shieldtex = vs_config->getVariable( "graphics", "shield_texture", "shield.bmp" );

engine/src/cmd/unit_xml.cpp
2036:        static std::string shieldtex = vs_config->getVariable( "graphics", "shield_texture", "shield.bmp" );

engine/src/cmd/atmosphere.cpp
62:    dome = new SphereMesh( params.radius, divisions, divisions, "white.bmp", "", NULL, true, ONE, ZERO, false, 0, M_PI/2 );

engine/src/gui/window.cpp
199:    static Texture  dummy( "white.bmp", 0, NEAREST, TEXTURE2D, TEXTURE_2D, GFXTRUE );

engine/src/gui/glut_support.cpp
188:    static Texture  dummy( "white.bmp", 0, NEAREST, TEXTURE2D, TEXTURE_2D, GFXTRUE );

engine/src/gfx/env_map_gent.cpp
328:                    suffix = ".bmp";
471:        err = f.OpenReadOnly( strcat( tmp, "_sphere.bmp" ), TextureFile );
477:                err = f.OpenReadOnly( strcat( tmp, "_up.bmp" ), TextureFile );

engine/src/gfx/particle.cpp
199:        std::string s = vs_config->getVariable( "graphics", prefix + "texture", "supernova.bmp" );

engine/src/gfx/aux_texture.cpp
658:    static Texture *white = new Texture( "white.bmp", 0, MIPMAP, TEXTURE2D, TEXTURE_2D, 1 );

engine/src/gfx/star.cpp
630:                decal[curtexture] = new Texture( "white.bmp", 0, near_stars_alpha ? NEAREST : BILINEAR );

engine/src/gfx/background.cpp
71:        temp = string( file )+"_up.bmp";
75:            suffix = ".bmp";              //backwards compatibility
81:            temp   = string( file )+"_sphere.bmp";
82:            suffix = ".bmp";             //backwards compatibility

engine/src/gfx/quadtree.cpp
81:     *  char name[]="terrainX.bmp";

engine/src/gfx/ani_texture.cpp
79:    static Texture *white = new Texture( "white.bmp", 0, MIPMAP, TEXTURE2D, TEXTURE_2D, 1 );
594:    char temp[512] = "white.bmp";
595:    char file[512] = "white.bmp";
596:    char alp[512]  = "white.bmp";
692:    char   file[512]   = "white.bmp";
693:    char   alp[512]    = "white.bmp";

engine/src/main.cpp
507:    static Texture dummy( "white.bmp", 0, MIPMAP, TEXTURE2D, TEXTURE_2D, 1 );

engine/src/gfx/background.cpp
67:    string     suffix = ".image";
68:    temp = string( file )+"_up.image";
78:        temp = string( file )+"_sphere.image";

data/modules/faction_xml_gen.py
18:fp.write('\t<Faction name="neutral" logoRGB="logos/confedPri.png" logoSec="logos/confedSec.png"/>\n')
24:    fp.write ('\t<Faction name="'+f+'" logoRGB="logos/'+f+'Pri.png" secLogoRGB="logos/'+f+'Sec.png" contraband="contraband">\n')
33:fp.write('\t<Faction name="upgrades" logoRGB="logos/confedPri.png" logoSec="logos/confedSec.png"/>\n')
34:fp.write('\t<Faction name="planets" logoRGB="logos/confedPri.png" logoSec="logos/confedSec.png"/>\n')

data/modules/spawner.py
44:            self.fac.append(VS.launchJumppoint(nam,"neutral","75 jump.png %s (ONE ONE)"%nam,"planet","sitting_duck",1,1,Vector.Add((0.,-1000.+i*200.,1000.),VS.getPlayer().Position()),"",""))

data/modules/quests/quest_explore.py
18:        self.jumppoint=VS.launchJumppoint("Base","neutral","1000 jump.png %s (ONE ONE)"%str,"planet","sitting_duck",1,1,(861946880,2132313,31337),"",self.newsys)

From the associated thread:
@BenjamenMeyer

Awesome. Will have to save this somewhere so we can act on it. Will try to take a look tonight

ermo (Rune Morling)

I think what you really want is the tool that produced the output? (so it's reproducible over time)

@P-D-E

it was just some manual inspection, trying to narrow down the tasks the tool will have to take care of (e.g. no need to handle gifs as there are none)

ermo (Rune Morling)

For context, my solution in pwcu was different: I renamed everything to .image and .texture and let the file magic dictate which loader would handle it. (IIRC) it's been a fair few years, so I don't really remember. But this is probably a fairly linux-y solution.

@P-D-E

on that note, to avoid dependencies, the tool might need magic-like detection for the relevant file types, kinda like this https://github.com/h2non/filetype.py The ones this doesn't handle are easy, dds start with DDS, bfxm start with BFXM

ermo (Rune Morling)

Not a bad idea!

[BenjamenMeyer]

Task 1: Vega Strike Engine should move the hard coded references to a config section so there is no hard-coding and/or own those files itself. This might get a little nuanced - that is, some files like the black clear, white, etc should probably be owned by the engine and get distributed as part of the engine installation. But we should still probably reference them via a configuration file so we can easily change them out instead of hard-coding them. We could even expose them to the Assets via Python if necessary. Other files might properly belong to the Assets. We need to figure out a good line to draw for what belongs to the engine and what should belong to the Assets. Hopefully now that we have three sets of assets (VS:UtCS, PWCU, and VSU:BP) we can make some intelligent decisions about that.

@royfalk @stephengtuggy thoughts on the above?

[stephengtuggy]

Yep, sounds about right.

[BenjamenMeyer]

Task 2: Yes, we should rename these. DDS format files probably should use the DDS extension and we probably need to update the engine to expect DDS for those files instead of PNG or JPG. We can certainly use file as a starter and GIMP for confirmation.

[stephengtuggy]

I seem to recall something about DDS being an ancient DirectX format that is no longer supported? @royfalk was it you who discovered that?

[BenjamenMeyer]

@stephengtuggy seems to be current - but yes, an MS DX format:
https://en.wikipedia.org/wiki/DirectDraw_Surface
https://learn.microsoft.com/en-us/windows/win32/direct3ddds/dx-graphics-dds-pguide

Makes sense they would have used it since it was introduced in 1999 right before development on the engine started.

[stephengtuggy]

@BenjamenMeyer OK, sounds good. 👍🏽

[BenjamenMeyer]

Task 3: Are there file formats that we defined?

cockpits
*.sprite, txt referring images
*.xmesh, xml referring images

If these are formats that the VS devs defined, then let's make an appropriate extension and define the format more formally. Those definitions probably belong to the VS:Engine even if they're carried out by the Assets.

@P-D-E @pyramid3d @Loki1950 do you any of you have any insight into this?

[BenjamenMeyer]

I don't really want to change the engine to use a standard format at this time; so if we defined something that later got standardized then let's document that and we can add it to a list of things TODO in a future release.

[P-D-E]

Not really, sorry.
For what I see in engine/objconv/blender_xmesh_import.py and engine/objconv/blender_xmesh_import.txt,
the VS xmesh format is meant to serve a similar purpose as this one https://docs.thinkboxsoftware.com/products/xmesh/1.10/1_Documentation/manual/xmesh_mx/xmesh_mx_saver/ref_output_files.html
but interacting with Blender instead of Maya.
VS xmesh files are well formed but technically not valid xml; they miss the xml declaration, probably to optimize file size.
They don't seem really referenced by the engine code, mostly used by mesher and objconv, so they're probably an intermediate working format.

Update: here's the xmesh specification from the wiki https://wiki.vega-strike.org/HowTo:Edit_XMESH_files
and, from https://wiki.vega-strike.org/HowTo:Add_Ships :
.XMESH - XML-Mesh format. A simple, manually editable file format designed for Vega Strike.
As for *.spr, check The ".spr" file format here: https://wiki.vega-strike.org/HowTo:Edit_HUDs

[BenjamenMeyer]

@P-D-E do you know if the engine is still using the BXFM (https://wiki.vega-strike.org/HowTo:Edit_BFXM_files) format?

We probably should pick those formats up and create RST files in the documentation of the Engine for them so we can keep the formats close to the engine and then reference those files.

[Loki1950]

BFXM files are binary versions XMESH and the target was Wings 3D not blender as it was only at version .9 at that time and not very stable yet.

[BenjamenMeyer]

Good to know. Thanks!

[P-D-E]

do you know if the engine is still using the BXFM format?

From src/cmd/drawable.cpp:

bool Drawable::DrawableInit(const char *filename, int faction,
        Flightgroup *flightgrp, const char *animationExt)
{
[...]
            string path = dir + "/" + unit_name + ".bfxm";
            if( VSFileSystem::FileExistsData( path, VSFileSystem::MeshFile ) != -1 )
            {
                    Mesh *m = Mesh::LoadMesh( path.c_str(), Vector(1,1,1), faction, flightgrp );
                    meshes->push_back( m );
    #ifdef DEBUG_MESH_ANI
                    BOOST_LOG_TRIVIAL(debug) << "Animated Mesh: " << path << " loaded - with: " << m->getVertexList()->GetNumVertices() << " vertices.";
    #endif
            }