Local Web User Interface

[isabelle-dr]

Feature request

An interface to improve the accessibility of this tool. The objective of this first iteration is to a simple Local Web UI.

Is your feature request related to a problem? Please describe.
Currently, the instructions to run the validator are long and need a certain technical literacy: download JAR file, open the terminal, navigate to the accurate directory, run a command line with selected parameters, open the report in a JSON viewer.
This is a pain point and even a blocker, for many organizations.

Proposed solution
Recover the work started by @lionel-nj in #421, and build an interface that will act as a Minimum Viable Product that we can build on.
We chose to use Spring Boot because it is community supported, scalable, industry-standard, easy to staff.

Describe alternatives you've considered
Better documentation with images so that users that never used the command line before can use this tool.

Additional context
MobilityData had planned to work a local Web UI in Q1 2022 by recovering the work started in #421.
A Web UI is included in contribution #1088. After reviewing and discussing with @lisaattrillium and @ed-g, MobilityData will continue with the initial plan of building a local Web UI on top of the work started in #421, and we will discuss integrating the amazing work done on the HTML output in this validator.

Acceptance Criteria

• As a user, I can open the local web UI with clear instructions.
• As a user, I can upload my GTFS feed and run the validator by clicking on a button.
• As a user, I can see the results on the Interface in a JSON Viewer.

Not doing

• Providing a hosted Interface. This will come later on.
• Work on the validation results. This will be part of another initiative.

cc @asvechnikov2

[barbeau]

@isabelle-dr Just to clarify what this issue is focusing on - from my perspective there are 3 possible milestones (in increasing level of implementation complexity) for improving the validation process:

1. Add HTML file output to existing CLI app - this is done in PR feat: html output, web front end #1088.
2. Add integrated web server that a user could run on their own computer to upload GTFS data and get a report in the browser - Partial work is done on this in PRs Local web UI #421 and feat: html output, web front end #1088.
3. Same as 2, but run as a hosted service somewhere public on the Internet.

Is this issue for 2 above? And #1088 would be split to only include 1 above?

[isabelle-dr]

Is this issue for 2 above?

Yes, it is.

And #1088 would be split to only include 1 above?

Yes, that's the plan!

[asvechnikov2]

Hi @isabelle-dr, thanks for sharing! What's the main problem that we're trying to solve? Is it easy browsing of validation results once validation is done or easy run of the whole tool? If it's the latter then probably running an integrated web server locally won't be any easier than the current workflow (which is "maybe install java", "download JAR", "run command", "see JSON"). If we want to simplify casual usage of the validator so that people can quickly get familiar with it, I can think of a couple of ways

• Create a simple public website that would generate a list of commands to run, depending on different parameters (e.g. OS, GTFS file path, etc)
• Create non-trivial website that will run the specified JAR via Java applet or any other way (I'll need more time for investigation to provide any information here, also, this might be hard to do in general and I have some security concerns)
• Create or extend an existing MobilityData website with a functionality to run the validator and see results. In this scenario MobilityData will run the validator on their servers and produce results (This requires understanding how to make sure server resources aren't over consumed) (Actually with this approach your usage of GTFS Validator will be similar to a GTFS Consumer company, which will be really helpful to proactively find and resolve issues)

Please let me know if I misunderstood the problem

[barbeau]

I agree with @asvechnikov2. ☝️

Using my original enumeration in #1112 (comment), I see 2) being an order of magnitude increase in implementation complexity over 1) with a negligible return on making the project easier to use for non-technical users (maybe you remove some command-line parameters, but that's about it). Also, there is only an incremental increase in implementation complexity between 2) and 3), with a much larger return on ease of use for non-technical users.

I see two primary blockers for non-technical users to run the validator locally:

• A. Security/permission constraints on their local computer for installing Java and executing the validator (e.g., if it's a computer managed by organization IT)
• B. Technical expertise and friction required to install Java, type in the correct command-line parameters, etc.

The A) "security issues" can only be solved by 3) hosted service.

One simpler way to reduce the B) "technical expertise" blocker is to convert the JAR file into a native executable (e.g., Windows .exe file) using something like http://launch4j.sourceforge.net/ or jlink. This would remove the requirement for downloading and installing Java.

The original Python validator had a Windows standalone version as a .exe:
https://github.com/google/transitfeed/wiki/FeedValidator#windows-standalone-version

This was widely used, as evidenced by multiple issues (1, 2, 3) of people looking for a Windows .exe for the last release of the tool (which for some reason didn't have a .exe attached).

With this .exe, you could also drag and drop the GTFS zip file onto the command-line window, which removed the need for command-line parameters. I imagine we could do the same in gtfs-validator.

I've never packaged a JAR as an EXE myself, but I imagine the downside is an increased file size of the validator, since it would need to bundle all the Java libraries used (vs those being installed by the user separately). But that probably still takes up far less space on a user's computer than installing the JRE.

[ed-g]

This requires understanding how to make sure server resources aren't over consumed

Just thought I'd mention that there's code in the current Spark server that does sanity checking of the zip inputs to make sure that they don't have embedded folders or .., or consist of terabytes of zeroes and fill up the server filesystem. I'll bet that it can be ported to any Spring boot web server with only minor changes.

There are some sanity-checks that might be too strict and reject valid gtfs files, we've only had a chance to test in on Linux and Mac with firefox and chrome, other browsers or platforms might upload the file a little differently.

https://github.com/Trillium-Solutions/gtfs-validator/blob/html_output/main/src/main/java/org/mobilitydata/gtfsvalidator/viewer/ValidatorWebServer.java#L187

[bdferris-v2]

I know this is getting away from the original discussion point for this issue, but I wanted to follow up on the points raised by @barbeau above. Specifically, I want to endorse the idea of packing up a Windows installer for the validator. I maintained the Windows exe for the original Python validator and I can confirm it was used quite a bit. I think a packaged installer for the validator, combined with a simple HTML visualization of the validation report, could go a long way in helping more people run the validator. More importantly, I think it could be done with less engineering resources than a hosted solution.

I've done some initial investigation work into the feasibility of this approach, documented in https://bit.ly/gtfs-validator-packaged-exe

Critically, I think this approach can achieve the following:

1. It can provide a packaged installer.
2. It includes a stripped-down JRE that means the user doesn't have to install Java directly.
3. that can be installed and run locally without administrative privileges.
4. It can support simple drag-and-drop operation.

So tl;dr I think it's a good idea :) Enough so that I'm willing to put together a PR for this if there is interest. I acknowledge this particular free cat comes with some upkeep costs, so definitely something to discuss.

[isabelle-dr]

Closing, we decided to implement another solution, described in #1124