Front-end state management

[hasan-sh]

Dear Hedy team, contributors and enthusiasts,

I hope this message finds you well. As we continue to roll out exciting features for Hedy, it's crucial to address certain limitations in our front-end that have surfaced since October. These limitations pose potential challenges and may impact the user experience negatively.

Front-End State Management: The Core Issue

In front-end state management, the user's app or browser traditionally maintains the program's state. This includes enabling or disabling certain buttons and UI features, ensuring a seamless interaction between the user and the application throughout the session.

A perfect example that illustrates the challenge that we face lies within the customize-class view. Specifically, reordering or adding/removing adventures to levels. Watch the following screencast:

Screencast.from.29-11-23.11.01.02.webm

In the previous screencast, we see the correct, expected behavior of the user considering the following, simple and sound user story for this view:

1. The teacher can customize a class
2. The customize class provides different options:
   1. ...
   2. The teacher can add/remove/reorder an adventure within different levels
3. The teacher saves their changes by clicking on the Save button.

However, this is not what we actually do, see the following:

Screencast.from.29-11-23.11.01.18.webm

You can see that for item (2.2) in our consturcted user story, the user doesn't have to perform (3). In other words, when adding/removing/reordering adventures, they don't really have to click on save.

This discrepancy arises from a lack of state management in the front-end.

Research Findings and Shortcomings

• Jinja2 provides list methods; e.g., remove an item or append a new one.
  • However, these changes are cumbersome and do not cause a re-render (which is important)
• Hyperscript with Htmx offers a solution but introduces the issue demonstrated earlier.

Exploring new Avenues

There are different solutions to this problem. After discussing with @jtwaleson and @jpelay and agreeing that this is an existing problem, different ideas emerged. Here's a list of the main ones:

1. Industry standard framework: such as React, Vue.js, or Angular (ordered by favorite!)
2. Global Store: implement a global store with persistence and the ability to issue and resolve actions and events.

Regarding the incorporation of industry-standard frameworks (1), it is evident that a robust structure needs to be meticulously constructed to integrate the chosen framework. The positive aspect here is that this transition may not necessitate radical changes, despite potential adjustments in navigation strategies.

Conversely, (2) presents a variety of possibilities. One option is to construct our own store with the specified capabilities. Alternatively, we could leverage existing solutions like RTK, Mobx, RxJS, or Recoil. Unfortunately, my knowledge in this domain does not extend to Flask (libraries). Notably, Flask provides the appContext (i.e., g ), which functions as a global namespace. In the documentation:

This is a good place to store resources during a request. For example, a before_request function could load a user object from a session id, then set g.user to be used in the view function.

If we were able and it's a good practice to use this app context as a store that maintains the state of the application, we can still move forward in the Pythonic way!

Plans and Call to Action
In navigating this decision, we must weigh the advantages and drawbacks of building our own solution versus adopting an existing one, considering factors such as ease of integration, maintenance, and alignment with our specific needs. A collaborative exploration of these options will be instrumental. Therefore, I encourage each developer to choose an area of interest related to this problem. Research and/or provide insights and potential solutions. While we don't have to make a decision immediately, having a well-informed plan is essential!

[rix0rrr]

This is a good discussion to have! We have to make sure that we have the discussion on the right principles though.

A bit of history

In the ancient past, we wrote things in unprincipled JavaScript because we wanted responsivity, and we didn't think there would be a lot of JavaScript. Of course, then things grew, and things grew, and you know how it goes :).

In that situation, looking at the problem holistically, we had sort of a 2-tier system: the sever-side code was written in Python, and Python is a core language of Hedy. Python code was (somewhat) easy to test, would automatically be reloaded by just hitting F5, etc. Then there was client-side code; originally JavaScript, later changed to TypeScript to help with programming errors. The TypeScript code was compiled and bundled, so needed a build step to see the updates, and it was harder to test. There also wasn't equal familiarity between JavaScript and Python.

That's when we said: why don't we move more functionality to Python, and with that came the introduction of HTMX. HTMX is a re-introduction of the "simple old ways" of building web applications, that were built on form submissions and page refreshes... but without the form submissions and the full page refreshes. Read the author's essays on why in many cases you should probably prefer hypermedia applications over client-side JavaScript frameworks here.

With the use of HTMX, we can now (mostly) standardize on one language: Python, with only a bit of work occasionally needing to be done in JavaScript. I would argue there is a value to this.

I also understand that the use of a library like HTMX, which prefers server-side development to client-side development is not extremely common. HTMX does it, and Hotwire (built by DHH). On the other hand, client-side JavaScript libraries to build SPAs are extremely common, and nearly everyone loves building client-side SPAs. Many recent web developers know more about React and its legion of plugins and build steps, than about HTTP itself.

Arguments for JavaScript

A little about the arguments fielded above in favor of JavaScript frameworks, revolving around the use of a Save button:

In other words, when adding/removing/reordering adventures, they don't really have to click on save.

This discrepancy arises from a lack of state management in the front-end.

I think this is making two points:

• An explicit Save button is a good thing to have
• The only way to have temporary state that is not the "official" state is by keeping it in JavaScript, and only sending it to the server when the user hits the Save button

I'm not sure that I agree with these arguments.

• Re: a "save" button is a good thing to have. I would like to ask the counter-question: well is it though? I think the prevailing paradigm these days for most applications, web and otherwise, is that you make changes and they just get applied straight away. Look at GitHub settings. Look at NPM organization owners. Look at Google Docs. I'm sure there are many other examples, which elude me at the moment. You don't "save" your changes: you just make them, and they are persisted automatically. Bonus points if you can supply an "undo" feature. I think the "save button" is mostly a UI paradigm from the past these days, one that us developers default to because our text editors still work that way.
• Re: the only way to have temporary state is in the client. Framed like this, you can see that it could equally well be implemented on the server. The server could have two states for each class. You would manipulate the non-visible state in the editor, and clicking the save button would promote the draft state to be the published state. Yes, it would be a bit of added work to do that. But that's the feature, the feature is to be able to edit unpublished state.

You could make the argument that you would get temporary state for free if you had a copy of the state in the client that you would only sync back on demand, but then I would point back to the first bullet: is that actually desirable?

The core of the discussion

I think the core of the discussion comes down to this:

• Do we think there's more value in keeping all development in Python, keeping development simpler overall but trading off we would be using technology that most professional web developers would not be automatically familiar with?
• Or do we think there's more value in going the JavaScript SPA route, making development more complicated overall , but that might be acceptable because there are many developers out there that are accustomed to the added complexity, and in fact even expect it.

It's a tough call. I know my preference, but I'm not a stakeholder in this anymore, just an advisor.

[Felienne]

Thanks for the discussion @hasan-sh and the historical perspective @rix0rrr!

I want to add two points (in separate messages to make conversation easier)

(Auto) Saving
I think I agree with @rix0rrr that an explicit save is not desired. I don't know why we went that way initially (because the sate of the customization page was a lot smaller then, so it probably made more sense) but people nowadays expect that things they do on a website are saved immediately (like Google Docs) and this is line we have been following recently. We never made this explicit but new features (like the reordering of tabs) show this "auto save" behaviour, while old parts use the button.

Thus, I think that the statement of @hasan-sh above is not really true:

This discrepancy arises from a lack of state management in the front-end.

This discrepancy arises from a lack of project management (on my part) really. When we introduced the "auto-save" with the reordering, we should probably have followed through for the rest of the page. And we never had the clear user stories as described above where "user clicks save" is an explicit point.

I propose opening an issue to use autosave in most places (maybe even for a new adventure, like a new file in Docs is also saved immediately) so that the front-end behaviour for users is more systematic.

Now, I do not know a lot about front-end development, but from this discussion I gather that going towards the auto-save would improve the situation we are in, as I understand the save button behaviour causes the need to save state and thus the issues.

So concrete question:

• If we move everything to auto-save, will the situation improve?

[Felienne]

Point two

Front-end changes
I am not against using a front-end framework per se, but a problem that is entirely unrelated to technology is that ultimately I will have to be able to understand what is going on, because I am the only constant factor in the project as it stands. If something breaks after you all have left, I wil still be on the hook to fix it, and I can understand Python and HTMX well.

Hence I really only want to introduce a front-end framework if this overhaul has a clear owner, that is: a long-term commitment to the project as a maintainer (meaning: a volunteer position at this point in time, because we don't have support for long-term contracts)

Because introducing new tech I don't understand means the project that I run can break in ways that I cannot easily fix, and I don't want be in that project (and I don't want to ask @rix0rrr all the time, who is of course capable of fixing everything, because it will ruin my marriage :D)

I hope that makes sense from a project perspective, again nothing to do with technical reasons apart from me not knowing these new frameworks.

We can totally discuss this in the next meeting as well: do we have people wanting to make this change because it really matters to them/to the project enough to commit to "owning the front-end", and how will that look like?

(but let's discuss the impact of the saving feature first!)

[jpelay]

Hi Hasan! I like that we are having this discussion, and since I was the person responsible of adapting the customization page to HTMX, I'll offer some insights!

The plan for that page was (is!) to convert it to all to use HTMX and managing the state of the page on Python directly through database calls. You are right that this page uses two systems: the ordering of the adventures is managed with HTMX while all of the other configurations are gathered using the old way of saving the customizations, by grabbing HTML elements trough JQuery.

Therefore, as Felienne have already said, the preferred solution would be to not have a Save button! We want to have a system where each change is registered automatically without the user having to click on a Save button. This was actually something we discussed when implementing this feature. So, mostly, it was an oversight from my part not coming back to this page and adapting it fully to HTMX.

With this we can see that we already have a way to manage state: Python with Dynamo, using HTMX to handle interaction. However I do think we need to be careful with this: if we use HTMX carelessly we incur in runtime problems, like on the Live Statistic page, which is painfully slow and I'm already working on improving it. A good, and simple example of handling the state with Python can be found in the Grid overview page, where we use a table to track the student submissions of adventures, the page is reasonably fast and no JavaScript code is required at all.

I think this problem is actually similar to our design inconsistency issue, where we sometimes have several ways of doing the same thing and we end up deploying all of them. So maybe the solution is less of a technical one and more directed towards project management, as Felienne suggests, with more of a clear way of doing things.

As you can see from my wording, you probably already know that I'm not keen towards replacing all of our code towards a front-end framework: the change will probably be painful and slow, we'll raise the bar for new contributors, and we need someone that owns the change and is compromised with replacing everything, otherwise we'll just add a new method of doing things and we'll end up having JQuery, HTMX and React or Vue.

So perhaps our focus right should be on improving the usability of our page, roll new features and polish existing ones using without the need to replace a large part of our code base.

But improving our code would also mean improving the usability!! Maybe we need to improve our front-end code (and define the scope of what that would mean) using the tools already at our disposal: TypeScript, Python and HTMX, and add new ones that can improve the code without needing such a large overhaul.

Maybe we can discuss in the next contributor meeting, and dedicate the entire meeting to it? Or perhaps we can do one exclusively for this?