emonESP dev. :: SPIFFS with SD card

[danbates2]

@jeremypoulter
Hi Jeremy, I'm modifying a version of emonESP for my emonDC board. I was hoping for your input on a decision I have to make.

I'm logging data to an SD card, using the ESP8266.

The firmware has class conflicts between the SPIFFS filesystem and the SDCard filesystem libraries. I've tried changing all the class definitions in the .h and .cpp file of the SD card lib to no avail.

So I have three possible paths.

1. Move all the SPIFFS files to the SDcard and run emonESP from an SD card.

• The main disadvantage is requiring an SD card to run the basic program, a limitation I don't want.

2. Get SD card and SPIFFS working alongside each other. The best resource I've found to change the emonESP firmware is this: Make SD Class be able to coexist with SPIFFS esp8266/Arduino#1524 (comment)

• A brief go at this method I couldn't make work because the function names differed I think.. I'll have another go at getting this working..

3. Write to a datalogging file in the SPIFFS and provide a way to download the file in emonESP.

• I imagine memory space to be the limitation here.

[jeremypoulter]

Actually there is a 4th option, on the OpenEVSE WiFi firmware we no longer use SPIFFS to reduce memory usage. All the HTML/JS/CSS files are compiled into the firmware so you would be able to use the SD lib to problem.

The two OpenEVSE WiFi firmware and EmonESP are similar code bases so should be possible to port the changes, that is certainly on the list of things to do. It is however a rather complex change I am afraid.

One thing that may help is this: https://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data, but that may only be for the ESP32.

PS really sorry I have not had a chance to look at the emonDC 😢

[danbates2]

Oh yeah I gave you one ages ago. Keep it? Bin it? Not sure what you could do with it really. Old version with more than one design issue! You're also welcome to send it back.
That 4th option makes sense, coincidentally it's something I mentioned to Trystan today. I first noticed it was possible to embed all the website stuff in the first flash, hard-code the images etc., in the Luftdaten.info project esp8266 firmware.

I think this 4th option will have to go on the list..

[danbates2]

Today the task is to make another menu box item in the emonESP webpage with a toggle switch to activate another mode.. "sampling with the external ADC and doing some maths", and sending through the values by the emonESP backend to appear at "latest data".
SD card afterwards..

[danbates2]

Right. I've got a tidy looking thing with a checkbox to enable the ADC, with a variable being changed in the javascript. Next I need to understand the simplest method of 'javascript variable - - > esp8266 variable', to toggle the sampling loop on or off..

…

On Mon, 29 Oct 2018, 21:02 Jeremy Poulter, ***@***.***> wrote: Actually there is a 4th option, on the OpenEVSE WiFi firmware <https://github.com/OpenEVSE/ESP8266_WiFi_v2.x> we no longer use SPIFFS to reduce memory usage. All the HTML/JS/CSS files are compiled into the firmware so you would be able to use the SD lib to problem. The two OpenEVSE WiFi firmware and EmonESP are similar code bases so should be possible to port the changes, that is certainly on the list of things to do. It is however a rather complex change I am afraid. One thing that may help is this: https://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data, but that may only be for the ESP32. PS really sorry I have not had a chance to look at the emonDC 😢 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAe3nnphqFmCL4BbZdxxjUoIcsl_740ks5up2ztgaJpZM4X_oHe> .

[danbates2]

Heh, just spent 4 hours doing something possibly superfluous that I didn't get working anyway. The other obvious option was to have the ADC on by default, and bring a JSON string into the relevant part directly, somewhere around the decoding of the serial input.

…

On Mon, 29 Oct 2018, 21:02 Jeremy Poulter, ***@***.***> wrote: Actually there is a 4th option, on the OpenEVSE WiFi firmware <https://github.com/OpenEVSE/ESP8266_WiFi_v2.x> we no longer use SPIFFS to reduce memory usage. All the HTML/JS/CSS files are compiled into the firmware so you would be able to use the SD lib to problem. The two OpenEVSE WiFi firmware and EmonESP are similar code bases so should be possible to port the changes, that is certainly on the list of things to do. It is however a rather complex change I am afraid. One thing that may help is this: https://docs.platformio.org/en/latest/platforms/espressif32.html#embedding-binary-data, but that may only be for the ESP32. PS really sorry I have not had a chance to look at the emonDC 😢 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAe3nnphqFmCL4BbZdxxjUoIcsl_740ks5up2ztgaJpZM4X_oHe> .

[jeremypoulter]

FYI I spoke with @glynhudson and I will be working on pulling some of the OpenEVSE changes into EmonESP, including the changes to embed the HTML/JS in the firmware, probably will be over the next couple of weeks.

Re your attempts to enabling the ADC directly may be an idea to raise a separate issue with a bit more detail, or if you already have some code create a PR so we can see the code.

[danbates2]

Yes, it's a separate issue thing. Spoke with Glyn also, sounds good. I got it working. Made a JSON sting and used a single line: input_string = my_keyvalue_string; Done for now. When I started this I though how I might document this to enable others to interface the esp easily with other things. This input_string is definitely a good find. I want to make a web interface which gives options like defining the posting interval and calibration constants of the ADC. Maybe this is food for thought with your library idea.

…

On Tue, 30 Oct 2018, 22:58 Jeremy Poulter ***@***.*** wrote: FYI I spoke with @glynhudson <https://github.com/glynhudson> and I will be working on pulling some of the OpenEVSE changes into EmonESP, including the changes to embed the HTML/JS in the firmware, probably will be over the next couple of weeks. Re your attempts to enabling the ADC directly may be an idea to raise a separate issue with a bit more detail, or if you already have some code create a PR so we can see the — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFAe3ilsiw6HkObLy9DPH50G9GyLLxv-ks5uqNmVgaJpZM4X_oHe> .

[danbates2]

Ah. It could be called a preprocessor module. EmonESP gets values. Preprocessor keeps hold of them and applies user defined stuff. Then send to emoncms.

…

On Tue, 30 Oct 2018, 23:04 Daniel Bates ***@***.*** wrote: Yes, it's a separate issue thing. Spoke with Glyn also, sounds good. I got it working. Made a JSON sting and used a single line: input_string = my_keyvalue_string; Done for now. When I started this I though how I might document this to enable others to interface the esp easily with other things. This input_string is definitely a good find. I want to make a web interface which gives options like defining the posting interval and calibration constants of the ADC. Maybe this is food for thought with your library idea. On Tue, 30 Oct 2018, 22:58 Jeremy Poulter ***@***.*** wrote: > FYI I spoke with @glynhudson <https://github.com/glynhudson> and I will > be working on pulling some of the OpenEVSE changes into EmonESP, including > the changes to embed the HTML/JS in the firmware, probably will be over the > next couple of weeks. > > Re your attempts to enabling the ADC directly may be an idea to raise a > separate issue with a bit more detail, or if you already have some code > create a PR so we can see the > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#49 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFAe3ilsiw6HkObLy9DPH50G9GyLLxv-ks5uqNmVgaJpZM4X_oHe> > . >

[danbates2]

Anyone could plug in with something like input_string and make finer changes on the preprocessor settings off a smartphone.

…

On Tue, 30 Oct 2018, 23:04 Daniel Bates ***@***.*** wrote: Yes, it's a separate issue thing. Spoke with Glyn also, sounds good. I got it working. Made a JSON sting and used a single line: input_string = my_keyvalue_string; Done for now. When I started this I though how I might document this to enable others to interface the esp easily with other things. This input_string is definitely a good find. I want to make a web interface which gives options like defining the posting interval and calibration constants of the ADC. Maybe this is food for thought with your library idea. On Tue, 30 Oct 2018, 22:58 Jeremy Poulter ***@***.*** wrote: > FYI I spoke with @glynhudson <https://github.com/glynhudson> and I will > be working on pulling some of the OpenEVSE changes into EmonESP, including > the changes to embed the HTML/JS in the firmware, probably will be over the > next couple of weeks. > > Re your attempts to enabling the ADC directly may be an idea to raise a > separate issue with a bit more detail, or if you already have some code > create a PR so we can see the > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#49 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFAe3ilsiw6HkObLy9DPH50G9GyLLxv-ks5uqNmVgaJpZM4X_oHe> > . >

[danbates2]

I'll close the issue. The SD card issue won't be a problem with moving everything to the main 4MB memory.
It's in my interest to improve EmonESP as a foundation to my DC project. So we could coordinate to bring the code together if you like.
my email address is dgbates2 .. At.. Gmail.. Dot.. Com.

[jeremypoulter]

Appreciate it. I will be trying to break out some of the code into reusable libraries that can be shared between projects.

…

On Sat, 3 Nov 2018, 15:09 Daniel Bates ***@***.*** wrote: I'll close the issue. The SD card issue won't be a problem with moving everything to the main 4MB memory. It's in my interest to improve EmonESP as a foundation to my DC project. So we could coordinate to bring the code together if you like. my email address is dgbates2 .. At.. Gmail.. Dot.. Com. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF1WgIzP8KOQaKhnwShwuQP9ko6pDKxXks5urbGjgaJpZM4X_oHe> .

[danbates2]

I'd probably be able to help but I haven't made a library before so I'd probably be a bit slow.
I get the concept of creating self-contained programs using classes and private initialisations, that's about it.
Maybe my help goes as far as janitorial duties :) Like, I changed config.js to point to emoncms.org by default.

function ConfigViewModel() {
  BaseViewModel.call(this, {
    "ssid": "",
    "pass": "",
    "emoncms_server": "emoncms.org",
    "emoncms_path": "",
    "emoncms_apikey": "",
    "emoncms_node": "",
    "emoncms_fingerprint": "",
    "mqtt_server": "",
    "mqtt_topic": "",
    "mqtt_feed_prefix": "",
    "mqtt_user": "",
    "mqtt_pass": "",
    "www_username": "",
    "www_password": "",
    "espflash": "",
    "version": "0.0.0"
  }, baseEndpoint + '/config');