Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first draft of feature query proposal #91

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

first draft of feature query proposal #91

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ Following are SysEx commands used for core features defined in this version of t
```
EXTENDED_ID 0x00 // A value of 0x00 indicates the next 2 bytes define the extended ID
RESERVED 0x01-0x0F // IDs 0x01 - 0x0F are reserved for user defined commands
REPORT_FEATURES 0x65 // report which optional Firmata features are supported by the firmware
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see a need in the future for having separate REPORT_CORE_FEATURES and REPORT_OPTIONAL_FEATURES where REPORT_CORE_FEATURES would not report a version for each feature since they are all covered under the Firmata protocol version. The distinction would be useful for ConfigurableFirmata where it's possible to exclude certain core features from a firmware build.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to stay away from CORE features versus EXTENDED. The notion of core and extended are propagating the "limitations" (as it sits, not hard limitations) of the Firmata implementation into the protocol. I think the protocol should be focused on solely describing the hardware and providing access to its capabilities.
If all features were part of a feature set, then this would allow you to describe any board in a way that was conducive to ConfigurableFirmata.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are hardware capabilities (GPIO, I2C, UART, SPI, etc) and then there are firmware-specific capabilities (OneWire, Stepper, Servo, Task Scheduler, etc). All of these features could be reported using a single command, or HW-specific features could be reported separately.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A user would then first query either hardware capabilities or all firmware capabilities (TBD), then if they need specifics for any board-level capability, query for the pins, etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually changed my mind while writing this...

I do like classing the features as either HARDWARE or FIRMWARE, because I don't see them as equal. I think our primary responsibility is to expose the hardware as hardware. Then, when limitations arise from the remote nature of the hardware, it becomes important to have FIRMWARE support; which I view as secondary. Theoretically, you can recreate any FIRMWARE you like if you are aware of the hardware, effectively leverage SYSEX messages, and are willing to modify the .ino (I know this is a crappy experience/solution).

So to summarize, I like HARDWARE versus FIRMWARE, instead of CORE vs EXTENDED. 😺

ANALOG_MAPPING_QUERY 0x69 // ask for mapping of analog to pin numbers
ANALOG_MAPPING_RESPONSE 0x6A // reply with mapping info
CAPABILITY_QUERY 0x6B // ask for supported modes and resolution of all pins
Expand Down Expand Up @@ -162,6 +163,67 @@ Receive Firmware Name and Version (after query)
N END_SYSEX (0xF7)
```

Report supported Firmata features
---

Report which optional Firmata features are supported by the firmware and the version of each feature.

As described in the [Feature registry documentation](https://github.com/firmata/protocol/blob/master/feature-registry.md),
Firmata features are split into 2 categories: *Core features* and *Optional features*. The feature
query reports the optional features that are supported by the firmware as well as the version
of each supported optional feature.

The feature query reduces the need to define a new pin mode for every introduced feature. Going
forward pin modes will only be defined and maintained for microcontroller specific features such as
digital and analog I/O, PWM, I2C, UART, SPI, DAC, etc. The use of pin modes for non microcontroller
Copy link
Member Author

@soundanalogous soundanalogous Jul 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still a bit torn regarding how to best support pin modes going forward. I think there are 3 options:

  1. (as proposed here, at least until Firmata 3.0), Pin modes will only correspond to features that are hardware specific. However this only partially reduces the size of the capability query response.
  2. Only have pin modes for core features that are hardware specific (e.g. Digital In/out, Pull-up, Pull-down, analog in, PWM). In this case the pins used for hardware specific optional features such as I2C, SPI, Uart, etc would need to be reported separately, most likely by querying each individual feature for its corresponding supported pins. This would scale better than option 1, but is a bigger departure for a pre-Firmata 3.0 release.
  3. Mirror Arduino pin modes: INPUT, OUTPUT, INPUT_PULLUP, INPUT_PULLDOWN (have seen this on some new hardware). This is the biggest departure from the existing functionality and would require a new solution for reporting which pins support analog input and PWM. Other hardware specific pins would be reported by a per feature query as described in option 2 above.

Currently I'm leaning towards option 2 as a good target for Firmata 3.0, but option 1 as a stop-gap.

Copy link
Member Author

@soundanalogous soundanalogous Jul 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option for handling pins modes for features like I2C, SPI, UART, etc would be to not worry about them at all as far as Firmata is concerned. This assumes a user has some familiarity with their board and which pins coincide with which features. The feature query in this case would inform the Firmata client library what the firmware supports and then the only time pins modes need to be specified are when they are technically required such as setting an input pull-up. Firmata may still need Analog In and PWM for the purpose of properly configuring pins if the mode is changed without physically restarting the board (but even then it would be unwise for a user to reconfigure a pin without unpowering the board first so perhaps that is moot as well), but generally with the feature query in place I think there will be no further need for reporting pin modes for most features.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can cover all the considerations mentioned above, if we were to take a step out from the pin level detail and first have a board level description.
Assuming you are not making a tailored solution (in which case any generic Firmata implementation becomes heavy weight and less valuable). As a rule, a person writing remote code doesn't have access to Boards.h, so they don't know anything about the hardware or the board itself. They should be able to query a board for it's feature set (i.e. GPIO, analog, SPI, I2C, etc...). Then, if they are interested in a particular feature like SPI, they could query it specifically. This would allow much greater detail to be returned about a pin or set of pins (i.e. SCLK, SS, MISO and MOSI pins).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A scenario I would like to enable, would be a teaching application that discovered a board connected to your computer and could interactively build out its capabilities and allow a person to tinker in a classroom setting with a point and click interface.
To accomplish this, we would need to associate a unique identifier with a particular board (this would allow someone to download a schematic for a particular board). Then if you could identify which board features are available through the protocol, you could enable specific UIs capable of utilizing each of the features.
If the protocol provided the ability to drill down, then we enable full introspection, providing great detail, all with minimal message sizes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be able to query a board for it's feature set (i.e. GPIO, analog, SPI, I2C, etc...). Then, if they are interested in a particular feature like SPI, they could query it specifically

I agree and like the suggestion of breaking out GPIO as well. This would allow us to fully deprecate the configuration query.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yeah I totally agree. I think we need one more query though:

  • Board feature query (what you describe as "first pass above")
  • HW feature detail ("second pass" from above)
  • Firmware feature query (non board-level features supported by the firmware such as Stepper, Servo, OneWire, etc)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would need a good strategy for revising the HAL if we provide a HARDWARE_QUERY. The current issue is the extent that the user can interact with the HW is constrained by the current firmware (how Firmata I2C, UART, Analog, PWM, etc is defined in the protocol and implemented). Does a HW_QUERY return an identifier for the current implementation (as described in the protocol for each supported HW feature) or does it simply return a description (from an enumerated list of microcontroller HW features) of the board regardless of how accurately it's features are implemented in firmware (if at all - SPI or DAC for example). In that case a FIRMWARE_QUERY would also be required to understand which interface is exposed for each feature, or maybe we get that in the second pass (feature id to identify implementation/interface, ss, mosi, etc).

Copy link
Contributor

@zfields zfields Aug 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In talking through this, I believe "board" above is synonymous with HARDWARE. Perhaps we should have two query roots, both HARDWARE and FIRMWARE.

         HARDWARE                     FIRMWARE
  __________|__________          ________|________
  |   |   |   |   |   |          |       |       |
GPIO ADC DAC I2C SPI ETC       SERVO  STEPPER ONEWIRE
            ______|________                      |
            |   |    |    |            <list of pins capable
           SS  MOSI MISO SCLK             of functionality>

Hardware functionality is physically bound to pins, while firmware can be applied to a certain subset of pins capable of supporting that functionality. The right thing seems to have separate roots (illustrated above), but similar models of interaction (so if you can understand one, then you can easily understand the other).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does a HW_QUERY return an identifier for the current implementation (as described in the protocol for each supported HW feature) or does it simply return a description (from an enumerated list of microcontroller HW features)?

My vote: Simply return a description (from an enumerated list of microcontroller HW features)

Copy link
Contributor

@zfields zfields Aug 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current issue is the extent that the user can interact with the HW is constrained by the current firmware

I think the firmware needs to be revisited... I would like to rearchitect the firmware to take on concepts of configurable Firmata (composable), as well as allowing it to operate as a library. This would make it a lot easier to add Firmata to existing sketches, instead of starting with Firmata and building a sketch into it.

specific features such as Servo, Stepper, OneWire and Encoder will be phased out over time. Reducing
the number of unnecessary pin modes also reduces the size of the Capability Query Response which has
grown to exceed 500 bytes for boards with a large number of pins.

Another function of the feature query is to move establish a versioning scheme for optional features.
The query will return a major and minor version for each optional feature supported by the firmware.
For features that existed prior to protocol version 2.6, the feature version begins at v1.0. New
features should begin at v0.1. The bugfix version should also be tracked for each optional feature
protocol revision (each optional feature has its own markdown file). All core features on the other
hand are not individually versioned, but are covered under the protocol version (at the top of this
file).

### Query supported features
```
0 START_SYSEX (0xF0)
1 REPORT_FEATURES (0x65)
2 REPORT_FEATURES_QUERY (0x00)
3 END_SYSEX (0xF7)
```

### Supported features query response

The response returns the Feature ID and the major and minor version for each optional feature
supported by the the firmware. A Firmata [Feature ID](https://github.com/firmata/protocol/blob/master/feature-registry.md#firmata-sysex-feature-registry)
is either 1 byte ID or a 3 byte Extended ID. If the first byte is zero, this indicates an Extended
ID so the parsing logic should evaluate the value of the first byte and if it is zero, expect two
additional ID bytes followed by a major version byte and a minor version byte. If the first byte
is > 0 then the following 2 bytes are the major and minor version numbers for that feature.

```
0 START_SYSEX (0xF0)
1 REPORT_FEATURES (0x65)
2 REPORT_FEATURES_RESPONSE (0x01)
3 1st FEATURE_ID (1-127, eg: Serial = 0x60, Stepper = 0x62)
4 1st FEATURE_MAJOR_VERSION (0-127)
5 1st FEATURE_MINOR_VERSION (0-127)
6 2nd FEATURE_ID (1-127, eg: Serial = 0x60, Stepper = 0x62)
7 2nd FEATURE_MAJOR_VERSION (0-127)
8 2nd FEATURE_MINOR_VERSION (0-127)
9 3rd FEATURE_ID (0x00, Extended ID)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently there are no Extended IDs defined, but even when they start to be added I think it will be rare that a firmware will include more than 1 or 2 at a time so in most cases there will only be 3 bytes used for each feature (ID, major, minor).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we were to use a drill-down approach in Firmata v3.0. Then instead of using a bit mask, we can use an integer which would provide significant room for feature growth, without the need for branching logic in the protocol.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intent here was to provide backwards compatibility in the same way the midi protocol handled it when it became evident 127 identifiers was not sufficient. There is no use of a bit mask in my proposal, but there would be the need to check if the first byte for each feature ID is 0x00 or not.

Are you suggesting using something like a 14-bit identifier for all Firmata features beginning in Firmata 3.0? That would definitely scale much better.

Copy link
Contributor

@zfields zfields Aug 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

14-bit identifier for all Firmata features beginning in Firmata 3.0

Yes, I would prefer it to be 14-bits, instead of a check byte. That being said, we are at 11 and 127 seems like a long way off.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, no I'm trying to deprecate most of those modes (and the capability query as well) and instead provide a separate FIRMWARE_QUERY and HARDWARE_QUERY as we've been discussing. Then make pinMode more like Arduino pin mode where you use it to set DIGITAL_IN, DIGITAL_OUT, and various PULL_UP states.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interface version could be in the follow on query

I think it should ONLY be in the follow on query. For example, if I'm not going to use I2C, then I don't want to waste bytes (time and possibly money) to learn about its version...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the advantage of separating out

2 HARDWARE_FEATURE_ID (from enumerated list of MCU features)
3 HARDWARE_FEATURE_INTERFACE (the sysex command corresponding to implementation)

They would both need to be unique, correct? Why can they not be the same value?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They would both need to be unique, correct? Why can they not be the same value?

It depends on if we allow multiple implementations for individual HW features. For example the current version of I2C only supports a single port and does not scale. So in that case there is a MCU feature, I2C, but a specific Firmata I2C protocol implementation that is highly constrained.

Copy link
Member Author

@soundanalogous soundanalogous Aug 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the version may be sufficient in that case. It sorta depends on if the HW query lands in Firmata v2 or Firmata v3. I would prefer to at least add the FIRMWARE query sooner than later to provide a way for client applications to understand the firmware capabilities when using ConfigurableFirmata without needing to keep creating pin modes to accomplish the task.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also means we will have to own and enforce the Firmata HAL. There should be a single definition for I2C, SPI, DAC, UART, etc. For those features that are already implemented (I2C and UART for example, but even overhauling analog and GPIO as well), they need a new definition that is more scalable. We can't simply copy the Arduino HAL in all cases since it does not always work over a transport but we can try to align where it makes sense.

10 3rd FEATURE_ID (lsb)
11 3rd FEATURE_ID (msb)
12 3rd FEATURE_MAJOR_VERSION (0-127)
13 3rd FEATURE_MINOR_VERSION (0-127)
...for all supported features
n END_SYSEX (0xF7)
```

Extended Analog
---

Expand Down