Skip to content

Commit

Permalink
Fix table rendering on Github
Browse files Browse the repository at this point in the history 
Requires removing `--columns=80` pandoc wrap on linter.

See jgm/pandoc#3171.
  • Loading branch information
@OJFord
OJFord committed Nov 4, 2016
1 parent 9b46b1b commit 166e007
Show file tree
Hide file tree
Showing 2 changed files with 57 additions and 182 deletions.
115 changes: 20 additions & 95 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,115 +1,40 @@
Restricting the Scope for Client Error in IoT Communication
===========================================================

With billions more IoT devices predicted to be connected every year, every
problem in IoT is realised at large scale. The connection of so many devices
inevitably means vendors will iteratively improve their products; change and
improve the deployed software. This rapid iteration may easily lead to
unintended incompatibilities between client and server APIs. Additionally, many
of the devices expected to be deployed will be deeply embedded in their
operating context - light switches, concrete-borne traffic or structural sensors
in roads or buildings, et cetera - such devices have relatively long lifetimes
and may quickly be outdated, but continue attempting to communicate with a
server using that outdated interface. Client-originating communication with a
server may cause errors of great concern to a user, but be visible only to a
vendor examining a server log, who may anyway assume that the error was indeed
"user error"; perhaps that of a developer rather than a device deployed long
ago, expecting an older version of the server API.
With billions more IoT devices predicted to be connected every year, every problem in IoT is realised at large scale. The connection of so many devices inevitably means vendors will iteratively improve their products; change and improve the deployed software. This rapid iteration may easily lead to unintended incompatibilities between client and server APIs. Additionally, many of the devices expected to be deployed will be deeply embedded in their operating context - light switches, concrete-borne traffic or structural sensors in roads or buildings, et cetera - such devices have relatively long lifetimes and may quickly be outdated, but continue attempting to communicate with a server using that outdated interface. Client-originating communication with a server may cause errors of great concern to a user, but be visible only to a vendor examining a server log, who may anyway assume that the error was indeed "user error"; perhaps that of a developer rather than a device deployed long ago, expecting an older version of the server API.

Proposal
--------

When developers create an application in C, for example, utilising some
third-party libraries to offload some of the work, there's no guesswork involved
in the correct use of that library's API - the developer does not (cannot)
simply compile and ship it, hoping that every `library_fn()` invocation will
work out okay. Instead, that library's header files are checked at compile-time,
in order to ensure that such calls match the signatures of functions the library
does indeed provide.
When developers create an application in C, for example, utilising some third-party libraries to offload some of the work, there's no guesswork involved in the correct use of that library's API - the developer does not (cannot) simply compile and ship it, hoping that every `library_fn()` invocation will work out okay. Instead, that library's header files are checked at compile-time, in order to ensure that such calls match the signatures of functions the library does indeed provide.

In the context of an IoT device, while third-party libraries can of course still
be linked to the embedded application, typically an analogous 'offloading' of
work is instead conducted via a remote server. This introduces a network
protocol, and (de)serialisation to either end of the API barrier. There is
typically no checking akin to that of library header files that the usage of the
remote API matches the implementation.
In the context of an IoT device, while third-party libraries can of course still be linked to the embedded application, typically an analogous 'offloading' of work is instead conducted via a remote server. This introduces a network protocol, and (de)serialisation to either end of the API barrier. There is typically no checking akin to that of library header files that the usage of the remote API matches the implementation.

It is easy to imagine a scenario in which IoT devices such as tiny sensors set
in concrete are all but forgotten about - having been deployed years even
decades prior - yet remain a component of a production system. One need only
look to the financial sector, the age of some of its crucial infrastructure, and
the relative few who can maintain it. Should the server to which such devices
communicate be upgraded, perhaps because some other client device is assumed to
be the only user, the impact on the forgotten devices of a breaking change to
the API would likely be severe - if not a critical component, the resulting
errors could go undetected buried deep in a server log, lost in the noise.
It is easy to imagine a scenario in which IoT devices such as tiny sensors set in concrete are all but forgotten about - having been deployed years even decades prior - yet remain a component of a production system. One need only look to the financial sector, the age of some of its crucial infrastructure, and the relative few who can maintain it. Should the server to which such devices communicate be upgraded, perhaps because some other client device is assumed to be the only user, the impact on the forgotten devices of a breaking change to the API would likely be severe - if not a critical component, the resulting errors could go undetected buried deep in a server log, lost in the noise.

Similarly, it may also occur that an IoT device in production receives an update
that causes it to no longer correctly communicate with the server - or perhaps
some combination of changes to each side. In either case, the device will
subsequently be filling server logs with client error status codes - such as
`404 Not Found`, `405 Method Not Allowed`, or `412 Precondition Failed`, for
example. It's not at all unlikely that such errors would be lost amongst the
noise of similar errors resulting from experimentation, or even routinely
ignored in favour of solely monitoring application-level errors.
Similarly, it may also occur that an IoT device in production receives an update that causes it to no longer correctly communicate with the server - or perhaps some combination of changes to each side. In either case, the device will subsequently be filling server logs with client error status codes - such as `404 Not Found`, `405 Method Not Allowed`, or `412 Precondition Failed`, for example. It's not at all unlikely that such errors would be lost amongst the noise of similar errors resulting from experimentation, or even routinely ignored in favour of solely monitoring application-level errors.

The high-level goal of this project is to investigate ways of preventing such
circumstances; to provide, in a manner akin to local library header files, some
mechanism for 'compilation against an API'. Compile-time checks against some
form of 'contract' or schema for the API could mitigate the potential for errors
in use, while strict semantic versioning of the API would prevent destroying
that trust.

The problem essentially has two levels of complexity. Initially, I will look at
classes of error which can be prevented without considering state. Not in a
sense of idempotency of the request itself - an HTTP `POST` request is not
idempotent, but the correctness of the data structure sent can still be analysed
- but rather without considering a stateful *session* of multiple interactions.

While static type checking may be a useful angle on enforcing correctness of
request method and payload, an investigation into session types may be similarly
fruitful for preventing errors caused by missing preconditions, or a lack of
authorisation to make the request.

Standards such as *JSON Schema* exist for specifying the structure of an API,
and may prove a useful basis for this extension. *Ethereum* provides distributed
and undeniable contracts, which might be worth investigating as a trusted
authority for the nature of the API.
The high-level goal of this project is to investigate ways of preventing such circumstances; to provide, in a manner akin to local library header files, some mechanism for 'compilation against an API'. Compile-time checks against some form of 'contract' or schema for the API could mitigate the potential for errors in use, while strict semantic versioning of the API would prevent destroying that trust.

### Evaluation

A wholly successful implementation would be unable to compile any program that
would then fail at runtime due to a fault of the client in communicating with a
server. It is not expected that the implementation would progress that far, or
even necessarily that the investigation will discover such a thing to be
possible.

Thus, the project can instead be evaluated by considering an enumeration of the
possible ways in which communication could fail with the client at fault, and
whether the project has been able to address or prevent each possibility.

As part of the project, a system will be created for attempting to compile and
run test programs with some randomised variations designed to target areas that
could cause a client error at runtime. The project will analyse which and how
many of these successfully compiled, and the run-time behaviour. In line with
the high-level goal of the project, success would be shown as the minimisation
of error classes that compiled successfully and erred at run-time.
The project's success will be demonstrated by the inability to compile client device software that would result in an error, through fault of the client, when communicating with a server at run-time. This will be shown by analysing automated and randomised test applications for compile-time and, if compilable, run-time success.

### (Rough) Timetable

| Milestone | When |
|-------------------------------------------------------|:-------:|
| Established clear idea for 'compilation against API' | Dec '16 |
| Established clear test strategy | Dec '16 |
| Implemented server API contract | Feb '17 |
| Implemented test suite | Mar '17 |
| Implemented client API contract checking | Mar '17 |
| Established clear idea for stateful session extension | Apr '17 |
| -- (Exams) | May '17 |
| Implemented stateful sessions | Jun '17 |
| Finalise analysis/report | Jun '17 |
| Milestone | When |
|:--------------------------|:-------:|
| SemVer-enforcing Cargo | Dec '16 |
| Investigate API contract | Jan '17 |
| Implement API contract | Feb '17 |
| Randomised test suite | Mar '17 |
| Investigate session types | Apr '17 |
| -- (Exams) | May '17 |
| ? Implement session types | Jun '17 |
| Finalise analysis/report | Jun '17 |

The break for exams, and unknown possibility of implementing session types also builds in some slack with which to catch-up, solve some unanticipated problem, or allow time for an idea arising by having had a mind on other matters for a while.

### Cost & Requirements

None expected, beyond perhaps AWS use within the quota freely available to
students.
There should be no particular hardware required. Real devices may be useful for testing, but existing devices or software can be used. Testing may be easier against some external infrastructure, such as AWS, in which case some small cost may be incurred - though a certain quota for students is available free of charge.
124 changes: 37 additions & 87 deletions checker-outline.md
View file
Original file line number Diff line number Diff line change
@@ -1,84 +1,40 @@
Outline of the Client Program-Checker
=====================================

The program checker must be designed in a way so as to be runnable at
compile-time. It should inspect HTTP (and perhaps other protocol) requests for
compliance with a schema found to match the requested URL, and determine as far
as possible whether and why the request would fail at run-time. This will
involve checking of (likely constant) parameters such as the request method
(`GET`, `PATCH`, et al.) and type-checking variable data used for e.g. updating
resource attributes.

The table below enumerates the 'official' (the specification allows for custom
extension, e.g. nginx adds a couple) HTTP client errors (`4xx` status codes),
and the proposed method for determining they will occur.

--------------------------------------------------------------------------------
Code Description Detection
------ --------------------------- ---------------------------------------------
400 Bad Request No unification of request and schema data

401 Unauthorized Partially detectable by typing the session

402 Payment Required Not detectable

403 Forbidden Not detectable

404 Not Found Schema not found; URI not in schema

405 Method Not Allowed Method not in schema\[URI\]

406 Not Acceptable Accept header or return type not in schema

407 Proxy Authentication Partially detectable by typing the session
Required

408 Request Timeout Not detectable

409 Conflict Not detectable

410 Gone URI not in schema

411 Length Required Missing Content-Length required by schema

412 Precondition Failed Partially detectable by typing the session

413 Payload Too Large Partially detectable by examining constant
data

414 Request-URI Too Long Extra query parameters not accepted by schema

415 Unsupported Media Type Content-\* headers or data type not in schema

416 Requested Range Not Not detectable
Satisfiable

417 Expectation Failed Not detectable

418 I'm a teapot Umm... Not detectable

421 Misdirected Request Method & URI combination not in schema

422 Unprocessable Entity Not detectable

423 Locked Type session, acquire precondition first

424 Failed Dependency Implicitly prevented if entire call-chain
checks

426 Upgrade Required Protocol does not match that required by
schema

428 Precondition Required Type session, match schema

429 Too Many Requests Not detectable

431 Request Header Fields Too Partially detectable by examining constant
Large data

451 Unavailable For Legal Not detectable
Reasons
--------------------------------------------------------------------------------
The program checker must be designed in a way so as to be runnable at compile-time. It should inspect HTTP (and perhaps other protocol) requests for compliance with a schema found to match the requested URL, and determine as far as possible whether and why the request would fail at run-time. This will involve checking of (likely constant) parameters such as the request method (`GET`, `PATCH`, et al.) and type-checking variable data used for e.g. updating resource attributes.

The table below enumerates the 'official' (the specification allows for custom extension, e.g. nginx adds a couple) HTTP client errors (`4xx` status codes), and the proposed method for determining they will occur.

| Code | Description | Detection |
|:----:|---------------------------------|--------------------------------------------------|
| 400 | Bad Request | No unification of request and schema data |
| 401 | Unauthorized | Partially detectable by typing the session |
| 402 | Payment Required | Not detectable |
| 403 | Forbidden | Not detectable |
| 404 | Not Found | Schema not found; URI not in schema |
| 405 | Method Not Allowed | Method not in schema\[URI\] |
| 406 | Not Acceptable | Accept header or return type not in schema |
| 407 | Proxy Authentication Required | Partially detectable by typing the session |
| 408 | Request Timeout | Not detectable |
| 409 | Conflict | Not detectable |
| 410 | Gone | URI not in schema |
| 411 | Length Required | Missing Content-Length required by schema |
| 412 | Precondition Failed | Partially detectable by typing the session |
| 413 | Payload Too Large | Partially detectable by examining constant data |
| 414 | Request-URI Too Long | Extra query parameters not accepted by schema |
| 415 | Unsupported Media Type | Content-\* headers or data type not in schema |
| 416 | Requested Range Not Satisfiable | Not detectable |
| 417 | Expectation Failed | Not detectable |
| 418 | I'm a teapot | Umm... Not detectable |
| 421 | Misdirected Request | Method & URI combination not in schema |
| 422 | Unprocessable Entity | Not detectable |
| 423 | Locked | Type session, acquire precondition first |
| 424 | Failed Dependency | Implicitly prevented if entire call-chain checks |
| 426 | Upgrade Required | Protocol does not match that required by schema |
| 428 | Precondition Required | Type session, match schema |
| 429 | Too Many Requests | Not detectable |
| 431 | Request Header Fields Too Large | Partially detectable by examining constant data |
| 451 | Unavailable For Legal Reasons | Not detectable |

Example
-------
Expand Down Expand Up @@ -151,14 +107,8 @@ for key in data_schema:
ERROR - missing required data
```

where the type matching could be simplified if the schema encoded the
expectation, e.g. by using
[TJSON](https://tonyarcieri.com/introducing-tjson-a-stricter-typed-form-of-json),
and comparing against the types of data given in the language being checked.
where the type matching could be simplified if the schema encoded the expectation, e.g. by using [TJSON](https://tonyarcieri.com/introducing-tjson-a-stricter-typed-form-of-json), and comparing against the types of data given in the language being checked.

Alternatively, something like [JSON-Schema](https://json-schema.org/) could be
used, which has some traction already, and could be inspected for or extended to
include types.
Alternatively, something like [JSON-Schema](https://json-schema.org/) could be used, which has some traction already, and could be inspected for or extended to include types.

The example program above would error for each key in the data sent, since the
schema expects no data to accompany a `GET` request.
The example program above would error for each key in the data sent, since the schema expects no data to accompany a `GET` request.

0 comments on commit 166e007

Please sign in to comment.