Update 28.3.2014: https://www.cs.helsinki.fi/u/ljlukkar/wot
Lasse Lukkari
University of Helsinki
lasse.lukkari@gmail.com

Popularity of JavaScript Object Notation has resulted in a wide application area and strong tool support. By developing JSON REST interfaces for microcontrollers enabled with internet connectivity, these tools and techniques can be adopted as well.

Categories and Subject Descriptors

C.0 [General]: Hardware/software interfaces
C.2.2 [Network Protocols]: Applications
H.5.2 [User Interfaces]: Graphical user interfaces (GUI)

General Terms

Documentation, Design, Experimentation

Keywords

Internet of Things, REST, JSON, Microcontrollers, User interfaces, JavaScript.

The first big wave of HTTP service interfaces came around ten years ago. The biggest buzzword back then seemed to be SOAP which stands for Simple Object Access Protocol. SOAP is a XML + XML Schema based messaging protocol. SOAP is not tied to any particular transfer protocol but it's most commonly used over HTTP. These XML based messaging approaches have drawbacks. XML is highly structured and therefore not easy to produce on simple systems and even less easy to read. Also quite often the situation is that XML messages carrie a lot of unnecessary information and the notation itself is not well optimized for performance and small message size.

But as things moved on along came AJAX. AJAX stands for asynchronous JavaScript and XML and is a group of interrelated web development techniques used to create web applications that work asynchronously on the client's web browser. While the XML-based solution are more suited for RPC type calls an alternative solution for the data transfer format emerged where less structured and typed messaging is more appropriate. JSON, the JavaScript object notation became widely popular.

While the XML formatted data, if not preformatted HTML, needs quite a lot of parsing and processing by the developer before finding its way to the clients browser and display, JSON became natively supported by most modern web browsers and became the de facto weapon of choice for messaging in asynchronous web applications. This tool set is widely used in the modern web applications already which make it well supported by many accompanying tools such as database engines and user interface frameworks for storing and displaying the information using same JSON format.

The objective of this paper is to show how these techniques, tools and protocols used in modern web applications can be applied to a lot less powerful microcontroller systems to create interfaces that are highly adaptable. The same interface can be used to create rich modern HTML5 graphical user interfaces for accessing and manipulating data by the end user, as well as for an example a sensor networks that provide the data in a format that is ready to be consumed by the document-oriented database systems such as NoSQL databases. Other machine to machine conversations are also possible.

Even though the programming environment and performance of these small microcontroller devices cannot be compared to modern web servers handling hundreds of HTTP requests per second the underlying principles for saving bandwidth, processing time and memory are the same and easy to adopt and implement.

To build a JSON REST web interface we need to be able to manipulate collections of JSON objects residing on server side using HTTP requests. The device used as a server here is a small microcontroller system with limited performance.

2.1 JSON

JSON is a short for JavaScript Object Notation and described in RFC 4627[1]. JSON is intended to be a human readable lightweight text based open standard notation for data interchange and is originally a subset of JavaScript programming language defined in the ECMA-262 Standard [2]. JSON is often considered to be a fat free alternative to XML based data transferring. Even though JSONs origins are in the JavaScript it can be considered language independent and the conventions are familiar to the C-based family of programming languages.

Syntax of JSON is straightforward and simple. JSON has two data structures: objects and arrays. Objects are presented as name value pairs between { (left brace) in the beginning and } (right brace) in the end. Arrays are ordered collection of values or objects separated with commas. Array´s items are grouped between [ (left bracket) in the beginning and ] (right bracket) in the end.



Figure 1:: JSON structure images borrowed from json.org

Values defined in JSON can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.

A simple example for JSON formatted message containing a UNIX timestamp and a single temperature reading could be the following:

{
   "time":1349011120,
   "temp":23.7
}

And an array containing 3 of these measurements would be:

{
   "measurements":[
      {
         "time":1349011120,
         "temp":23.7
      },
      {
         "time":1349011180,
         "temp":23.3
      },
      {
         "time":1349011240,
         "temp":23.5
      }
   ]
}

2.2 RESTful web services

The term REST stands for REpresentational State Transfer. REST is a software architecture style that consists of clients and servers which co-operate by requests made by the client that are processed on a server and then responded back to the client. REST based web services working over HTTP are commonly called RESTful web services and are highly coupled to the types of HTTP methods. RESTful web services make use of GET, POST, PUT and DELETE methods. These method types and their intended functionalities are defined in the RFC 2616[3].

Simplified, a resource identified by Uniform resource identifier, URI, can be manipulated using the four previously mentioned HTTP method types. Let's consider an example of list of users for some web service. The URI for a collection of user items (a complete list of all users) could be http://service.fi/users/ and for a single item http://service.fi/users/13 where the last part of the URI after the slash would be a unique identifier for the user object.

By following these principles the interface is made compatible with wide variety of tools out of the box. Tools that expect the interface to work like this will automate many parts of creating, reading, updating and deleting the items (CRUD) contained in the collection.

2.2.1 HTTP GET

Making a GET request to the collection URI should return a complete listing of all users and making the GET request to the URI followed by the UID should return us one single user item corresponding to the identifier.

2.2.2 HTTP POST

The POST method is used to add new items to the collection. The data is included in the message payload after the HTTP headers. Making a POST request to our example collections URI with the new item in the message body should result in the item to be added in the collection on the server side. If the request is made to a URI followed by an ID the addressed member should be considered as a collection in its own right and create a new entry in it. As a response the item added to the server side collection should be returned if any changes like addition of unique id have been performed.

2.2.3 HTTP PUT

PUT is used for updating. If the request is targeted to the collections URI, the whole collection is to be replaced by the collection carried by the request or if in the case of single item identified by an ID only one item is updated.

2.2.4 HTTP DELETE

Using the delete method the whole collection or single item should be deleted from the server side.

2.3 Comparison to other interfacing practices

Although it can't be considered to be an interface, it's still quite common practice to hard code all variables and settings when dealing low level microcontroller based systems, even when the machine is connected to the Internet. In this case, for example changing settings like measurement interval etc. requires a full update of the program running on the system. If connected to the Internet these type of solutions usually just send the data off to a external server on a local network or Internet by use of simple HTTP GET or POST requests. Without an Internet connection the most common interfacing method is to connect the device to another computer using a serial port communication.

Another obvious option is to create a user interface that works on physical buttons possibly accompanied by a display that come in various flavors. The biggest downside with this traditional approach is that the physical user interface is strongly tied to the UI logic and completely static for physical changes. For example device already in production and in use will not get a new button in to its front panel even though a firmware update bringing new features would highly benefit from the addition.

HTML user interfaces over HTTP are quite popular already. For example most cable modems are configured using a web user interface running on a browser. Usually these solutions are implemented so that there is a web server running on an embedded Linux distribution taking care of the whole user interface logic and the html notation on top of it. Pages are served from the system as complete HTML documents. The biggest problem with this approach is that storing the HTML content and it's dependencies like image files or style sheets on the server side requires relatively large amounts of program memory and also just serving the content over HTTP is time and resource consuming. Harnessing this entire capacity for the UI doesn't seem practical if the main purpose of the device is just for example to collect sensor data and expose the data to the outside world.

The proposed solution is trying to solve these current limitations. By creating RESTful interface using JSON as the messaging format we are able significantly minimize the data needed to be transferred. The same interface can be used for big data solutions like NoSQL database engines as well as for HTML user interfaces and at the same time enables the machines to communicate with each other on a local network or even to the other side of the world through Internet connection. The possible applications of these interfaces vary from home automation systems to sensor networks. The same device can even simultaneously be part of many cooperative systems. All this can be achieved by using the same simple interface without the need of any kind of mediator in between.

By loading the actual graphical user interface markup and logic from another server, it's possible to reduce the need of program space and processing time on the system serving the interface. It also makes the GUI able to be updated without any changes to the RESTful interface itself. Same origin policy is fulfilled by serving the index page from the microcontroller containing the linked resources that will be loaded from elsewhere. We also get the opportunity to use all the techniques used in modern web user interfaces and it can be done cost efficiently and using higher level of programming tools than would be possible otherwise.

The RESTful JSON interface makes it possible for people who have little or no experience at all on microcontroller systems to use tools they already know for creating feature rich user interfaces for devices close to the hardware level. At the same time the UI logic is completely separated from the back-end implementation enabling work on the back-and front end to be carried out independently. If needed, the user interface can be developed against a temporary local interface implemented on some other programming language to be later switched to the one implemented on the microcontroller system.

The required hardware is available for cheap which makes these type of solutions competitive even for the most simplest machines which would normally be controlled for example with buttons and a LCD display. Every device doesn't need its own physical UI when most of us are surrounded by devices capable of controlling these machines over the Internet.

Update: more examples here.


Intent of these examples is to show how these simple interfaces can be used in variety of ways. The source code is shown only for the important parts, but full source code listings can be found at https://github.com/lasselukkari.

Both of these examples are programmed to be used on Arduino Mega 2560 microcontroller development board. The 8 bit 16 MHz processor on the board can be considered quite slow when compared to many of the alternatives. These examples will only go through the making of the RESTful HTTP interface for the microcontroller. Detailed walkthroughs of the implementation of HTML and JavaScript user interfaces are outside the scope of this paper, but full source code listings are available.

Both of the examples work on the same principle. They offer a minimal interface for accessing and manipulating data. In addition both serve a simple html page from the root address which contains references to the external JavaScript files and CSS style sheets.

For the microcontroller back-end running on the Arduino Mega I developed an inbred between two Arduino C++ libraries. The server implementation automating the creation, reading, updating and deleting of the JSON objects consists of a two open source libraries called aJson and Webduino. Webduino is a web server library which was extended to allow bindings of an aJson objects with URIs to be served over HTTP. The aJson library already had methods for parsing and printing textual presentations of JSON objects from and to C FILE streams so adapting it to parse and print using the Webduino server instance was a straightforward task.

The resulting combination can be used to implement a RESTful JSON interface that can be manipulated according to the HTTP method type standards. Binding the JSON object to the server instance requires only one line of code and the rest of the operations for full CRUD are performed automatically by the web server instance. The aJson library's methods can be used to add an additional persistence for the JSON objects. The objects can be stored in files located on a SD card or some other media.

These sequence diagrams show a common behavior scenario wherein first one the UI is loaded to the user's browser and in the second one settings are saved back to the microcontroller.


Figure 2: Sequence diagrams demonstrating how different parts of the implementation communicate with each other.

3.1 Example: TodoMVC back-end

“TodoMVC is a project which offers the same Todo application implemented using MV* concepts in most of the popular JavaScript MV* frameworks of today.”
-http://todomvc.com/

TodoMVC project has been created to help developers choose between the many front-end JavaScript frameworks available. In the examples on the project's website the same to-do list application is implemented using many different MV* frameworks. It's a good starting point for anyone interested in learning the basics of creating modern web applications and choosing a framework to support the development.

While the example is not so practical for the microcontroller systems considering that the device would have the possibility to interact with the real word in numerous ways, it serves its purpose here to use it as the starting point because it's a simple and representative example of a web app that demonstrates the creating, reading, updating and deleting of objects.

The original examples on the TodoMVC website use browser's local storage instead of HTTP interface. For this reason the Backbone.js implementation was modified a bit to perform the needed HTTP calls for the back-end. Backbone.js pretty much automates everything needed for the HTTP API calls when the models and collections are provided with correct URIs and appropriate methods are called for the model instances when saving and fetching them.

The interface needed for this example is simple. It consists of only one URI, todos.json, working according to the RFC 2616 specification.

3.1.1 User interface

The functionality of the front-end user interface is outside the scope of this paper. Full source code listings are available from the GitHub page https://github.com/lasselukkari/Todo. But for illustrative purposes a screenshot is provided here.


Figure 3: User interface screenshot.

3.1.2 REST Interface definition

Making a HTTP GET request to the URI todos.json will result in a collection of all items to be returned as a JSON array.

HTTP Request MethodGET
Example request URLtodos.json
Example Request payload-
Example JSON response[{"title":"Arduino JSON REST interface","completed":true,"order":1,"id":1}]

For updating a single todo item i.e. a single BackBone model in a BackBone collection a HTTP POST request is made to URI todos.json followed by a slash and the models id. The updated model is included in the request payload.

HTTP Request MethodPOST
Example request URLtodos.json
Example Request payload{"title":"Updated todo title","completed":false,"order":1,"id":1}
Example JSON response-

When a new item is added to the collection HTTP PUT request is called to the URI with the new object that doesn't yet have an id. The interface adds the item to the collection on the server side and assigns a proper id to it. Response to the request is the original item that was added but included with an id.

HTTP Request MethodPUT
Example request URLtodos.json
Example Request payload{"title":"New todo item","completed":false,"order" :2}
Example JSON response{"title":"New todo item","completed":false,"order" :2,"id":2}

To delete an item a simple HTTP DELETE request to the URI is made followed by the items ID. This instructs the back-end to delete the item specified by the ID.

HTTP Request MethodDELETE
Example request URLtodos.json
Example Request payload-
Example JSON response-

3.1.3 C++ back-end implementation for Arduino Mega 2560

The backend implementation is code wise relatively short. All three implementation specific code lines have been commented. This back-end example demonstrates how a simple interface can be defined. The different HTTP method type calls explained previously are automatically handled by the web server instance managing the CRUD.

#include "SPI.h"
#include "Ethernet.h"
#include "WebServer.h"
#include "aJSON.h"

byte mac[] = {
0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x02 };
EthernetClient client;
WebServer webserver("", 80);
aJsonObject* todos;

void indexCmd(WebServer &server, WebServer::ConnectionType type, char *, bool) {
server.httpSuccess();
if (type != WebServer::HEAD) {
P(helloMsg) =
"<!doctype html>\n"
"<html lang=\"en\">\n,"
"<head>\n"
" <meta charset=\"utf-8\">\n <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\n"
" <title>Backbone.js \u2022 TodoMVC</title>\n"
" <link rel=\"stylesheet\" href=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/css/base.css\">\n"
" <!--[if IE]>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/ie.js\"></script>\n"
" <![endif]-->\n"
"</head>\n"
"<body>\n"
" <section id=\"todoapp\">\n"
" <header id=\"header\">\n"
" <h1>todos</h1>\n"
" <input id=\"new-todo\" placeholder=\"What needs to be done?\" autofocus>\n"
" </header>\n"
" <section id=\"main\">\n"
" <input id=\"toggle-all\" type=\"checkbox\">\n"
" <label for=\"toggle-all\">Mark all as complete</label>\n"
" <ul id=\"todo-list\"></ul>\n"
" </section>\n"
" <footer id=\"footer\"></footer>\n"
" </section>\n"
" <div id=\"info\">\n"
" <p>Double-click to edit a todo</p>\n"
" <p>Written by <a href=\"https://github.com/addyosmani\">Addy Osmani</a></p>\n"
" <p>Part of <a href=\"http://todomvc.com\">TodoMVC</a></p>\n"
" </div>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/jquery.min.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/lodash.min.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/lib/backbone-min.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/models/todo.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/collections/todos.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/views/todos.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/views/app.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/routers/router.js\"></script>\n"
" <script src=\"http://www.cs.helsinki.fi/u/ljlukkar/todo/js/app.js\"></script>\n"
"</body>\n"
"</html>";
server.printP(helloMsg);
}
}

void setup() {

todos =
aJson.parse(
"[{\"title\":\"Arduino JSON REST interface\",\"completed\":true,\"order\":1,\"id\":1}]");


// aJsonObject* todo = aJson.createObject();
// aJson.addItemToObject(todo, "title", aJson.createItem("Arduino JSON REST interface"));
// aJson.addTrueToObject(todo, "completed");
// aJson.addNumberToObject(todo,"order", 1);
// aJson.addNumberToObject(todo,"id", 1);

// aJsonObject* todos = aJson.createArray();
// aJson.addItemToArray(todos, todo);

webserver.addJSONBinding(&todos, "todos.json");
webserver.setDefaultCommand(&indexCmd);

Serial.begin(9600);
Ethernet.begin(mac);
webserver.begin();

for (byte i = 0; i < 4; i++) {
// print the value of each byte of the IP address:
Serial.print(Ethernet.localIP()[i]);
if (i < 3) {
Serial.print('.');
}
}
}

void loop() {
webserver.processConnection();
}

3.2 Example: Weather Station

The weather station example shows further how the JSON objects can be used to store program variables and how those variables can be accessed. The full source code is too long to be included completely but important parts of it are covered. The microcontroller implementation uses an AM2303[4] air temperature and humidity sensor. If no sensor is available the measurement values are randomly generated so that the example can be run.

3.2.1 User interface

The front-end makes use of Backbone.js JavaScript framework as the previous example and the UI is constructed using jQueryMobile which is JavaScript web application framework geared towards mobile devices such as tablet computers and other mobile devices like cellular phones providing a user interface which mimics more the device's native functionality. Because it is used only for the user interface and not for the application logic some of functionalities have to be crippled. For the graph showing the measured temperature and humidity values a JavaScript chart library called Highcharts is used. All values are updated according to the interval set by the user.


Figure 4: User interface screenshot, measurements view.


Figure 5: User interface screenshot, settings view.

3.2.2 REST interface

The REST interface consists of two JSON objects: one for the measurements and one for the settings. The measurements object contains the latest measurements which are read from the sensor and updated with an interval specified in the settings. The same interval is used in the front-end for timing how often the measurements should be fetched using the JSON interface The measurements object is only read and the settings object is read and updated but not deleted or created by the interface

3.2.3 C++ back-end implementation for Arduino Mega 2560

First in the beginning of the setup function the JSON objects are created and bound to the web server.

//The setup function is called once at startup of the sketch
void setup() {

measurements = aJson.parse("{\"temperature\":0.0, \"humidity\":0.0, \"time\":0}");
settings = aJson.parse("{\"id\":\"interval\", \"interval\":1}");

webserver.addJSONBinding(&measurements, "measurements.json");
webserver.addJSONBinding(&settings, "settings.json");

webserver.setDefaultCommand(&indexCmd);

lastMeasured = 0;

Serial.begin(9600);
Ethernet.begin(mac);
webserver.begin();

for (byte i = 0; i < 4; i++) {
// print the value of each byte of the IP address:
Serial.print(Ethernet.localIP()[i]);
if (i < 3) {
Serial.print('.');
}
}
}

In the main loop the measurements are taken according to the interval specified in the settings object.

// The loop function is called in an endless loop
void loop() {

webserver.processConnection();
if (millis()
> lastMeasured
+ aJson.getObjectItem(settings, "interval")->valueint
* 1000) {
takeMeasurements();
lastMeasured = millis();
}
}

In the takeMeasurements() method readings from the sensor or the random values are updated to the measurements object.

void takeMeasurements() {
byte rounds = 0;
//try to read five times
while (++rounds < 5) {
if (DHT.read22(DHT22_PIN) == DHTLIB_OK) {
aJson.getObjectItem(measurements, "temperature")->valuefloat =
DHT.temperature;
aJson.getObjectItem(measurements, "humidity")->valuefloat =
DHT.humidity;
break;
}
}
//no sensor present, random numbers
if (rounds == 5) {
aJson.getObjectItem(measurements, "temperature")->valuefloat = random(
-300, +300) / 10.0;
aJson.getObjectItem(measurements, "humidity")->valuefloat = random(1000)
/ 10.0;
}
//add time stamp, model on the front-end doesn't update if none of the values change. rtc would fit here
aJson.getObjectItem(measurements, "time")->valueint = millis() / 1000;
}

While the implementation seems to be working fine with the examples given a lot more work and testing needs to be done. The usability or the JSON library is at the moment limited by the data types available. The operations manipulating the JSON objects according to the HTTP requests would require more sanity checks. There are also quite a lot of open problems concerning security aspects.

Basic HTTP authentication is quite easy to implement and is already supported by the Webduino web server library, but authentication doesn't crypt the actual traffic. For using TLS or SSL cryptographic protocols a more powerful processing unit or additional cryptography chip would be required. Another obvious security problem is the limited HTTP request handling capacity. Opening an access to a device via public Internet makes the system vulnerable to DOS attacks. Malicious attacker wouldn't even need a large botnet to make the target struggle with flooding requests. A table of IP addresses keeping track of request rate intervals would be a partial solution but wouldn't completely remove the vulnerability.

Automatic service discovery will open even more interesting possibilities for home automation solutions. By using Bonjour/Zeroconf protocol to publish services of the devices on a local network the devices could be controlled centralized or they could find each other and cooperate without assistance. For example after connecting to a local wireless network using a cell phone all available devices could be automatically listed on a mobile phone application. Or ventilation and heating system could react to temperature readings taken by another device on the network. Defining a common instruction vocabulary would make it possible for example to control all the lights of an apartment simultaneously. I have made some preliminary tests using the Arduino prototyping platform and an Android cell phone.

The toolset used for the examples have been proven to be useful and worth of future development. Adopting the techniques used widely in web development to these devices with lot less horsepower seems to be reasonable and opens up new possibilities.

A big thank you goes to my little brother, Jaakko, who has helped me out a lot especially with JavaScript problems and questions.

[1] Crockford, D., 2006. The application/json Media Type for JavaScript Object Notation (JSON), IETF RFC 4627 [Online].
Available: http://www.ietf.org/rfc/rfc4627.txt

[2] Ecma International, 2011. ECMAScript Language Specification [Online].
Available: http://www.ecmainternational. org/publications/files/ECMA-ST/Ecma-262.pdf

[3] Fielding, R., Irvine, UC, Gettys, J., Compaq/W3C, Mogul, J., Compaq, Frystyk, H., W3C/MIT, Masinter, L., Xerox, Leach, P., Microsoft, Berners-Lee, T., W3C/MIT, 1999. Hypertext Transfer Protocol -- HTTP/1.1, IETF RFC 2616 [Online].
Available: http://www.ietf.org/rfc/rfc2616.txt

[4] Aosong(Guangzhou) Electronics Co.,Ltd., AM2302 Product Manual [Online].
Available: http://www.aosong.com/asp_bin/Products/en/AM2302.pdf