Implement Internet of Things Edge Analytics with IBM Quarks

Although many interested readers may have a focus on the hardware side of the Internet of Things, there is much that can be done with the data and software side as well. One interesting new and open-source solution is the Quarks framework from IBM.

Quarks is a framework for implementing and deploying edge analytics on a variety of different data streams and devices. Quarks allows you to push data analysis and machine learning out to "edge" devices in Internet-of-Things applications - for example, routers, gateways, sensors, appliances and machines at the edge of the network, rather than centralised servers.

This new system runs locally on gateways and edge devices, analysing their streams of data. This means these devices can perform analytics or make decisions based on the data they receive locally, resulting in faster responsiveness and reduced communications costs.

Quarks enables continuous streaming analytics on gateways and edge devices which can work together with centralised server-side systems to provide efficient and timely analytics across the whole network and the IoT ecosystem, from the centre out to the edge.

Performing some analysis of data at the network edge where this data is generated means that the amount of data that needs to be transmitted back to a central server is reduced, and the amount of data that needs to be stored is reduced. Quarks was developed to provide an SDK and an embeddable runtime for these kinds of streamable analytics applications on lightweight devices. It provides APIs and a lightweight runtime to allow you to build analytics for streaming data at the network-edge devices in IoT applications.

This means that Quarks enables you to process data locally - in a lightweight device such as an embedded computer in a car engine, an Android phone, or a lightweight platform such as a Raspberry Pi, for example, before data is transmitted over the network. Quarks provides modularity and a micro-kernel runtime which is resource-efficient to allow deployments on these kinds of devices with constraints in hardware resources such as memory, including only the components in the build that are needed for that specific device or application.

Quarks addresses requirements for analytics at the edge in IoT use-cases that are not addressed well by central server-based analytics solutions. Using centralised analytical tools to make interpretations and decisions from IoT data traditionally means that the data must be transmitted over a network - and this can be problematic when using systems such as long-range wireless, satellite or cellular communications where bandwidth is constrained or expensive.

Reducing server-side analytics in favour of edge-device analytics reduces the amount of data that needs to be transmitted. This decision making at the edge-node IoT device also allows for relatively fast "decision making" and control of connected devices, without waiting for communications back from the server.

Quarks uses Apache Common Math to provide simple analytics aimed at device sensors, for example windowing, aggregation and simple filtering. This local processing means that devices can react locally, offload processing from central servers, and reduce bandwidth costs.

Furthermore, Quarks applications use analytics to determine when data needs to be transmitted to a back-end system for further analysis, action or storage. For example, an IoT sensor may determine whether a system is running outside of normal parameters, such as an engine that is running too hot.

If the system is running normally, you probably don't need to send the data to the back-end system - this means additional load, bandwidth costs, and extra storage that is needed. But if your Quarks-based analytics detect an abnormal condition, you can then decide to log that data, and send the data to the back-end system to determine why this issue is occurring and how to resolve it.

This makes the whole system much more efficient, with reduced bandwidth and reduced storage requirements, while still capturing the important data where it matters. By detecting only the abnormalities and discarding the "normal" data - you shift from sending a continuous flow of data to sending only the essential and meaningful data, locally filtering the interesting from the mundane and reducing these costs.

You can write an edge application on Quarks and connect it to a cloud-computing service such as IBM's Watson IoT platform. Quarks can also be used for enterprise data collection and analysis, such as log collectors, application data, and data centre analytics, for example.

And Quarks can be integrated with centralised analytics systems such as IBM Streams or Apache Storm to provide edge-to-centre analytics. This means you have the best of both worlds, with the benefits of both edge-device and server-based analytics and tools for decision making.

Quarks supports connectors for MQTT, HTTP, JDBC, Apache Kafka and IBM's Watson platform, - as well as analytics systems such as IBM Streams and IBM Bluemix streaming analytics or open-source systems such as Apache's Spark, Storm, Flink and Samza platforms.

And the best thing is that because Quarks is open-source and highly extensible, it's possible to add support for custom connectors and analytics applications of your choice, if the currently supported systems do not meet your needs.

And this is where the LX Group us ready to work with you. We have end-to-end experience and demonstrated results in the entire process of IoT product development, and we’re ready to help bring your existing or new product ideas to life. Getting started is easy – click here to contact us, telephone 1800 810 124, or just keep in the loop by connecting here.

LX is an award-winning electronics design company based in Sydney, Australia. LX services include full turnkey design, electronics, hardware, software and firmware design. LX specialises in IoT embedded systems and wireless technologies design.

Published by LX Pty Ltd for itself and the LX Group of companies, including LX Design House, LX Solutions and LX Consulting, LX Innovations.