Autoplant, Part 1: Overview

At a recent plant sale at my University, I bought myself both a parlour palm and an areca palm for my desk. They look lovely, but I always worry that I'm going to forget to water them.

Having some experience with arduino already, I decided to resolve the issue by implementing an arduino-based system to monitor my new plants and log the resulting data in my existing collectd-based (see also CGP which I use, but sadly it's abandonware. Any suggestions for alternatives are greatly appreciated) monitoring system I use for the various servers and systems that I manage.

The eventual aim is to completely automate the watering process too - but before I can do that I need to first get the monitoring up and running so that I can calibrate the sensors, so this is what I'll be focusing on in this post.

Circuit design

To do this, I've used a bunch of parts I have lying around (plug a few new ones), and wired up a NodeMCU v0.9 to some sensors:

(Above: The circuit I've wired up. See the Fritzing file.)

A full list of parts can be found at the end of this post, along with links to where I got them from. The sensors I'm using are:

• 2 x Capacitive soil moisture sensors
• 2 x Liquid level sensors
• 1 x BME280 that I had lying around and thought "why not?"

Both the soil sensors and the liquid level sensors give out an analogue signal (shown in orange), but unfortunately the NodeMCU v0.9 (based on the ESP8266) only has a single analogue pin, so I bought myself a CD4051 from switch electronics (link at the bottom of this post) to act as a multiplexer. Given 3 digital wires to act as a channel selector (shown in purple), it allows you to select between 8 different analogue channels - perfect for my use-case. You can see it in the diagram at the left-hand side of the larger breadboard.

While the Fritzing design for a USB breakout board isn't exactly the same as the ones I have, I plan on using them to transport power, ground, and the 2 analogue outputs from the 2 sensors for the plant on the other side of my desk.

The other component here is a BME280 I have lying around to monitor temperature, humidity, and air pressure. This isn't required, but since I have one lying around anyway I thought it was a good idea to use it.

Networking

With the circuit designed and implemented (still got to finalise the USB breakout boards), the next thing to organise is the transport and logging for the data generated. MQTT is easy to use on the Arduino because of PubSubClient, so that's what I decided on using.

Time for another diagram!

(Can't see the above? Try a PNG instead.)

If you've followed my blog here for a while, you might remember that I have a cluster that's powered by a bunch of Raspberry Pis running Hashicorp Nomad and Consul.

On this cluster - amongst a many other things - I have an MQTT server (check out my earlier post on how to set one up) running inside a Docker container as a Nomad task. Connections to my Mosquitto MQTT server are managed by Fabio, which reverse-proxies connections to Mosquitto. Fabio exposes2 ports by which one can connect to the MQTT server:

• TCP port 1883: Unencrypted MQTT
• TCP port 8883: (TLS encrypted) MQTTS

In doing so, Fabio terminates TLS so that Mosquitto doesn't need to have access to my TLS certificates. For those interested, the proxy.addr in your fabio.properties file is actually a comma-separated list, and to achieve TLS termination for MQTTS you can add something like this to proxy.addr (make sure to use an up-to-date cipher list):

:1883;proto=tcp,:8883;proto=tcp;cs=mooncarrot;tlsmin=tls12;tlsciphers="TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

Then if we continue working backwards, the next piece of the puzzle is port forwarding on my router. While Fabio exposes both MQTT and MQTTS, I only port-forward MQTTS and not unencrypted MQTT. My autoplant system will only be communicating over MQTTS, so it doesn't make sense to expose the less secure unencrypted port to the world too.

From here, no matter where in the world my autoplant monitoring system is located it will still be able to send sensor values back to be logged.

Working forwards again from the Mosquitto MQTT server, the Arduino is going to send MQTT messages to the sensors/data topic with messages that look something like this (but minified):

{
    "id": "plant-a",
    "sensor": "soil",
    "value": 2.41
}

The MQTT plugin for Collectd doesn't support JSON input though, so I'm going to write a broker that translates messages from the JSON format above to the format that Collectd likes. My reasoning for this is twofold:

1. I also want to log data to a tab-separated-values file for later long-term analysis
2. The collectd format is somewhat complicated and has the potential to mess up other parts of my monitoring system, so I want to do some extra validation on incoming data from remote sensors

I haven't fully decided on which language I'm going to write this validating broker in, but I'm thinking it might end up being a shell script. Weird as it sounds, I found a cool MQTT library for Bash called bish-bosh which I want to try out. My second choice here would be Rust, but I unfortunately can't find a (preferably pure-rust) MQTT(S) library, which I'm finding rather strange.

Either way, if possible I'm going to package up the completed implementation of this broker into a Docker container and write a quick Hashicorp Nomad job file to get it running on my cluster so that it benefits from the reundancy of my Nomad cluster. Then, with collectd listening on another topic, it can transparently bridge the 2. I'm not quite sure how will collectd's MQTT plugin actually works though, so this shell script may end up being a child process of my main collectd server using the exec plugin instead.

Conclusion

In this post, I've outlined my solution to a seemingly simple problem of watering plants automatically (because all simple problems need complex solutions :P). With an arduino-based system, I'm going to send messages via MQTTS containing sensor data, which will then be passed to a backend for processing. In future posts in this series, I want to (in no particular order):

• Go through the code for the arduino I've written
• Look at the code I have yet to write for the translation broker
• Explore the 3d printed parts I have yet to design for various purposes
• Show off the final completed circuit in action
• Look at some initial statistics collected by the sensors
• Start to play with pumping water around and different holding containers / designs, etc

Not all of these warrant a separate post, but there are definitely multiple posts to come on this project :D

Full parts list

• 1 x NodeMCU v0.9 (anything with WiFi will do, I just had an old on lying around)
• 1 x Half size breadboard
• 1 x Mini breadboard (the smallest I had)
• 2 x Capacitive soil moisture sensor (source, significantly cheaper if you're willing to wait ages)
• 2 x Waveshare liquid level sensor (source, original website)
• 1 x CD4051 8:1 analogue multiplexer (source - breadboard compatible)
• 2 x USB type a breakouts
• 1 x BME 280 (I got mine cheap on AliExpress)
• Lots of jumper wires

Summer Project Part 5: When is a function not a function?

Another post! Looks like I'm on a roll in this series :P

In the last post, I looked at the box I designed that was ready for 3D printing. That process has now been completed, and I'm now in possession of an (almost) luminous orange and pink box that could almost glow in the dark.......

I also looked at the libraries that I'll be using and how to manage the (rather limited) amount of memory available in the AVR microprocessor.

Since last time, I've somehow managed to shave a further 6% program space off (though I'm not sure how I've done it), so most recently I've been implementing 2 additional features:

• An additional layer of AES encryption, to prevent The Things Network for having access to the decrypted data
• GPS delta checking (as I'm calling it), to avoid sending multiple messages when the device hasn't moved

After all was said and done, I'm now at 97% program space and 47% global variable RAM usage.

To implement the additional AES encryption layer, I abused LMiC's IDEETRON AES-128 (ECB mode) implementation, which is stored in src/aes/ideetron/AES-128_V10.cpp.

It's worth noting here that if you're doing crypto yourself, it's seriously not recommended that you use ECB mode. Please don't. The only reason that I used it here is because I already had an implementation to hand that was being compiled into my program, I didn't have the program space to add another one, and my messages all start with a random 32-bit unsigned integer that will provide a measure of protection against collision attacks and other nastiness.

Specifically, it's the method with this signature:

void lmic_aes_encrypt(unsigned char *Data, unsigned char *Key);

Since this is an internal LMiC function declared in a .cpp source file with no obvious header file twin, I needed to declare the prototype in my source code as above - as the method will only be discovered by the compiler when linking the object files together (see this page for more information about the C++ compilation process. While it's for regular Linux executable binaries, it still applies here since the Arduino toolchain spits out a very similar binary that's uploaded to the microprocessor via a programmer).

However, once I'd sorted out all the typing issues, I slammed into this error:

/tmp/ccOLIbBm.ltrans0.ltrans.o: In function `transmit_send':
sketch/transmission.cpp:89: undefined reference to `lmic_aes_encrypt(unsigned char*, unsigned char*)'
collect2: error: ld returned 1 exit status

Very strange. What's going on here? I declared that method via a prototype, didn't I?

Of course, it's not quite that simple. The thing is, the file I mentioned above isn't the first place that a prototype for that method is defined in LMiC. It's actually in other.c, line 35 as a C function. Since C and C++ (for all their similarities) are decidedly different, apparently to call a C function in C++ code you need to declare the function prototype as extern "C", like this:

extern "C" void lmic_aes_encrypt(unsigned char *Data, unsigned char *Key);

This cleaned the error right up. Turns out that even if a function body is defined in C++, what matters is where the original prototype is declared.

I'm hoping to release the source code, but I need to have a discussion with my supervisor about that at the end of the project.

Found this interesting? Come across some equally nasty bugs? Comment below!

Summer Project Part 4: Threading the needle and compacting it down

In the last part, I put the circuit for the IoT device together and designed a box for said circuit to be housed inside of.

In this post, I'm going to talk a little bit about 3D printing, but I'm mostly going to discuss the software aspect of the firmware I'm writing for the Arduino Uno that's going to be control the whole operation out in the field.

Since last time, I've completed the design for the housing and sent it off to my University for 3D printing. They had some great suggestions for improving the design like making the walls slightly thicker (moving from 2mm to 4mm), and including an extra lip on the lid to keep it from shifting around. Here are some pictures:

(Left: The housing itself. Right: The lid. On the opposite side (not shown), the screw holes are indented.)

At the same time as handling sending the housing off to be 3D printed, I've also been busily iterating on the software that the Arduino will be running - and this is what I'd like to spend the majority of this post talking about.

I've been taking an iterative approach to writing it - adding a library, interfacing with it and getting it to do what I want on it's own, then integrating it into the main program.... and then compacting the whole thing down so that it'll fit inside the Arduino Uno. The thing is, the Uno is powered by an ATmega328P (datasheet). Which has 32K of program space and just 2K of RAM. Not much. At all.

The codebase I've built for the Uno is based on the following libraries.

• LMiC (the matthijskooijman fork) - the (rather heavy and needlessly complicated) LoRaWAN implementation
• Entropy for generating random numbers as explained in part 2
• TinyGPS, for decoding NMEA messages from the NEO-6M
• SdFat, for interfacing with microSD cards over SPI

Memory Management

Packing the whole program into a 32K + 2K box is not exactly an easy challenge, I discovered. I chose to first deal with the RAM issue. This was greatly aided by the FreeMemory library, which tells you how much RAM you've got left at a given point in the execution of your program. While it's a bit outdated, it's still a useful tool. It works a bit like this:

#include <MemoryFree.h>;

void setup() {
    Serial.begin(115200);
    Serial.println(freeMemory, DEC);
    char test[] = "Bobs Rockets";
    Serial.println(freeMemory, DEC); // Should be lower than the above call
}

void loop() {
    // Nothing here
}

It's worth taking a moment to revise the way stacks and heaps work - and the differences between how they work in the Arduino environment and on your desktop. This is going to get rather complicated quite quickly - so I'd advise reading this stack overflow answer first before continuing.

First, let's look at the locations in RAM for different types of allocation:

• Things on the stack
• Things on the heap
• Global variables

Unlike on the device you're reading this on, the Arduino does not support multiple processes - and therefore the entirety of the RAM available is allocated to your program.

Since I wasn't sure about preecisly how the Arduino does it (it's processor architecture-specific), I wrote a simple test program to tell me:

#include <Arduino.h>

struct Test {
    uint32_t a;
    char b;
};

Test global_var;

void setup() {
    Serial.begin(115200);

    Test stack;
    Test* heap = new Test();

    Serial.print(F("Stack location: "));
    Serial.println((uint32_t)(&stack), DEC);

    Serial.print(F("Heap location: "));
    Serial.println((uint32_t)heap, DEC);

    Serial.print(F("Global location: "));
    Serial.println((uint32_t)&global_var, DEC);
}

void loop() {
    // Nothing here
}

This prints the following for me:

Stack location: 2295
Heap location: 461
Global location: 284

From this we can deduce that global variables are located at the beginning of the RAM space, heap allocations go on top of globals, and the stack grows down starting from the end of RAM space. It's best explained with a diagram:

Now for the differences. On a normal machine running an operating system, there's an extra layer of abstraction between where things are actually located in RAM and where the operating system tells you they are located. This is known as virtual memory address translation (see also virtual memory, virtual address space).

It's a system whereby the operating system maintains a series of tables that map physical RAM to a virtual address space that the running processes actually use. Usually each process running on a system will have it's own table (but this doesn't mean that it will have it's own physical memory - see also shared memory, but this is a topic for another time). When a process accesses an area of memory with a virtual address, the operating system will transparently translate the address using the table to the actual location in RAM (or elsewhere) that the process wants to access.

This is important (and not only for security), because under normal operation a process will probably allocate and deallocate a bunch of different lumps of memory at different times. With a virtual address space, the operating system can defragment the physical RAM space in the background and move stuff around without disturbing currently running processes. Keeping the free memory contiguous speeds up future allocations, and ensures that if a process asks for a large block of contiguous memory the operating system will be able to allocate it without issue.

As I mentioned before though, the Arduino doesn't have a virtual memory system - partly because it doesn't support multiple processes (it would need an operating system for that). The side-effect here is that it doesn't defragment the physical RAM. Since C/C++ isn't a managed language, we don't get _heap compaction_ either like in .NET environments such as Mono.

All this leads us to an environment in which heap allocation needs to be done very carefully, in order to avoid fragmenting the heap and causing a stack crash. If an object somewhere in the middle of the heap is deallocated, the heap will not shrink until everything to the right of it is also deallocated. This post has a good explanation of the problem too.

Other things we need to consider are keeping global variables to a minimum, and trying to keep most things on the stack if we can help it (though this may slow the program down if it's copying things between stack frames all the time).

To this end, we need to choose the libraries we use with care - because they can easily break these guidelines that we've set for ourselves. For example, the inbuilt SD library is out, because it uses a global variable that eats over 50% of our available RAM - and it there's no way (that I can see at least) to reclaim that RAM once we're finished with it.

This is why I chose SdFat instead, because it's at least a little better at allowing us to reclaim most of the RAM it used once we're finished with it by letting the instance fall out of scope (though in my testing I never managed to reclaim all of the RAM it used afterwards).

Alternatives like µ-Fat do exist and are even lighter, but they have restrictions such as no appending to files for example - which would make the whole thing much more complicated since we'd have to pre-allocate the space for the file (which would get rather messy).

The other major tactic you can do to save RAM is to use the F() trick. Consider the following sketch:

#include 

void setup() {
    Serial.begin(115200);
    Serial.println("Bills boosters controller, version 1");
}

void loop() {
    // Nothing here
}

On the highlighted line we've got an innocent Serial.println() call. What's not obvious here is that the string literal here is actually copied to RAM before being passed to Serial.println() - using up a huge amount of our precious working memory. Wrapping it in the F() macro forces it to stay in your program's storage space:

Serial.println(F("Bills boosters controller, version 1"));

Saving storage

With the RAM issue mostly dealt with, I then had to deal with the thorny issue of program space. Unfortunately, this is not as easy as saving RAM because we can't just 'unload' something when it's not needed.

My approach to reducing program storage space was twofold:

• Picking lightweight alternatives to libraries I needed
• Messing with flags of said libraries to avoid compiling parts of libraries I don't need

It is for these reasons that I ultimately went with TinyGPS instead of TinyGPS++, as it saved 1% or so of the program storage space.

It's also for this reason that I've disabled as much of LMiC as possible:

#define DISABLE_JOIN
#define DISABLE_PING
#define DISABLE_BEACONS
#define DISABLE_MCMD_DCAP_REQ
#define DISABLE_MCMD_DN2P_SET

This disables OTAA, Class B functionality (which I don't need anyway), receiving messaages, the duty cycle cap system (which I'm not sure works between reboots), and a bunch of other stuff that I'd probably find rather useful.

In the future, I'll probably dedicate an entire microcontroller to handling LoRaWAN functionality - so that I can still use the features I've had to disable here.

Even doing all this, I still had to trim down my Serial.println() calls and remove any non-essential code to bring it under the 32K limit. As of the time of typing, I've got jut 26 bytes to spare!

Next time, after tuning the TPL5110 down to the right value, we're probably going to switch gears and look at the server-side of things - and how I'm going to be storing the data I receive from the Arudino-based device I've built.

Found this interesting? Got a suggestion? Comment below!

Summer Project Part 3: Putting it together

In the first post in this series, I outlined my plans for my Msc summer project and what I'm going to be doing. In the second post, I talked about random number generation for the data collection.

In this post, I'm going to give a general progress update - which will mostly centre around the Internet of Things device I'm building to collect the signal strength data.

Since the last post, I've got nearly all the parts I need for the project, except the TPL5111 power manager and 4 rechargeable AA batteries (which should be easy to come by - I'm sure I've got some lying around somewhere).

I've also wired the thing up, with a cable standing in for the TPL5111.

The power management board there technically doesn't need a breadboard, but it makes mounting it in the box easier.

I still need to splice the connector onto the battery box I had lying around with some soldering and electrical tape - I'll do that later this week.

The wiring there is kind of messy, but I've tested each device individually and they all appear to work as intended. Here's a clearer diagram of what's going on that drew up in Fritzing (sudo apt install fritzing for Linux users):

Speaking of mounting things in the box, I've discovered OpenSCAD thanks to help from a friend and have been busily working away at designing a box to put everything in that can be 3D printed:

I've just got the lid to do next (which I'm going to do after writing this blog post), and then I'm going to get it printed.

With this all done, it's time to start working on the transport for the messages - namely using LMIC to connection to the network and send the GPS location to the application server, which is also unfinished.

The lovely people at the hardware meetup have lent me a full 8-channel LoRaWAN gateway that's connected to The Things Network for my project, which will make this process a lot easier.

Next time, I'll likely talk about 3D printing and how I've been 'threading the needle', so to speak.

Summer Project Part 2: Random Number Analysis with Gnuplot

In my last post about my Masters Summer Project, I talked about my plans and what I'm doing. In this post, I want to talk about random number generator evaluation.

As part of the Arduino-based Internet of Things device that will be collecting the data, I need to generate high-quality random numbers in order to ensure that the unique ids I use in my project are both unpredictable and unique.

In order to generate such numbers, I've found a library that exploits the jitter in the inbuilt watchdog timer that's present in the Arduino Uno. It's got a manual which is worth a read, as it explains the concepts behind it quite well.

After some experimenting, I ended up with a program that would generate random numbers as fast as it could:

// Generate_Random_Numbers - This sketch makes use of the Entropy library
// to produce a sequence of random integers and floating point values.
// to demonstrate the use of the entropy library
//
// Copyright 2012 by Walter Anderson
//
// This file is part of Entropy, an Arduino library.
// Entropy is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// Entropy is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with Entropy.  If not, see <http://www.gnu.org/licenses/>.
// 
// Edited by Starbeamrainbowlabs 2019

#include "Entropy.h"

void setup() {
    Serial.begin(9600);

    // This routine sets up the watch dog timer with interrupt handler to maintain a
    // pool of real entropy for use in sketches. This mechanism is relatively slow
    // since it will only produce a little less than two 32-bit random values per
    // second.
    Entropy.initialize();
}

void loop() {
    uint32_t random_long;
    random_long = Entropy.random();
    Serial.println(random_long);
}

As you can tell, it's based on one of the examples. You may need to fiddle around with the imports in order to get it to work, because the Arduino IDE is terrible.

With this in place and uploaded to an Arduino, all I needed to do was log the serial console output to a file. Thankfully, this is actually really quite easy on Linux:

screen -S random-number-generator dd if=/dev/ttyACM0 of=random.txt bs=1

Since I connected the Arduino in question to a Raspberry Pi I have acting as a file server, I've included a screen call here that ensures that I can close the SSH session without it killing the command I'm executing - retaining the ability to 'reattach' to it later to check on it.

With it set off, I left it for a day or 2 until I had at least 1 MiB of random numbers. Once it was done, I ended up with a file looking a little like this:

216767155
986748290
455286059
1956258942
4245729381
3339111661
1821899502
3892736709
3658303796
2524261768
732282824
999812729
1312753534
2810553575
246363223
4106522438
260211625
1375011617
795481000
319056836

(Want more? Download the entire set here.)

In total, it generated 134318 numbers for me to play with, which should be plenty to graph their distribution.

Graphing such a large amount of numbers requires a special kind of program. Since I've used it before, I reached for Gnuplot.

A histogram is probably the best kind of graph for this purpose, so I looked up how to get gnuplot to draw one and tweaked it for my own purposes.

I didn't realise that you could do arbitrary calculations inside a Gnuplot graph definition file, but apparently you can. The important bit below is the bin_width variable:

set key off
set border 3

# Add a vertical dotted line at x=0 to show centre (mean) of distribution.
#set yzeroaxis

# Each bar is half the (visual) width of its x-range.
#set boxwidth 0.05 absolute
#set style fill solid 1.0 noborder

bin_width = 171797751.16;

bin_number(x) = floor(x / bin_width)

rounded(x) = bin_width * ( bin_number(x) + 0.5 )

set terminal png linewidth 3 size 1920,1080

plot 'random.txt' using (rounded($1)):(1) smooth frequency with boxes

It specifies the width of each bar on the graph. To work this out, we need to know the maximum number in the dataset. Then we can divide it by the target number of bins to get the width thereof. Time for some awk!

awk 'BEGIN { a = 0; b=999999999999999 } { if ($1>0+a) a=$1; if ($1 < 0+b) b=$1; } END { print(a, b); }' <random.txt

This looks like a bit of a mess, so let's unwind that awk script so that we can take a better look at it.

BEGIN {
    max = 0;
    min = 999999999999999
}
{
    if ($1 > max)
        max = $1;
    if ($1 < min)
        min = $1;
}
END {
    print("Max:", max, "Min:", min);
}

Much better. In short, it keeps a record of the maximum and minimum numbers it's seen so far, and updates them if it sees a better one. Let's run that over our random numbers:

awk -f minmax.awk 

Excellent. 4294943779 ÷ 25 = 171797751.16 - which is how I arrived at that value for the bin_width earlier.

Now we can render our graph:

gnuplot random.plt >histogram.png && optipng -o7 histogram.png

I always optimise images with either optipng or jpegoptim to save on storage space on the server, and bandwidth for readers - and in this case the difference was particularly noticeable. Here's the final graph:

As you can see, the number of numbers in each bin is pretty even, so we can reasonably conclude that the algorithm isn't too terrible.

What about uniqueness? Well, that's much easier to test than the distribution. If we count the numbers before and after removing duplicates, it should tell us how many duplicates there were. There's even a special command for it:

wc -l <random.txt 
134318
sort <random.txt | uniq | wc -l
134317
sort <random.txt | uniq --repeated
1349455381

Very interesting. Out of ~134K numbers, there's only a single duplicate! I'm not sure whether that's a good thing or not, as I haven't profiled any other random number generated in this manner before.

Since I'm planning on taking 1 reading a minute for at least a week (that's 10080 readings), I'm not sure that I'm going to get close to running into this issue - but it's always good to be prepared I guess......

Found this interesting? Got a better way of doing it? Comment below!

Sources and Further Reading

• Entopy Library for Arduino
• Gnuplot
• Optipng
• Jpegoptim
• Optipng on wapm.io - I didn't even know that WebAssembly had a package manager. I definitely need to investigate that now.....
• LoRa library that has a random byte generation feature based on untuned radio input

Summer Project Part 1: LoRaWAN Signal Mapping!

What? A new series (hopefully)! My final project for my Masters in Science course at University is taking place this summer, and on the suggestion of Rob Miles I'll be blogging about it along the way.

In this first post, I'd like to talk a little bit about the project I've picked and my initial thoughts.

As you have probably guessed from the title of this post, the project I've picked is on mapping LoRaWAN signal coverage. I'm specifically going to look at that of The Things Network, a public LoRaWAN network. I've actually posted about LoRa before, so I'd recommend you go back and read that post first before continuing with this one.

The plan I've drawn up so far is to build an Internet of Things device with an Arduino and an RFM95 (a LoRa modem chip) to collect a bunch of data, which I'll then push through some sort of AI to fill in the gaps.

The University have been kind enough to fund some of the parts I'll need, so I've managed to obtain some of them already. This mainly includes:

• Some of the power management circuitry
• An Arduino Uno
• A bunch of wires
• A breadboard
• A 9V battery holder (though I suspect I'll need a different kind of battery that can be recharged)
• Some switches

(Above: The parts that I've collected already. I've still got a bunch of parts to go though.)

I've ordered the more specialised parts through my University, and they should be arriving soon:

• TPL-5111
• microSD card breakout
• microSD card
• GPS receiver
• Dragino LoRaWAN Shield

I'll also need a project box to keep it all in if I can't gain access to the University's 3D printers, but I'll tackle that later.

I'll store on a local microSD card for each message a random id and the location a message was sent. I'll transmit the location and the unique id via LoRaWAN, and the server will store it - along with the received signal strength from the different gateways that received the message.

Once a run is complete, I'll process the data and pair the local readings from the microSD card up with the ones the database has stored, so that we have readings from the 'block spots' where there isn't currently any coverage.

By using a unique random id instead of a timestamp, I can help preserve the privacy oft he person carrying the device. Of course, I can't actually ask anyone to carry the device around until I've received ethical approval from the University to do so. I've already submitted the form, and I'm waiting to hear back on that.

While I'm waiting, I'm starting to set up the backend application server. I've decided to write it in Node.js using SQLite to store the data, so that if I want to do multiple separate runs to compare coverage before and after a gateway is installed, I can do so easily by just moving on to a new SQLite database file.

In the next post, I might talk a little bit about how I'm planning on generating the random ids. I'd like to do some research into the built-in random() function and how ti compares to other unpredictable sources of randomness, such as comparing clocks.

AT24C64 EEPROM and the Arduino

For a project of mine I've bought a bunch of parts. One of those are a bunch of AT24C64 EEPROM chips - which are basically really small SD cards which, in this case, can store 64 KiB of data - even when the power is switched off, as you'd expect.

I ended up having a bit of trouble getting it to work though, as the Arduino IDE appears to have been abandoned and I don't think it's still in development. Still, it works well enough. Anyway, I thought I'd document my findings here for future reference, and to save you a bit of trouble if you find yourself in a similar situation!

The first issue I ran into was in trying to get the associated library to work. I kept getting errors like these:

sketch/eeprom.ino.cpp.o:(.text.loop+0x1c): undefined reference to `AT24CX::writeChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o:(.text.loop+0x20): undefined reference to `AT24CX::readChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o:(.text.loop+0x49): undefined reference to `AT24CX::writeChars(unsigned int, char*, int)'
sketch/eeprom.ino.cpp.o: In function `loop':

Strange. I thought I'd added the #include "lib/AT24Cx/AT24CX.h" to the top? Sure enough, I had. It turns out that the problem actually lay in the fact that I'd used a git submodule to bring in the AT24Cx library, such that the library was not located in the same folder as the .ino file - so the Arduino IDE, in all its wisdom, decided that including the library's .cpp files was hardly necessary O.o

The solution I found was to #include the .cpp explicitly like so:

#include "lib/AT24Cx/AT24CX.cpp"

The other issue I stumbled across was that it couldn't find my EEPROM chip via I2C. Even the demo I2C scanner couldn't find it! It turned out, after searching up a storm on DuckDuckGo, that I needed a pair of 1kΩ resistors stretching from the I2C pins tot he +5V power rail. Here's a diagram I created in Fritzing to show you what I mean:

(svg, fritzing file)

As usual with the various arduino test programs I find / write, you can get a hold of them in my main arduino repository on my personal git server.

Deep Sleep on ESP-Based Chips

If you're interested in the Arduino ecosystem, you've no doubt come across the Wemos family of boards. Based on the ESP8266, they have WiFi and TCP / UDP support built in! While that's very cool indeed for such a low-power device, in this post I'll be focusing on another cool aspect of the chipset, as I'm going to need to for a project in the nearish future (which I might blog about too!).

Curiously, the ESP chipset carries a unique ability to go into so-called 'deep sleep', which turns off everything but an internal counter, which emits a pulse on a specified pin when the sleep time is up. By wiring this specified pin to the RST (Reset) pin with a jumper cable, we can get it to automagically wake itself up after the deep sleep cycle is completed.

This is a lot simpler than the sleep modes available on other (non-ESP) chips - which are explained here for those interested. Here's an example:

void setup() {
    Serial.begin(9600);

    Serial.print("Initialising - ");

    pinMode(D0, WAKEUP_PULLUP);

    Serial.println("done");

    Serial.println("Waiting: ");
    for(int i = 0; i < 5; i++) {
        delay(1000);
        Serial.print(".");
    }
    Serial.println();

    Serial.println("Entering deep sleep. Goodnight!");
    ESP.deepSleep(5 * 1000000);
}

void loop() {

}

Nice and easy, right? You can see how you'd need to factor this into the design of your program before you get too far into it. Note that I multiply the number of seconds by 1000000 - this is because the sleep time is specified in microseconds - not milliseconds or seconds.

When in deep sleep, people have managed to reduce it's power consumption down to ~100µA(!) - I'll update this post once I manage to make some measurements of my own.

That's about everything I wanted to mention - just to remind myself on how to do it in a few weeks time :-)

Source and Further Reading

• Making the ESP8266 Low-Powered With Deep Sleep by Taron Foxworth
• Hush little microprocessor… AVR and Arduino sleep mode basics - Sleep mode for regular Arduino (ATMega-based) boards

Found this useful? Got a question? Comment below!

An (unscientific) Introduction to I2C

I've recently bought an LCD display for a project. Since I don't have many pins to play with, I ended up buying an I²C-driven display to cut the data pins down to just 2: One for outgoing messages, and one for receiving incoming messages from other devices.

It's taken me some time to get to grips with the idea of I²C, so I thought I'd write up what I've learnt so far here, along with some helpful hints if you run into problems yourself.

In effect, I²C is a wire protocol that allows multiple devices to talk to each other over a single pair of cables. Every I²C device has an 8 bit hardware address burned into it that it uses to address itself - much like the Internet Protocol when it comes to it, actually. Devices can send messages to one another using these addresses - though not all at the same time, obviously!

If you want to talk directly over I²C with a device, then Wire.h is the library you want to use. Normally though, devices will come with their own library that utilises Wire.h and communicates with it for you.

As a good first test to see if I²C is working, I found an I²C scanner that scans for connected devices. Since the address space is so limited, it doesn't take long at all:

/* --------------------------------------
 * i2c_scanner
 * 
 * Version 1
 *  This program (or code that looks like it)
 *  can be found in many places.
 *  For example on the Arduino.cc forum.
 *  The original author is not know.
 * Version 2, Juni 2012, Using Arduino 1.0.1
 *  Adapted to be as simple as possible by Arduino.cc user Krodal
 * Version 3, Feb 26  2013
 *  V3 by louarnold
 * Version 4, March 3, 2013, Using Arduino 1.0.3
 *  by Arduino.cc user Krodal.
 *  Changes by louarnold removed.
 *  Scanning addresses changed from 0...127 to 1...119,
 *  according to the i2c scanner by Nick Gammon
 * 
 * Version 5, March 28, 2013
 *  As version 4, but address scans now to 127.
 *  A sensor seems to use address 120.
 * Version 6, November 27, 2015.
 *  Added waiting for the Leonardo serial communication.
 * This sketch tests the standard 7-bit addresses
 * Devices with higher bit address might not be seen properly.
 */

#include <Wire.h>

void setup()
{
    Wire.begin();

    Serial.begin(9600);
    while (!Serial);             // Leonardo: wait for serial monitor
    Serial.println("\nI2C Scanner");
}

void loop()
{
    byte error, address;
    int nDevices;

    Serial.println("Scanning...");

    nDevices = 0;
    for(address = 1; address < 127; address++ )
    {
        // The i2c_scanner uses the return value of
        // the Write.endTransmission to see if
        // a device did acknowledge to the address.
        Wire.beginTransmission(address);
        error = Wire.endTransmission();

        if (error == 0)
        {
            Serial.print("I2C device found at address 0x");
            if (address<16)
                Serial.print("0");
            Serial.print(address,HEX);
            Serial.println("  !");

            nDevices++;
        }
        else if (error==4)
        {
            Serial.print("Unknown error at address 0x");
            if (address<16)
                Serial.print("0");
            Serial.println(address,HEX);
        }
    }
    if (nDevices == 0)
    Serial.println("No I2C devices found\n");
    else
    Serial.println("done\n");

    delay(5000);           // wait 5 seconds for next scan
}

As the initial comment mentions, I can't claim ownership of this code! I got it from here.

With the code in mind, it's time to look at the circuit design.

(Above: A simple I2C circuit. Credits go to openclipart.org for the images.)

The above connects an Arduino Uno version 3 with a simple LCD display via a breadboard to allow for expansion to connect future devices. The power (the red and blue cables) link the 5V and GND pins from the Arduino to the appropriate pins on the back of the LCD (the image of an LCD I found didn't have the pins showing :P), and the I²C pins (green and yellow) connect the SDA and SCL pins on the Arduino to the LCD display.

With the circuit done, that completes the system! All that remains now is to build something cool with the components we've put together :D

PixelBot Part 2: Devices need protocols, apparently

So there I was. I'd just got home, turned on my laptop, opened the Arduino IDE and Monodevelop, and then.... nothing. I knew I wanted my PixelBot to talk to the PixelHub I'd started writing, but I was confused as to how I could make it happen.

In this kind of situation, I realised that although I knew what I wanted them to do, I hadn't figured out the how. As it happens, when you're trying to get one (or more, in this case) different devices to talk to each other, there's something rather useful that helps them all to speak the same language: a protocol.

A protocol is a specification that defines the language that different devices use to talk to each other and exchange messages. Defining one before you start writing a networked program is probably a good idea - I find particularly helpful to write a specification for the protocol that the program(s) I'm writing, especially if their function(s) is/are complicated.

To this end, I've ended up spending a considerable amount of time drawing up the PixelHub Protocol - a specification document that defines how my PixelHub server is going to talk to a swarm of PixelBots. It might seem strange at first, but I decided on a (mostly) binary protocol.

Upon closer inspection though, (I hope) it makes a lot of sense. Since the Arduino is programmed using C++ as it has a limited amount of memory, it doesn't have any of the standard string manipulation function that you're used to in C♯. Since C++ is undoubtedly the harder of the 2 to write, I decided to make it easier to write the C++ rather than the C&sharp. Messages on the Arduino side are come in as a byte[] array, so (in theory) it should be easy to pick out certain known parts of the array and cast them into various different fundamental types.

With the specification written, the next step in my PixelBot journey is to actually implement it, which I'll be posting about in the next entry in this series!