Fullscreen HTML5Med Res (??MB)HTML4

Build an Arduino Data Logger with GPS Part 2 by Nicholas Vinen As promised, here is a follow-up to the Arduino Data Logger article from the last issue, with more details about how its software operates. We will also take you through the steps required to add support for new sensors and show some photos of the completed shield PCB. PCB W hile the bulk of this article details the operation of the critical Arduino software for the data logger, we also have some important information on building the shield PCB, shown in the photos. Upon building the PCB, we discovered an error in the overlay diagram, Fig.2, on page 30 of the August 2017 issue. The pins for the DS3231 real time clock and calendar module were labelled incorrectly. The revised diagram, shown here, gives the correct labelling. The PCBs that we supply from our Online Shop will have the correct labelling. Regarding mounting the DS3231 module, our module came fitted with a 6-pin right-angle header. We straightened this with a pair of pliers and then soldered a 4-pin straight header at the opposite end. This module can then be soldered directly to the PCB, as shown in the photos. Make sure to trim the pins so that they can’t short against anything on the Arduino board below. The advantage of this approach is that you don’t need to use any screws or spacers to retain it on the board. As for the microSD card module, we used four M2 machine screws and nuts to hold it on the board, along 86  Silicon Chip with short untapped spacers. You could use Nylon nuts or washers as spacers. These parts were not listed in the Parts List last month (see Extra Parts on page 90). It needs to be pretty close to the board if you’re using a socket to make the electrical connections, as we did, or else the socket pins will not reach the pads on the board. The remaining construction details were in the article last month. So refer to that article to complete the shield. Now let’s move on to the Arduino software sketch details. Software description The SdFat and SPI libraries are used to read and write data on the microSD card, which is formatted with either FAT16 or FAT32. RTClib is used to control the DS3231 real-time clock and calendar module. The TinyGPS library is used to decode data from the optional GPS unit, although the software serial interface used to receive data from it has been customised, as explained below. We also use the OneWire library to communicate with a DS18B20 temperature sensor, if present, and the MsTimer2 library to manipulate hardware Timer2 if we have set up any of the digital inputs to measure frequency. We use Timer2 to provide the normally one second gating period to count pulses on the relevant pin, because Timer2 can be left running in one of the sleep modes. This mode uses more power than the normal sleep mode, so we only use it when the frequency counting feature is active. We also have some custom routines to put the ATmega328P microcontroller on the Arduino into sleep mode and wake it up when required. The setup() routine is run at power-up time and first sets up the input and output pins. It then checks that the real-time clock module is present and whether it has the current time. If not, it sets the clock time to be the time that the sketch was compiled, plus 20 seconds (to allow for the approximate time required to compile and upload the sketch). It then stores the new time in EEPROM. If the Arduino is reset within one minute, as determined by comparing the clock time to that stored in EEPROM to the RTCC, the time is advanced to the next whole minute. Thus, if you programmed the chip at say 11:37:25, and reset it at exactly 11:38:00, the clock would have the correct time, accurate to the second. The setup routine then initialises the microSD card and assuming that’s siliconchip.com.au siliconchip.com.au The finished Arduino Datalogger without the Elecrow charger module, Li-ion cell and solar panel connected. Compared to building it with the prototyping shield, it's much neater and easier to solder. successful, the main loop starts and runs as long as there is 5V power available. If either the real-time clock or microSD card initialisation fails, LED1 flashes in an endless loop to alert the user. It flashes at 2Hz for a real-time clock fault and 4Hz for an SD card fault. The main loop The main loop() function first checks for the presence of a GPS module, if one has not already been detected. In the absence of a GPS unit, pin D8 is always high. If D8 is found to be low, the software serial port is set up to receive data from this pin at 9600 baud, with TTL signal levels. This serial port is then monitored for ten seconds, looking for the string “$GPRMC,” which is part of the standard NMEA data stream. If during those ten seconds this string is identified, a flag is set in the software indicating that a GPS module is present. Otherwise, the serial port is closed down and the low level on D8 is assumed to be from electrical noise. If a GPS module is determined to be present, the unit will then wait for the programmed period for a position lock (defaulting to five minutes). If a lock siliconchip.com.au occurs during this time, the latitude and longitude (along with the number of satellites in view and the current time) are stored for future reference and pin D7 is driven low, to switch off the GPS module and conserve power. It is only brought high again once the GPS details need updating, which by default is once per hour. If a lock does not occur during this time, the location data is not updated but the GPS module will still be powered down for an hour, at which point it will try again. If the unit had GPS lock previously, those co-ordinates will be preserved. Otherwise, they will be kept blank. While it is doing all this, the normal data logging tasks are still going on, as we don’t want to lose any data just because the GPS module is active. Data logging Each time through the loop, the unit checks the state of D9. If D9 is high (it is pulled high by default), and the unit has not received a message on its serial interface to pause logging, it will then check whether the configured logging interval has passed. If it has, the states of the analog and digital inputs are queried and stored in a RAM buffer, along with the data The revised PCB overlay diagram from last month. The difference is that every connection on the DS3231 RTC was flipped horizontally, eg, GND ↔ SCL, VCC ↔ SQW, etc. September 2017  87 Top and bottom views of the assembled shield board. The right-angle polarised connector at lower left (on the top view) is for the four analog inputs plus ground while the digital inputs are on the 5-pin header to its right. Note the real-time clock module is mounted so the cell is accessible. from any extra sensors that the logger is configured to query. Once this RAM buffer is full, typically after about a minute, the microSD card is brought out of sleep mode and the values are converted into humanreadable format and appended to the log file. While this is happening, LED1 is lit. As a result, is flashes very briefly about once per minute, to indicate that logging is occurring. If pin D9 has been driven low, or a pause command is received on the serial console, the log file on the microSD card is closed and the unit will go into sleep mode to conserve power until it is told to continue logging. The next time there’s logged data to be written to the microSD card, a new file will be created with the name containing the date and time of the first log entry and the entries will subsequently be written to that file. Once all the logging tasks have been completed, the software checks the main USB serial console to see if any data has been received. If it has, it compares it against a list of commands and if a valid command has been received, it processes it. There are four commands: “stop”, “cont”, “list” and “dump”. They are terminated with a newline (enter/return). “stop” pauses logging and “cont” resumes it; pausing is equivalent to pulling pin D9 low, so when a “stop” command is received, any buffered data is 88  Silicon Chip written to the microSD card and it can then be removed. When it’s replaced, the “cont” command will then cause the log file to be re-opened (assuming D9 is not held low). The “list” and “dump” commands can only be used when the log file is closed, so will normally be preceded by a “stop” command. “list” displays a list of all the log files on the microSD card over the serial console. Dump then allows one of them to be downloaded through the serial console. The log file name to be written must be sent immediately after the dump command, for example, “dump ArduinoLog_2017-06-28_094837.log”. Sleep mode When the unit is not doing any logging, handling any serial commands and the GPS unit is not powered up, it will go into sleep mode to conserve power. We couldn’t find a suitable Arduino library to perform this sleep function so we wrote the SleepMilliseconds() function ourselves. This will put the chip into sleep mode for a period between 16ms and eight seconds, using the low-power watchdog time to wake it up. During this time, the ATmega328P consumes well under 1mA. However, other circuitry on the Arduino board (eg, regulators) brings the total up to around 8mA. Still, this is a much lower power consumption than when it is active and allows for a decent battery life. One of the tricks we’ve employed is that we temporarily disable the Arduino’s hardware UART which provides the main USB serial port when entering sleep mode, so that we can enable a Pin Change Interrupt on pin D0, the RXD pin for that serial port. This means that the chip will automatically wake up if the state of that pin changes, which occurs whenever there’s any serial data being transmitted to the unit. Hence, the unit can be in low-power sleep mode but still respond to commands on the serial port. We ran into one slight problem with the Arduino SdFat library which is that the first time you open a file on the SD card, the card’s current consumption jumps from under 1mA to around 15mA and even if you close the file, it will continue to operate at the higher power level. We solved this by closing the file after each write and resetting the SD card interface, via a call to the sd.begin() function. We then re-open the file later and append data as required. This means its power consumption is back under 1mA all the time, except when we are actively writing to it. Apparently this is a well-known and longstanding bug in the Arduino version of the SdFat library and it’s mystifying that it has never been fixed. GPS serial interface The Arduino Uno only has one hardware serial port which is hooked up to its USB port (via a second Atmel chip on the Uno board). We wanted to keep this for communications with a PC, so that logged data could be off-loaded without removing the microSD card (although in some cases, removing the card would be easier/faster). That means that the serial data from the GPS unit must be received using a “software serial port”, where the RXD pin is just a normal digital input (with internal pull-up enabled) and software routines count the time between state changes on that pin to decode the serial data. The Arduino IDE comes with a popular library called SoftwareSerial to do just that but we discovered in writing this software that it has serious limitations. Basically, the problem is that it’s a “blocking” type library, where the siliconchip.com.au CPU is 100% busy during the time that serial data is being received. Since a GPS unit sends out quite a large burst of data each second, of several hundred bytes, without a huge RAM buffer, the buffer would always overflow. That’s because while the CPU is busy receiving serial data, it has no time left to actually process it. There’s another problem with SoftwareSerial which is that it assumes that if you’re allocating a pin to receive serial data, you also want to allocate a second output pin to send serial data. We don’t need to send any data to the GPS unit and we don’t have any spare pins. We found a library called AltSoftSerial which solves the first problem. It uses a piece of hardware in the Atmel chip known as an “input compare unit” which, in combination with a hardware timer, effectively provides time stamps indicating when the state of a pin has changed. This allows the processor to continue running other code while it waits for transitions on the serial input pin and since the library is interruptbased, it provides a software serial port that works almost as well as the hardware port (at the low 9600 baud rate we’re using, anyway). Its major limitation is that the input compare unit is hooked up to pin D8, so you must use this as the RXD pin (and therefore you can only have a single AltSoftSerial port). Similarly, it uses “output compare” hardware to produce the TXD signals in an asynchronous manner, which means the TXD pin must be on D9. In a stroke of luck, it just so happened that we had hooked up the GPS TXD pin to D8 on our prototype, and its RXD pin to D9, so we could use AltSoftSerial without having to make any hardware changes. However, when we subsequently decided to add S1 to the design, we found that AltSoftSerial also forced you to use the transmit and receive functions together. So we made a copy of the library, renamed it ReceiveOnlyAltSoftSerial and deleted the sections which enable transmission. That library is provided along with our sketch. Adding new sensors One of the major advantages of this data logger over our previous projects is that it’s quite easy to customise. While we provided it with a wide range of standard features, we haven’t tried to account for every possible sensor that you might want to attach. For example, you may want to log data from an I2C barometric pressure or humidity sensor. Since it’s written using the Arduino IDE and already has built-in I2C support, you just can download some example code for the sensors you want to use, check that the example sketch works and then integrated it into the data logger code. This does require some programming experience but there’s a lot of information available on the internet on programming Arduino. One minor issue to consider is the amount of free flash memory space. With all the features enabled in our code, it uses 96% of the total flash memory (30,978 bytes out of 32,256 bytes). However, if you don’t need the DS18B20 or frequency counter support, that immediately drops to 84% (27,394 bytes). Disabling serial debugging (by removing the #define SERIAL_DEBUG line near the top of the file) drops this further, to 81% or 26,406 bytes. We realise that modifying the software can seem daunting, so we'll give a concrete example showing you the modifications to make to interface a GY-68 I2C barometric pressure sensor to the unit (and we will be offering this sensor in our online shop in case you want to give it a go; Cat SC4343). This sensor will be described in some detail in a future “El Cheapo Modules” article. It contains a BMP180 sensor and has a 4-pin SIL header with the pins labelled VIN, GND, SCL and SDA. Wiring it up to the Arduino is easy; we just used four male-to-female jumper leads to connect these pins to 5V, GND, A5 and A4 respectively. We then downloaded the sample Arduino code for this module and discovered it uses an I2C address of 0x77. The sample code contains a number of helper function to interface with the sensor. The first step to integrating this with our Data Logger code is to remove the line near the top of the file which reads “#define DS18B20_INPUT 2”. We don’t need the DS18B20 temperature sensor features since the GY-68/ BMP180 has an onboard temperature sensor. This frees up some flash memory, giving us 10% free. We then copied and pasted the entire GY-68 sample code into the bottom of the Data Logger sketch but deleted the setup() and loop() functions (as these would conflict with those used by the Data Logger). The sample code does two things in its setup() function: sets up the serial port, then calls the function “bmp085Calibration”. So our next step was to add a call to this function at the bottom of our setup() routine. The end of the setup() function now looks like this: bmp085Calibration(); From the side you can see that due to the depth of the screw head that the PCB doesn't fit entirely flat relative to the Arduino board. If this is an issue for you, simply omit the screw in that corner; as it will still have three others to support it. siliconchip.com.au #ifdef SERIAL_DEBUG Serial.println(F(“SILICON CHIP Arduino Datalogger ready”)); #endif September 2017  89 Extra Parts for the Arduino Datalogger Used for mounting the microSD module to the shield PCB 4 M2 x 10mm machine screws 4 M2 hex nuts 4 short (~4mm) tapped or untapped spacers to suit M2 screws OR 4 M3 Nylon nuts OR 8 M2/M3 Nylon washers, 1mm thick Looking at the loop() function in the sample code, the following four lines at the top are responsible for reading data from the sensor: // MUST be called first float temperature = bmp085GetTemperature (bmp085ReadUT()); float pressure = bmp085GetPressure (bmp085ReadUP()); // “standard atmosphere” float atm = pressure / 101325; // Uncompensated calculation // - in metres float altitude = calcAltitude(pressure); Note that there is a bug in this code; the third float variable should be set to: float atm = pressure / 101325.0; Otherwise, it will round the atmospheric pressure to the nearest bar (ie, it will pretty much always be 1.0)! Anyway, having looked at this code, we need to create some RAM buffers for storing these values before we can log them. Towards the top of the Data Logger code, at the end of the section labelled “// Other stuff”, we add the following line to do this: float BMP180buf [LOG_RAM_ENTRIES][2]; This gives us two floating point values per log entry to store the pressure and temperature data. So now, we modify the end of the function “write_ RAM_log_entry” to look like this: // in degrees Celcius BMP180buf[log_ram_filled][0] = bmp085GetTemperature (bmp085ReadUT()); // in bar BMP180buf[log_ram_filled][1] = bmp085GetPressure (bmp085ReadUP()) / 101325.0; ++log_ram_filled; 90  Silicon Chip Now we just need to modify the “write_buffered_log_entries” functions so that the temperature and pressure values are written to the log file. First, we modify the CSV header, so that the line which used to look like this: if( !file.println(F(“Date,Time,VA0, VA1,VA2,VA3, D0,D1,D2,D3, Lat,Lon,NumSats, SecondsSinceLock”)) ) Now looks like this: if( !file.println(F(“Date,Time,VA0, VA1,VA2,VA3, D0,D1,D2,D3, Temp,Pres,Lat,Lon,NumSats, SecondsSinceLock”)) ) We also need to modify this section: #else static const char LogEntryTemplate[] PROGMEM = “%02d/%02d/%04d,%02d:%02d: %02d,%d.%02d,%d.%02d,%d. %02d,%d.%02d,%d,%d,%d,%d”; #endif That’s rather hard to understand but basically, it just defines the format of each number that’s stored in a log entry in the CSV file. We need to add two, both with decimal points, at the end (GPS data is not included in this line). After adding these, the new line looks like: static const char LogEntryTemplate[] PROGMEM = “%02d/%02d/%04d,%02d:%02d: %02d,%d.%02d,%d.%02d, %d. %02d,%d.%02d,%d,%d,%d,%d, %d.%01d,%d.%03d”; Note that we have set it up to log the temperature with one decimal place (%01d) and pressure with three decimal places (%03d). Next, we need to make sure that the RAM buffer used to temporarily store the log lines before writing to the SD card is large enough, so change this section: #ifdef COUNTER_INPUT char buf[56+38]; #else char buf[56+30]; #endif to: #ifdef COUNTER_INPUT char buf[72+38]; #else char buf[72+30]; #endif Now all that’s left is to add the code to actually write the temperature and pressure data to the log file. Just after the line which reads: // add any extra logged data here We insert the following: ,(int)BMP180buf [log_ram_filled-1][0] ,(int)((int)(BMP180buf [log_ram_filled-1][0]*10))%10 ,(int)BMP180buf [log_ram_filled-1][1] ,(int)((int)(BMP180buf [log_ram_filled-1] [1]*1000))%1000 This is a bit complex because unfortunately, the Arduino sprintf() function (used for converting numbers into text) does not support floating point numbers. So what we do is first print the integral portion of each value, then the digits after the decimal point; one for temperature and three for pressure. Running the Verify/Compile command from the Sketch menu then gives us the following output at the bottom of the screen: Sketch uses 30,608 bytes (94%) of program storage space. Maximum is 32,256 bytes. Global variables use 1,470 bytes (71%) of dynamic memory, leaving 578 bytes for local variables. Maximum is 2,048 bytes. So all the extra code for the BMP180 pressure/temperature sensor takes just 4% of the flash memory space and leaves plenty of RAM free, despite the extra buffering. Uploading this new code to our prototype gives the following log output: siliconchip.com.au Customising the software You can simply download the software and then upload it to an Arduino Uno to get started with the data logger. However, since each logging application is different, we went to some effort to make the software easily customisable. The top of the sketch looks like this: #define LOG_INTERVAL_SECONDS #define VRAIL_5 #define A0_DIV_RATIO #define A1_DIV_RATIO #define A2_DIV_RATIO #define A3_DIV_RATIO //#define DS18B20_INPUT //#define COUNTER_INPUT #define COUNTER_AVG_MS #define LOG_RAM_ENTRIES #define GPS_TIMEOUT #define GPS_CHECK_INTERVAL #define SERIAL_DEBUG 6 5.000 (100.0/47.0) (100.0/47.0) (100.0/47.0) (100.0/47.0) 2 3 1000 6 (60*5) // 5 minutes (60*30) // half an hour You can change the first line to vary the logging interval, in the range of 1-60 seconds. The second line should be altered to provide maximum accuracy for the analog inputs. Simply power up the data logger with your preferred power supply and measure the voltage between the 5V and GND pins. Change the VRAIL_5 value to this figure and (re-)upload the sketch. Note that if you’re using the solar option, it’s best to make this measurement while the unit is running off battery power since this will be the normal condition and it’s likely to result in a different measurement than when USB/solar power is connected, as this will bypass the power supply regulator. The next four lines, Ax_DIV_RATIO, allow you to change the 100kW/47kW dividers for the four analog inputs to measure higher voltages. Simply increase the 100kW value or decrease the 47kW value to allow higher voltages to be measured, then alter the relevant lines in the software to compensate. If you don’t, you will get incorrect readings. Since the four values are defined separately, you can use different divider values for each analog input. The next two lines define which of the four digital inputs (#0-3) are used for a DS18B20 temperature sensor and as a frequency counting input. These features are disabled by default, to save power, so the four inputs operate as general purpose digital inputs. Remove the two slashes at the start of the line to enable that feature. Leaving these features disabled will also increase the amount of free flash memory. Note that if you are using a DS18B20, it must be connected directly to one of pins D2-D5 rather than via a 1kW resistor (or replace the relevant 1kW resistor with a wire link) and you also need to fit a 4.7kW pull-up resistor from 5V to that pin – see Fig.1 last month. If using the frequency counting feature, the maximum frequency is limited to roughly 10kHz and readings can be expected to be within a few percent of the actual frequency. The next line defines the number of log entries to buffer in RAM. A larger value reduces power consumption since the microSD card only needs to be powered up each time the buffer fills. In the default case, with a 6-second interval and 6-entry buffer, that’s once every 36 seconds. Basically, you probably don’t need to change this, but you can reduce the value to free up some RAM (to a minimum of one) and increase it if you’re confident that there’s enough free memory to do so. The next two lines define how often and for how long the GPS unit is powered up, if it is connected. By default, the unit will wait for a lock for a maximum of five minutes and it will power up the GPS module once per hour to get a fresh reading. You can increase the timeout value if your logger will be in a marginal signal area but this will increase power consumption for those times where it can’t get a lock. Similarly, you can reduce the check interval to update the GPS co-ordinates more often than once per hour but this will also come with a power consumption penalty. If the last line is removed, the unit will not print debugging messages on the serial console, other than log entries (as they are created). This reduces flash usage, as described in the text, making room for more code if required. Date,Time,VA0,VA1,VA2,VA3,D0,D1,D2,D3,Temp,Pres,Lat,Lon,NumSats,SecondsSinceLock 29/06/2017,12:57:04,0.00,0.00,0.00,0.00,1,1,1,0,20.8,1.002,33.760280,151.280291,6,25 29/06/2017,12:57:10,0.00,0.00,0.00,0.00,1,1,1,0,20.7,1.002,33.760280,151.280291,6,31 29/06/2017,12:57:16,0.00,0.00,0.00,0.00,1,1,1,0,20.7,1.002,33.760280,151.280291,6,37 29/06/2017,12:57:22,0.00,0.00,0.00,0.00,1,1,1,0,20.6,1.003,33.760280,151.280291,6,43 29/06/2017,12:57:28,0.00,0.00,0.00,0.00,1,1,1,0,20.7,1.003,33.760280,151.280291,6,49 29/06/2017,12:57:34,0.00,0.00,0.00,0.00,1,1,1,0,20.7,1.004,33.760280,151.280291,6,55 29/06/2017,12:57:40,0.00,0.00,0.00,0.00,1,1,1,0,20.8,1.004,33.760280,151.280291,6,61 siliconchip.com.au So those log entries show that the new sensor is working, giving us an indoor temperature reading of just over 20°C and a pressure of just over 1 bar. This modified sketch, titled Arduino_Data_Logger_Barometer. ino, is supplied in the download package. SC September 2017  91