Fullscreen HTML5Med Res (??MB)HTML4

Make it with Micromite Phil Boyce – hands on with the mighty PIC-powered, BASIC microcontroller Part 48: Long strings and sorting arrays F or those of you following the Smart-Light Controller (SLC) project, the title of this month’s article may not be as expected. The reality is, when setting out on a journey of building a conceptual project, there will likely be some unexpected challenges. Here, for the SLC, two things cropped up while developing the final version of the of the program code. Thankfully, both challenges were not too difficult to overcome, however, we thought that they were worth writing about in a short article as they could prove to be useful building blocks in other PicoMite projects that you set out to build. So, the two new topics we will learn about this month are long strings, and how to sort data that is stored in an array. Too much data Essentially, the PicoMite SLC communicates wirelessly to a Philips Hue hub to allow control of any of the smart lights. Most of the time, this means sending commands from the PicoMite to the Hub, for example to turn on a certain light by specifying the relevant light’s ID number. However, other commands may be used to fetch information from the Hub instead, such as a list of all available smart lights that the Hub knows about. As seen previously, a list of smart lights is obtained by sending a GET request to: https://hub_ip_address/clip/v2/ resource/light Depending on the size of your Hue setup, this could result in a lot of information being sent back to the PicoMite; for my own setup, this was around 16,000 bytes of data. Before we go further, it is worth giving a brief overview of the format of the data, and why there are so many bytes. Hue JSON data Whether data is being read from the Hub, or is being sent to the Hub, it uses 62 a format known as ‘JavaScript Object Notation’ (more commonly referred to as ‘JSON’). JSON is a common standard file format that uses human-readable text to transmit data. It comprises one or more ‘attribute’ & ‘value’ pairs (and/or arrays). An example of a JSON extract sent from the Hue Hub is shown in Listing 1. If you look carefully, you will see that essentially there is a parameter name followed by a parameter value for each piece of data that is sent (formatted with lots of brackets, commas, colons and speech marks). For example, the small sample shown in Listing 1 highlights the parameter name brightness, which has a parameter value of 75.59. Note that this is one of two pieces of data that belong to the dimming array (along with the min dim level), and this array is just part of all the data that relates to the smart light with ID: "5baa****-********-****-******8794b6" The Listing 1 extract is the ‘formatted’ output that is reasonably easy to read. However, if you consider the unformatted raw bytes that are sent by the hub (for just the first smart light) then the JSON Listing 1. ’Formatted’ JSON data sent from the Philips Hue Hub. "errors": [], "data": [ { "id": "5baa****-****-****-****-******8794b6", "id_v1": "/lights/*", "owner": { "rid": "b7206469-****-****-****-******5152a4", "rtype": "device" }, "metadata": { "name": "Outhouse Porch", "archetype": "spot_bulb" }, "on": { "on": false }, "dimming": { "brightness": 75.59, "min_dim_level": 0.20000000298023225 }, "dimming_delta": {}, Practical Electronics | August | 2023 Listing 2. ’Unformatted’ JSON data sent from the Philips Hue Hub. {"errors":[],"data":[{"id":"5baa****-****-****-****-******8794b6","id_v1": "/lights/6", "owner":{"rid":"b7206469-****-****-****-******5152a4","rtype": "device"},"metadata":{"name" :"Outhouse Porch","archetype":"spot_bulb"}, "on":{"on":false},"dimming":{"brightness":75.59 ,"min_dim_level": 0.20000000298023225},"dimming_delta":{},"color_temperature":{"mirek":286, "mirek_valid":true,"mirek_schema":{"mirek_minimum":153,"mirek_maximum":500}}, "color_temperature_delta":{},"color":{"xy":{"x":0.4074,"y":0.3929},"gamut":{"red":{"x":0.6915 ,"y":0.3083},"green":{"x":0.17,"y":0.7},"blue":{"x":0.1532,"y":0.0475}},"gamut_type":"C"}, "dynamics":{"status":"none","status_values":["none","dynamic_palette"],"speed":0.0, "speed_valid":false},"alert":{"action_values":["breathe"]}, "signaling":{"signal_values":["no_signal","on_off","on_off_color","alternating"]}, "mode":"normal","effects":{"status_values":["no_effect","candle","fire","prism"], "status":"no_effect","effect_values":["no_effect","candle","fire","prism"]}, "powerup":{"preset":"powerfail","configured":true,"on":{"mode":"previous"}, "dimming":{"mode":"previous"},"color":{"mode":"previous"}},"type":"light"} data will look something like the data in Listing 2. You can see from Listing 2 that where there are more than just a handful of smart lights in a setup, the amount of data that the Hub will output (in response to specific commands) can become very large, very quickly. Buffer-size In our SLC, when certain commands (ie, requests) are sent from the PicoMite to the Hub, it is the ESP32 Wi-Fi module that will wirelessly receive the JSON data (ie, response) from the Hub. This data will be temporarily stored in the ESP32’s internal buffer (something that doesn’t concern us), and simultaneously the ESP32 will send out this buffered data (one byte at a time) from its serial output pin (Tx). The PicoMite simply needs to have a serial COM port configured so that any data received from the ESP32 module on the PicoMite’s input pin (Rx) will automatically be stored in the PicoMite’s COM port buffer. Put simply, the PicoMite’s COM port buffer stores the data sent by the Hub (via the ESP32) and makes it available for processing by the program code. In theory, all we need to do is ensure that when we configure the COM port in MMBASIC, it is given a large enough buffer size to hold all the data. However, this buffer size will vary considerably and is ultimately dependent upon the number of smart lights in the setup, but for now, let’s go with the fact that the PicoMite has enough memory to allow us to configure a large enough COM port buffer. Normal practice is for the program code to then remove any data stored in the PicoMite’s COM port buffer and load it into a string variable so that the data can be processed. Processing JSON data The concept for processing the JSON data is relatively straightforward. Begin by copying the contents of the COM port buffer into a string variable (let’s call it Practical Electronics | August | 2023 Buff$). Next, look for the first occurrence of the required parameter name; this can be achieved by using the INSTR function. The associated parameter value will be located immediately after the parameter name (as can be seen in the above JSON extract) – hence the value can be extracted by the program code by using the MID$ function, ideally into an array. Once the parameter value has been extracted, delete all the characters in Buff$ up to the end point of the parameter value and then simply repeat the process multiple times for each piece of information (and for each smart light) until the end of the JSON data (ie, the end of Buff$) is reached. Note that this method requires all of the JSON data contained in the COM port buffer to first be copied into a string variable so that the program code can use MMBASIC’s various string manipulation commands, along with some neat tricks, to extract the required data. However, the maximum size (ie, length) of a string variable is limited to just 255 bytes (characters), and this is nowhere near big enough for what we need. So, our first challenge is how to store data in a string variable that allows for more than 255 bytes so that we can process it in the method outlined above. Thankfully, MMBASIC incorporates functionality for long strings, and as the name implies, this allows for many more characters to be stored. In fact, long strings are limited in size only by the available RAM in the PicoMite. (Use the MEMORY command to check how much RAM is available). Therefore, a single long string is perfect for our application here as it will be able to store the complete JSON message that is received into the PicoMite’s COM port buffer. Before we continue, there are a few things that we first need to understand when using long strings. MMBASIC long strings The ability to store a lot more than 255 characters in a long string has many practical uses, all of which allow the contents of the string to be manipulated. In our SLC we will use a single long string to store the complete JSON response from the Hue Hub so that the required information about all the smart lights can be extracted (name, ID, status and brightness). In a similar manner, a long string could be used to store serial data sent from other hardware devices such as a GPS receiver. In this case, we could store all of the NMEA sentences that a GPS module outputs, and then relevant data, such as longitude and latitude could be extracted by using the same technique that we will be describing shortly. So, as you can see, this is valuable technique to master. However, be aware that we can’t use the usual string manipulation commands that we are familiar with, such as LEFT$, RIGHT$ and MID$ as these only work on standard string variables. In their place, MMBASIC has an extensive set of dedicated commands and functions used purely with long strings. Declaring a long string Before we can use a long string, we must first declare it, in other words, the equivalent of DIM Buff$ which would be used for a standard string declaration. Warning! The next couple of sentences may sound confusing, but just go with it for now; the example given below will help you make sense of it. A variable for holding a long string must be defined as an integer array (even though it will be storing string data). The array must be single-dimensional, with the number of elements set to the number of characters required for the maximum string length divided by eight. The reason for dividing by eight is that an integer occupies 8 bytes (but we are not storing integers, we are storing characters). Basically, this is the way that the firmware needs to handle the data internally, and it is used to reserve memory for holding the contents of the long string. 63 Let’s use an example to help clarify this. For a standard string (let’s stick with calling it Buff$), we would make the declaration by using: DIM Buff$ or DIM STRING Buff However, to declare the equivalent as a long string (with the ability to store a maximum string length of 22,000 characters) we would simply use: DIM INTEGER Buff(22000/8) Using this declaration will create an empty (zero length) long string, with the name Buff. Now that we know how to declare a long string, let’s look at the dedicated long string commands that are available for us to use. Long string commands All long string commands are easy to spot because they begin with LONGSTRING and are then followed by a specific word, which in most cases is self-explanatory. Rather than going into the specific details of each command, below is a table of all the LONGSTRING commands (along with the required parameters, and a brief description). Further details can be found in the PicoMite User Manual. Note that those shown in bold are ones that we will be using in our program-code. Long string functions In addition to the 15 long string commands there are four long string functions, also shown in the table below. It must be stressed here that whenever a long string variable is used with one of the dedicated long string commands or functions, it should be sent as the name of the long string, followed by empty brackets. For example, to empty the long string named Buff, you would use the long string CLEAR command as follows: Extracting the data To demonstrate how to extract information from JSON data, we will use the first three lines of the extract shown in Listing 2. Therefore, for the purpose of this exercise, let’s assume that the content of the long string Buff() is as follows (ie, an incomplete JSON message): {"errors":[],"data":[{"id":"5baa****-********-****-******8794b6","id_v1": "/lights/ 6","owner":{"rid":"b7206469-****-****-**********5152a4","rtype": "device"},"metadata": {"name":"Outhouse Porch","archetype": "spot_bulb"}, "on":{"on":false},"dimming": {"brightness":75.59,"min_dim_level": Let’s extract just four bits of data: the ID value of the smart light, along with its name, its on-status (true or false), and finally its brightness value. First, we will use the INSTR function to search for the first required parameter name (in this case, id) along with any characters either side that are fixed. In other words, we will search for the following characters: {"id":" Note this is highlighted in red in the three-line extract. However, because this search string contains speech mark characters (as highlighted in purple), we must define the search string incorporating CHR$(34) in place of any speech mark (note: CHR$(34) represents the ASCII character for a speech mark). Hence, we need to search for the string: "{" + CHR$(34) + "id" + CHR$(34) + ":" + CHR$(34) To get the position of the first occurrence within the long string Buff() where this search string occurs, we will use the following long string function, and place the answer in a variable that we will call j_ID LONGSTRING CLEAR Buff() j_ID = LINSTR( Buff(), "{" + chr$(34) + "id" + chr$(34) + ":" +chr$(34), 1) Now that we know how to declare a long string, and also what commands (and functions) are available to manipulate a long string, let us now discuss how we are going to extract the relevant information from the JSON data received from the Hue Hub. Hence, j_ID will contain the value 22, which is the character position of the first character highlighted in red. We can now use the TRIM command to remove all the characters from the start of Buff() up to the end of the Long string commands LONGSTRING APPEND array%(),string$ LONGSTRING CLEAR array%() LONGSTRING COPY dest%(),src%() LONGSTRING CONCAT dest%(),src%() LONGSTRING LCASE array%() LONGSTRING LEFT dest%(),src%(),nbr LONGSTRING LOAD array%(),nbr,string$ LONGSTRING MID dest%(),src%(),start,nbr LONGSTRING PRINT [#n,]src%() LONGSTRING REPLACE array%(),string$,start LONGSTRING RESIZE addr%(),nbr LONGSTRING RIGHT dest%(),src%(),nbr LONGSTRING SETBYTE addr%(),nbr,data LONGSTRING TRIM array%(),nbr LONGSTRING UCASE array%() Append an ordinary string to a long string. Empty a long string (set length to zero). Copy a long string. Concatenate two long strings. Convert a long string to lowercase. Get the left nbr characters from a long string. Copy characters to a long string. Get characters from the middle of a long string. Print a long string. Replace characters in a long string. Set the length of a long string. Get the right nbr characters from a long string. Set a byte in a long string. Trim nbr characters from the left of a long string. Convert a long string to uppercase. Long string functions r = LGETBYTE(array%(),n) r$ = LGETSTR$(array%(),start,length) r = LINSTR(array%(),search$[,start]) r = LLEN(array%()) Returns the value of a byte in a long string Returns part of a long string as a normal string Returns the position of a string in a long string Returns the length of a long string 64 Practical Electronics | August | 2023 search string itself. This will result in the long string Buff() containing the following characters: 5baa****-****-****-****-******8794b6","id_v1": "/ lights/6","owner":{"rid":"b7206469-****-****-**********5152a4","rtype": "device"},"metadata": {"name":"Outhouse Porch","archetype":"spot_bulb"}, "on":{"on":false},"dimming":{"brightness":75.59, "min_dim_level": To do this, we use the following command: LONGSTRING TRIM Buff(),j_ID+6 Note that here we are using the value stored in j_ID (from the first step) and adding ‘6’ to it to ensure that we are removing the correct number of characters from the start of Buff() – in this case, 28 characters need to be trimmed. More importantly, the parameter value (ie, data) that we need to extract is now right at the start of Buff(). To be able to extract the data (in this case the ID of the first smart light), we will need to know the length of it first – or in other words, work out the position of the speech mark character highlighted in red, and subtract ‘1’ from that value to get the length of the data. Hence, we will first need to use the LINSTR function to get the position of the highlighted speech mark and subtract ‘1’ from the answer, and store the result in the variable j_end. This is achieved using the following: j_end = LINSTR(Buff(), chr$(34), 1) – 1 Finally, we can use the LGETSTR$ function to extract the required number of characters (the length of data as stored in j_end – in this example the length is 36 characters) and store the result in a standard string; in this case L_ID$. The following shows how to do this: L_ID$ = LGETSTR$(Buff(), 1, j_end) The above process can be repeated in turn for each piece of data you wish to extract. Essentially you are repeatedly performing the following steps: 1. Search for the relevant parameter name, including any ‘fixed’ characters up to the start of the parameter value, by using the LINSTR function. 2. Remove all the leading characters by using the LONGSTRING TRIM command. 3. Search for the character immediately after the required data to be extracted by using LINSTR to determine the data length. 4. Use the data length from step 3 to extract the required data by using the LGETSTR$ function. The above is the concept for extracting information from the JSON data. In the final version of our program code, we use a loop (with a counter) to extract the required data into various arrays. So, rather than L_ID$ as used above, we will be using L_ID$(counter). This will allow us to more easily reference each smart light – more on this next month. Sorting Data in an array In the final version of the SLC code, we will be displaying a list of all available smart lights so that the user can select which one to control. To make it easier to find a specific smart light, it makes sense to display the list of their names sorted alphabetically. To achieve this, we will now discuss the MMBASIC SORT command. Practical Electronics | August | 2023 Let’s assume that we have all the smart light names extracted from the JSON data stored unsorted in an array named L_Name$(). For this example, let’s use the following sequence of names (taken from my Hue setup): L_Name$(1)="Outhouse Porch" L_Name$(2)="Dining Room Light" L_Name$(3)="Dressing Table Striplight" L_Name$(4)="Hut Light" L_Name$(5)="Water Feature" L_Name$(6)="Phil’s Lamp" L_Name$(7)="Globe Light" L_Name$(8)="Girl’s Room" L_Name$(9)="Jenson’s Light" L_Name$(10)="Flood Light 2" Rather than re-sequence this list back into the L_Name$() array, we will use the SORT command to generate a list of sorted ‘index’ numbers, and store this information in a new array that we will call L_SortIndex%() For this example, we will also store the sorted list of names in a new L_Temp$() array too. First, we need to declare the two new arrays with: DIM L_SortIndex%(10) and DIM L_Temp$(10) Note that they must both be exactly the same size as the L_Name$() array that we wish to sort (in the above example the L_Name$() array has 10 elements). Next, we need to copy the contents of the L_Name$() array into the L_Temp$() array as follows: FOR i = 1 to 10 L_Temp$(i)=L_Name$(i) NEXT i Finally, to sort the data, we use the SORT command in the following manner: SORT L_Temp$(),L_SortIndex%() This will result in the content of each array (shown below just for element 1) as follows: L_Name$(1) = "Outhouse Porch" (unchanged original name) L_Temp$(1) = "Dining Room Light" (sorted alphabetical name) L_SortIndex%(1) = 2 (index number from L _ N a m e $ ( ) array that is first name alphabetically) For completeness, the contents of the L_SortIndex%() array will be as follows: L_SortIndex%(1) = 2 L_SortIndex%(2) = 3 L_SortIndex%(3) = 10 L_SortIndex%(4) = 8 L_SortIndex%(5) = 7 L_SortIndex%(6) = 4 L_SortIndex%(7) = 9 L_SortIndex%(8) = 1 L_SortIndex%(9) = 6 L_SortIndex%(10) = 5 65 As a quick exercise, check the mapping of the values stored above from the L_SortIndex%() array, and look up the corresponding element from the L_Name$() array; and you will see that the names are sorted alphabetically. To display the sorted list of smart lights in code, we have two options. The more obvious option is to use the following: FOR i = 1 to 10 PRINT L_Temp$(i) Next i However, we will use the following option instead: FOR i = 1 to 10 k=L_SortIndex%(i) PRINT L_Name$(k) Next i Both versions display the same result, but the second version allows other data stored in other arrays such as L_Status$(), L_Brightness$() and L_ID$() to be referenced correctly by using the index value stored in k (obtained from the L_SortIndex%() array) without having to re-sequence any of these other arrays. Don’t worry if this is getting difficult to follow, just be mindful that to display the sorted list of smart light names, we will use the second option shown above. Demo code displaying the smart lights in the same sequence that they are contained in the To demonstrate all of the techniques that we JSON file; and the second list is the same have been discussing this month, we have information but sorted alphabetically by created a demo program – ExtractJSON. smart light name. txt – you can download this code from the In addition, the sorted list of smart light August 2023 page of the PE website at: names is displayed on the touchscreen https://bit.ly/pe-downloads (this will get tidied up next month!). Load it into your PicoMite, but before Do be sure to have a look at the running it, ensure that you edit the first program code to see if you can few lines of uncommented code so that understand how it works. You should it can connect to your wireless network, be able to see how the topics covered and also set your Hub’s IP address, and this month have been used to generate your unique Hub-API key. the two lists of smart lights displayed Once you have set these required in your terminal screen. parameters, R U N the program and observe the information displayed on your terminal screen. Next Time After a short delay, you should see Now that our program code can a list of all your Hue smart lights (and successfully extract a list of smart lights any smart sockets) – see Fig 1. For each from the Hue Hub (along with the unique smart light that the Hue Hub knows ID number that we will need to be able about, it will display the unique smart to control a specific smart light), we are light ID, its current status and if the able to finish the software for the SLC. status is ‘ON’. Then it will display the Next month, we will discuss how the current brightness as a percentage (but software implements the features of only for dimmable smart lights). Finally, selecting a specific light from the list the name of the smart light (or smart of all available smart lights, and how it socket) is also shown (as configured in then then controls it. the Hue App). Until then, stay safe, and have FUN! Note that the list of smart lights (and Micromite code any smart sockets) The code in this article is available for download from is shown twice. The the August 2023 page of the PE website. first list (at the top) is Fig 1. This month’s demo program (ExtractJSON.txt) extracts data from the Hue Hub and displays it in two lists. The first list is unsorted, but the second list is sorted alphabetically by the smart light name. (Note that the ID numbers have been obscured for security reasons!) 66 Practical Electronics | August | 2023