MQTT - picaxe transactions?

geoff07

Senior Member
Having now nailed simple picaxe to Thingspeak comms I'm wondering if I could talk picaxe/ESP-01 to an mqtt server. Does anyone have any thoughts or experience of this? A protocol designed for satcoms telemetry ought to be simple enough for a picaxe, I would have thought.
 

hippy

Technical Support
Staff member
It should be possible, but probably not through AT commands. You would likely need to use some other language which can support MQTT and then code the part which interfaces between PICAXE and the internal library functions.

Not sure which language would be best but I would imagine something 'event driven' so you are basically just coding two functions; receiving from a PICAXE and sending to the MQTT broker, receiving an MQTT packet and passing that to the PICAXE. Though reality is likely to be a bit more complicated than that.

I would look for examples, particularly of pushing a button or reading a sensor and sending that out, receiving something back which lights a LED or updates a text LCD, which should provide the basic skeleton of what needs to be done, and give an indication of which language seems easier to use.
 

kranenborg

Senior Member
Hello Geoff, Hippy,

What a nice coincidence!
I have no experiences yet, but I do have kind of a plan for such an interface device (initial stage, but very happy to reveal here for cooperation), since I have been thinking - very much in line with Hippy - about an interface between one or more PICAXES and an mqtt-based network service. Two weeks ago I set up a simple mqtt-network in my house using an old Raspberry Pi as a mqtt-broker, and am currently investigating options to show/visualize data (from a DS18B20 sensor on an ESP8266) on a phone or PC . I like the mqtt protocol as a high-level solution for exchanging information using simple subscriptions to broadcast channels. The fact that you can send either simple data or more structured patterns (JSON) makes it all very flexible.

My suggestion would be to implement such an picaxe-mqtt interface device using MicroPython, but that is also motivated simply by the fact that I have started to dip my toes in learning MicroPython on ESP and PyBoard systems a few months ago. However, I can highly recommend micropython as a very expressive language with excellent support for internet services like mqtt (but it requires some processing power). In my view such a device (based on an ESP or Pyboard) could maybe have the following architecture:
  • On the PICAXE side:
    • A SERIN-compatible data line
    • A SEROUT-compatible data line
    • A digital signal line indicating that one (or more) messages are available from the mqtt service to the picaxe
    • A simple i2c interface to program the device for the serial protocol (for example defining the baudrate, and maybe other useful stuff). Alternatively a simple hardware interface (tying inputs high/low) for defining this is possible as well
  • In the interfacing device:
    • A MicroPython driver for the mqtt interface (see this link: Peter Hinch)
    • A LIFO-Stack between the mqtt-driver and the picaxe serial stream for allowing the picaxe to simply read or push messages in an appropriate sequence (and not to be overrun)
    • A predefined command sequence (also using SerOut) to allow the PICAXE to subscribe to one or more topics/broadcasts, and tying a number/indentifier to it for use in the picaxe itself
This is a bit of a brain dump, but maybe it gives some ideas. With such a device two very nice but wildly different products (PICAXE and MicroPython-Devices) are married to provide maximum functionality.

My goal in the end is to define a PICAXE SerialPower - MQTT bridge where processes/nodes on the picaxe network can subsribe to whatever mqtt-topic they like ... . (right at this moment I am writing the SerialPower II documentation for final release, therfore I was logged in and noticed this thread ...)

Regards,
Jurjen
 

geoff07

Senior Member
Thank you both. Indeed a coincidence. And I thought I was catching up with stuff that I had missed whilst preoccupied for a few years with sports coaching teenagers. Now stopped of course.

My first thought is something really simple. Can i not send mqtt commands as payload on an AT command to ESP01? Exactly as per the simple Thingspeak example (see my recent posting). Easy on a picaxe. That would get to an mqtt server on a local ubuntu server that I use, and then to a website on the server and then to e.g. an android client. How it gets used and where the website comes from is unclear. But if IoT is going to work it can't be complex.

My target ultimately is home controls. For example I currently have Conrad/elv soft heating zones. Very good but their kit is no longer available. So wifi-linked programmers in each zone, talking to the heating scada system (a 40x2), would be a replacement. More fun and way cheaper than Honeywell.
 

hippy

Technical Support
Staff member
My first thought is something really simple. Can i not send mqtt commands as payload on an AT command to ESP01? Exactly as per the simple Thingspeak example (see my recent posting). Easy on a picaxe.
Possibly. Getting data to the MQTT broker is pretty much the same as publishing to ThingSpeak; "This is my topic, this is its data". Simple as that.

But I don't know enough about MQTT to be able to say "and that's it". I imagine it might be for something which is just a publisher, but there may be some other house-keeping it has to do, there may be interactions required beyond just 'opening a URL'.

Most of the low-level stuff is hidden inside libraries so, for a computer, it's just -

result = mqtt.Publish( "My Beer Keg", ds18b20.ReadTemperature() )

But for a microcontroller one would need to implement what's in that library, what's in the libraries that uses. Basically unfurl the whole lot, then implement everything needed.

Hence why it's almost certainly better to have had someone else provide all that library code, write the program which does the above on an ESP or Pi, interfacing to a PICAXE which only has to issue SEROUT ( "My Beer Keg=", #w0, CR ).

For subscriptions and receiving data I really don't know, not even whether that data is pulled or pushed. It's the less likely use case for a PICAXE but I doubt one can simply ignore it. I'm sure it's easy with a library, something as simple as -

if mqtt.DataAvailable( "My Beer Keg" ): beerTemp = mqtt.ReadData( "My Beer Keg" )

But there will be a whole lot of stuff hidden within those libraries, all of which has to be implemented within a PICAXE for an AT type solution.

The latest MQTT 5.0 specification can be found here -

https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html

My suggestion would be to implement such an picaxe-mqtt interface device using MicroPython,
That's my inclination as well. A Pi or Linux PC is probably the best choice for implementing a local MQTT broker. Either can be used to get PC-based MQTT working locally, and both support Python and even MicroPython, both can interface to UART or AXE027.

So, get it working on a PC, and it should be easy to swap the PC for an ESP, PyBoard or other module running MicroPython using already proven code. Or just keep using the PC.

The advantage is that others can participate even without dedicated hardware. Problems can be more easily debugged on a PC.

But the big advantage is that others have done it all themselves. So there are plenty of examples and one only needs to worry about a few library calls and how they should be handled, not have to worry about how any of it works behind the scenes.

The smart peripheral approach means one can tell a PICAXE programmer, this is what you need, connect it like this, SEROUT ( "topic=", #dataValue, CR ) will send data, job done.

With the PICAXE sending AT Commands it's a whole lot more complicated, even the experienced will struggle and be battling PICAXE limited resources.

Two weeks ago I set up a simple mqtt-network in my house using an old Raspberry Pi as a mqtt-broker, and am currently investigating options to show/visualize data ... on a phone or PC
Which is the path I am going to walk. We'll see how it goes.
 

kranenborg

Senior Member
Some info here that I found useful:
  • Installing a Mosquitto Broker (mqtt protocol 3.1.1) on a RasPi, as well as instructions to test it: Link
  • A simple mqtt client on an Android phone: MQTT Dash
  • Lost of info on ESP8266 in combination with MicroPython, MQTT etc. etc. : RandomNerd Tutorials
  • Lots of info on the MQTT-protocol, platforms, dashboard etc. from Steve's Internet Guide (excellently explained)
/Jurjen
 

geoff07

Senior Member
This is another serious learning curve!

The only necessary involvement of a Picaxe for me is as the data source and thus sending to the server. Whether it does that via new code running on the ESP or using AT commands is not important, other than the effort and time involved.

I already have Mosquitto running under Ubuntu on a little fanless server that does other duties. I think the visualisation bit is a bit of a blank for me at the moment (installing a local thingspeak looked to be a tad complicated). Thanks for the references, I shall peruse, and also be interested in how you get on.
 

hippy

Technical Support
Staff member
MQTT is easy enough on a Raspberry Pi, as the broker, and for publishing and subscribing, and just a dozen lines to do both using the 'paho-mqtt' library for Python 2 or 3.

It would be easy enough to have that sending serial data to a PICAXE with handshaking and pacing, and/or taking serial data to publish from a PICAXE.

As to porting that to an ESP or PyBoard I don't know. It seems possible in C and MicroPython has its own MQTT library which I guess would be similar.

On using AT commands to do MQTT I would imagine not from what's within the MQTT library.

A far easier option there, is to take a Pi or Linux PC and run a server which can take data provided as per ThingSpeak which then publishes that via MQTT. That would be incredibly lightweight and can run on the same Pi or PC running the MQTT broker. All one would need to do in the PICAXE code is replace "thingspeak.com" with your Pi's IP address.

For viewing published data I would run yet another server on a Pi or Linux PC which can subscribe to the data streams and draw graphs or tables which can then be read and shown by any web browser or a DIY phone app.

It's moving the more difficult stuff to the server where it's easier to develop and run it, keeping the extremities, the PICAXE and phone parts, as simple as possible. The beauty of it all is that it can be developed, tested, and used, without a PICAXE, without an ESP, without a phone.

That's the approach I would take.
 

geoff07

Senior Member
I think I largely agree. And I now appreciate the limitations of the AT command set (expansive though it is) but where does this bit "a server which can take data provided as per ThingSpeak" come from? I could install Thingspeak, though it seems to be a mass of dependencies and non trivial. Do you have something else in mind?
 

hippy

Technical Support
Staff member
where does this bit "a server which can take data provided as per ThingSpeak" come from? I could install Thingspeak, though it seems to be a mass of dependencies and non trivial. Do you have something else in mind?
Rather than a complete ThingSpeak server installed locally, a more lightweight server can act as a bridge which presents itself as ThingSpeak would but instead publishes to MQTT. That should only be a couple of dozen lines of code.
 

geoff07

Senior Member
Thanks for all the various inputs. Having done a lot of reading around I am coming to a conclusion regarding what I need to do to achieve my goals and also be more widely useful. This summarises my current thinking. I would welcome any views regarding feasibility and usefulness to others.

Goal

to be able to add wifi to any Picaxe project by the addition of an ESP-01 wifi module, connected through the 8 pin header. To have a minimal footprint on the Picaxe software and hardware (Tx, Rx, perhaps one GPIO line). Once implemented, any Picaxe could communicate with any other Picaxe as long as they both had wifi access to the internet. What possibilities would that provide?

Design

There is a lot of material around and many competing solutions. The core issue seems to be the choice of http or mqtt for communication. What seems to be common to many solutions, avoid some shortcomings of the http traffic model, and thus appropriate here is the use of MQTT. This defines the dialogue between the parts. There is a good free 2012 pdf book from IBM that gives good background "Building Smarter Planet Solutions with MQTT and IBM WebSphere MQ telemetry". MQTT-SN is defined here: "MQTT for sensor networks Protocol Specification v1.2" In essence, the components are:

- MQTT-SN as the bi-directional dialogue between the Picaxe and the ESP-01 (the SN indicates a lightweight version of MQTT for sensors and is very simple). MQTT-SN has the added advantage that it does not rely on TCP/IP so could be used within local clusters of nodes, linking to a common EPS-01 as a gateway to the MQTT server, but using simple radio links. MQTT is not tied to any chip technology.

- the ESP-01 programmed to talk MQTT-SN hardwired to the Picaxe and to talk MQTT to the MQTT server via its wifi capability

- a Mosquitto server (mine runs on my Ubuntu print server)

- if required, an (existing) Android or other MQTT client for control and access to data.

Components to be created

a) code and a reference design for the Picaxe code, to implement MQTT-SN in Picaxe Basic

b) code and a reference design for the ESP-01, written in perhaps 'C'

The Picaxe component is straight-forward. The ESP-01 code requires decisions on which OS, which sdk and which development host and toolchain to use. This isn't straight-forward, as there has to be an MQTT library. And my 'C' skills (or Lua/C, C++, or Python or whatever) will need resurrection. Some SDKs are open-source, some are proprietary. Given that this is a hobby project I intend to stick to free tools. I'm tending towards LUA under the Espressif ESP8266 SDK at the moment, LUA being an interpreted language integrated with C. My development environment will be Ubuntu, though I also run XP in a VM in order to run PE6.

As sensors are typically battery-powered, there would be an emphasis on energy management. Both Picaxe and the ESP8266 have the necessary capabilities.

The purpose of posting this is simply to see if there is any interest from others and to receive any experience regarding effective toolchains under Linux. And perhaps in the hope that someone has done some of this already?
 

1968neil

Senior Member
i use MQTT with a home automation system (OpenHab)
I made my own sensors and used a little program called espeasy is loaded onto the ESP8266 chip.
This then gives you all the I/O's in a format that makes sense, including a serial interface.

Worth a look, may save you some time and effort ?

Regards
Neil
 

kfjl

Member
The purpose of posting this is simply to see if there is any interest from others
I think the interest for IOT stuff on this forum is evident.

Now that ESP are abandonning NON_OS SDK and supporting only RT_OS SDK, (which looks about as friendly as my dog when someone sits in his armchair), I'm inclined to think Buzby is right about Rev-Ed having missed out on a good opportunity here.
 

geoff07

Senior Member
Thanks guys. ESPeasy sounds interesting, I will have a look. I read that RT_OS didn't have an mqtt library, perhaps that is fixed. Are they abandoning NON_OS? The last version was 23 March this year.

All rev-ed need to do is build the interface that I describe. Then the rest is easy to add wifi to Picaxe. It wouldn't take long for someone writing C every day.
 

hippy

Technical Support
Staff member
I'm inclined to think Buzby is right about Rev-Ed having missed out on a good opportunity here.
But, let's be honest; that's mostly wanting Rev-Ed to do what people don't want to do themselves, would prefer not to have to figure out how to.

Here's a solution which works for a Raspberry Pi, Linux, Windows or Mac PC. Connect a PICAXE to a UART, or via an AXE027 or other USB-to-serial interface, and run this code with the configuration adjusted for the serial port used and for connection to an MQTT broker. Then every line of text sent from any PICAXE will be sent to every other PICAXE similarly connected -
Code:
#!/bin/usr/python2

import os; windows = os.name == "nt"
import sys
import serial
import threading
import time
import paho.mqtt.client as mqtt

uart_port        = "/dev/ttyUSB0"
uart_baud        = "9600"

mqtt_broker_ip   = "127.0.0.1"
mqtt_broker_port = "1883"
mqtt_username    = "pi"
mqtt_password    = "raspberry"
mqtt_topic       = "axenet"

client  = None
comPort = None

def on_connect(client, userdata, flags, rc):
  client.subscribe(mqtt_topic)

def on_message(client, userdata, msg):
  comPort.write(str(msg.payload) + "\r\n")

def UartReceiveThread():
  s = ""
  while True:
    if comPort.inWaiting() > 0:
      c = comPort.read(1)
      if c == "\n" or c == "\r":
        if s != "":
          client.publish(mqtt_topic, s)
          s = ""
      else:
        s = s + c
    else:
      time.sleep(0.1)

print("Bridge from " + uart_port + "," + uart_baud + ",N,8,1 to MQTT (" + mqtt_topic + ")")

comPort = serial.Serial \
          (
            port     = uart_port,
            baudrate = int(uart_baud),
            parity   = serial.PARITY_NONE,
            bytesize = serial.EIGHTBITS,
            stopbits = serial.STOPBITS_ONE
          )

threading.Thread(target=UartReceiveThread).start()

client = mqtt.Client()
client.username_pw_set(mqtt_username, mqtt_password)
client.on_connect = on_connect
client.on_message = on_message
client.connect(mqtt_broker_ip, int(mqtt_broker_port))
client.loop_forever()
It's as easy as that, though it should really be coded for Python 3 these days, and made more robust.

To turn it into something more useful and usable there are four things which need to be done -

1) Define what it is the PICAXE will send and what it receives, effectively creating the protocol for the PICAXE.

2) Decide upon a mechanism within that protocol for controlling and pacing data sent to the PICAXE to avoid having to use an X2 PICAXE with background receive, and to prevent the background receive buffer being overwhelmed, over-flowing.

3) Update the bridging code so it can handle what the PICAXE sends and create what should be sent to the PICAXE so it works as desired.

4) Port that to to whatever platform one wants to use if not wanting to use a Raspberry Pi.

Of course, if one doesn't want to use MQTT one will have to choose some other mechanism for getting data from one thing to another. And, if wanting to interface to something else, one needs to figure out how this will be done.

The key thing to remember is that one cannot really implement anything until it is decided what must be done, what it has to support. And that's got to be more detailed than "IoT" or "Connected by WiFi".
 

kfjl

Member
Are they abandoning NON_OS? The last version was 23 March this year.
See the last post of the "Where's the Wizard?" thread. Cut and pasted from ESP's doc.

But, let's be honest; that's mostly wanting Rev-Ed to do what people don't want to do themselves, would prefer not to have to figure out how to.
Let's be really honest. If everyone figured out everything by themselves, they'd program PICs in assembler and Rev-Ed would have no customers.

And to be really, really honest, if Picaxes don't make programming easier, what have they got to offer over their faster competitors?
 

Buzby

Senior Member
kfjl said:
I'm inclined to think Buzby is right about Rev-Ed having missed out on a good opportunity here.
But, let's be honest; that's mostly wanting Rev-Ed to do what people don't want to do themselves, would prefer not to have to figure out how to.
That's not quite what I intended to say, and I don't think anyone else saw it like that either.

Whether it's AXE-ESP-to-AXE-ESP, or AXE-ESP-to-AnyWebService, a well defined set of AT commands and some example PICAXE code would cover the vast majority of requirements. A couple of AXE-ESPs and a PE6 wizard could get 95% of users up and running in 30 minutes.

Why does PICAXE exist at all, if it's not to make the difficult easy ?.

As you know, I tend to write code to push the envelope on a PICAXE, and will wrestle with a problem until I solve it - or just give it up as a bad idea.

Not everyone is like that. Many users will not feel confident struggling with the vagaries of AT commands, or faffing with 3rd party SDKs, so they'll just use something else for their first IOT experiments.

Imagine some youngster at home showing his/her parents a Thingspeak graph of the temperature in their living room.

"Very clever !. How did you do that ?"

"I got an A******, a DS18B20, and a specific WiFi shield, then just loaded some code off the internet."

There will be some who will be happy with pre-written applications, and some who want to 'roll their own' system from scratch, but I'm sure most would just rather have some working modules and a bit of guidance.

That novice IOT user will at first be just using pre-written libraries, but if he wants to do more then he needs to do a bit of work. Would Rev-Ed prefer that work was on a PICAXE, or not ?.

I think it's a bit disengenious to say I was asking for Rev-Ed to do the all the work for me !

Cheers,

Buzby
 

Buzby

Senior Member
Anyway, to get this thread back on topic, last year I got an RPi and set up NODE-RED on it.
( If anyone doesn't know what NODE-RED is then Google is your friend. )

I installed a local MQTT broker ( Mosca I think ) and very soon I had RPi IO triggering MQTT messages through NODE-RED, and a second RPi ( also with NODE-RED ) subscribed to print text when the IO was triggered.

But that was as far as I bothered to go. Why did I need GHz quad-core processors and GBs of RAM just to send IO states ?. Surely I could do it with a pair of PICAXEs and some WiFi modules.

Then real life got in the way, and I didn't touch IOT again till lockdown.

Now I know how I would go about implementing a serious IOT project. It wouldn't be PICAXE based, ( you can't run NODE-RED on a PICAXE ), but a PICAXE as an MQTT IO node should be possible, if enough hoops are jumped through.

The whole IOT idea is too ambiguous to allow a 'one size fits all' solution, but if you can get a PICAXE to send messages to an MQTT broker that's a building block for many things. I'm fairly certain all you would need is the right sequence of AT commands on an ESP-01.

I think I'll resurrect my RPis.

Cheers,

Buzby
 
Last edited:

geoff07

Senior Member
The more I look, the more I find. Always hoping for convergence, right now it is continual divergence. But the learning is interesting.

Having had a tentative go at coding MQTT-SN for a Picaxe, I find that is likely to be impractical, for a number of reasons, particularly lack of data structures leading to complex code and the volume of individual message types to code, and then to follow the protocol logic which would eat into the code space. However, given the hardwired-to-esp-01 connection, this isn't going to be necessary as there is a QoS -1 option that can be used reliably. QoS -1 is a fire-and-forget simple message within mqtt-sn. The problem then reduces to code for an 8266 that can talk QoS -1 to the Picaxe (which is one simple message type) and can talk mqtt to Mosquitto.

It was unlikely that there wouldn't already be suitable code available, and there (almost) is. Others have mentioned several interesting places to look, and I'm currently looking at Esp-link, which seems to do what I want in principle (awesomeopensource.com/project/jeelabs/esp-link). It even includes an over-the-air programmer, though not for Picaxe sadly (but not surprisingly). Maybe the AVR programmer can be adapted. Assuming this or another does what I want with existing firmware (time will tell) it avoids the problem of choosing the OS as the developers get that problem.

As for what Rev-Ed can do, well I imagine that right now they will largely be furloughed (as a director of a small charity with furloughed staff I know this is rather inhibiting). But I'm tending to think that what is needed for Picaxe novices is not so much code as a comprehensive tutorial. Which is partly why I'm posting all this stuff, to get helpful direction. The problem is not a lack of solutions, it seems to be a surfeit of things that are almost-right but not actually fit-and-forget. Knowing what to choose, tweak, or expect seems to be the real problem in this vast aray of tools, platforms, firmware, protocols, servers etc.
 

geoff07

Senior Member
esp-link apparently needs the uC to talk SLIP. This seems quite unnecessary to me. However, has anyone got code for a simple slip interface?
 

hippy

Technical Support
Staff member
I don't think it does require SLIP because that would require a full TCP/UDP stack in the microcontroller. SLIP is just a means of getting a raw TCP/UDP packet from one thing to another via serial.

And that would be at odds with esp-link's claim to offer high-level functionality; one can't get much lower level than SLIP. But it's not clear how any of it works to me.

And I think trying to 'implement a solution' is the wrong way to go about it when it hasn't been decided what the exact problem is we are trying to solve. That's working from the middle out rather than from the outside in. The first requirement is defining what the PICAXE needs to do; what data it sends and receives and what convenient format it uses. The 'protocol' as mentioned earlier.

I would start from this premise -

I have a PICAXE with a button, DS18B20 temperature sensor and a LED. What do I send when the button is pushed, and how do I define to what it is sent ? Likewise what do I send when I have read a temperature and how do I define what it is sent to ? What do I receive to turn the LED on or off and how do I define where that will be received from.
 

lbenson

Senior Member
Here's a PICAXE MQTT node example using a 20m2, ESP8266 ESP-01, and Annex RDS to translate from PICAXE serial to MQTT and back, with control of LED and MQTT message notification of button press or change of temperature reading of DS18B20:
 

geoff07

Senior Member
This looks to be just the job, thank you. I hadn't yet come across AnnexRDS. So many firmwares.

re hippy's question, what to do, well this pretty much sums up my current ambition - to share data fields across Picaxes linked by wifi, and possibly also to control devices. And to do it via mqtt, and to write as little code as possible, preferably none outside the Picaxe (shoulders of giants).
 
Top