Receiving variable length serial data

[micrometal]

I want to link a Picaxe project with a mobile 'phone app. I have a Bluetooth link working with an HC-05 module - no problems. The data that I want to transfer (in both directions) are variable length binary records. The first byte of data defines the data length, typically 10 to 20 bytes.

I think that I know how to transmit variable length serial data records (in a loop - one byte at a time). I cannot work out how to receive variable length data - how can I do that?

 

[Aries]

The easiest way (if you are using an X2 chip) is to use hserin, and then use hserptr to indicate how much data you have received. You can then check it against the first byte and handle any discrepancies.
Alternatively, if the data is coming slowly enough, you can simply receive one byte at a time in a tight loop. The first byte will tell you how big the loop needs to be. HOWEVER, this does depend on not losing any data (otherwise the loop will not exit cleanly, or possibly at all).

 

[micrometal]

I am planning to use a 08M2, or maybe a 14M2. I think that the nature of your reply tells me that I am not missing something obvious - I thought that I might be.

If there is really no straightforward method then I will have to think again.

 

[inglewoodpete]

Possibly the best way to receive variable length data strings is the use a unique start-of-string character. Eg the '$' (dollar) symbol used in GPS data strings, although I'm not certain that this character is part of the UK character set. Regardless, the character you select must not appear elsewhere in the string being received.

When your chosen unique start-of-string character is received, your PICAXE serial reception code needs to reset all of its related variables and prepare to receive string-length value, followed by the actual data.

The M2-series chips can struggle to keep up with fast incoming data, especially strings of variable lenght, so it will need to run as fast as possible (32MHz). Also, it will help the have the bluetooth sending data as slowy as possible Eg 4800 baud or less.

As suggested by Aires I would be using a 20X2, due to its 'scratchpad' and backgroud serial data reception capability.

 

[micrometal]

I have just looked through my bits box and found a 20X2 that I had forgotten I had, so I guess that the world is trying to tell me something. I will certainly give it a try, although I am a bit constrained by board space. I will in any case learn something new - I have never looked at X2 modules before and I can see that they have a lot of attractions.

On the subject of learning something new, Aries answer started me looking at hserin and I discovered that it has a non-blocking action (for two bytes) that, as I understand it, works on all modules. If so that might solve a problem I have with another project which I have already split across two 08M2s so that I can run two interrupt-driven time critical inputs. I was wondering how to add in serial communication (across a Bluetooth link to a 'phone again) without losing the interrupts, and hserin could be the answer. I mention this because I believe that whatever question you ask on this forum you are probably going to learn something useful.

 

[Flenser]

The M2 and X2 chips do both have a background receive feature but the details are distinctly different.

The background receive feature on the M2 chips simply takes advantage of the underlying PIC serial hardware.
The PIC chips have a shift register and a 1 byte buffer. When the shift register is filled it overwrites the contents of the buffer.
Provided that you keep up with reading received bytes from this buffer the hardware provides the 2 byte buffer that you refer to but if you are receiving many more than 2 bytes then you still end up having to process each byte on average in the time it takes one byte to be transmitted.

On the x2 chips the underlying PIC hardware is the same but the PICAXE firmware works in the background to take care of copying the received characters out of the PIC hardware buffer into the PICAXE BASIC scratchpad memory.
The scratchpad memory area used as a large circular buffer which makes it more powerful background receive function. The scratchpad memory is 128 bytes on the 20X2 and 1024 bytes on the 28X2 & 40X2 so if the message length is less than this and there is a long delay between messages then it is easier for your program to perform the processing of the message.

 

[hippy]

Provided that you keep up with reading received bytes from this buffer the hardware provides the 2 byte buffer that you refer to but if you are receiving many more than 2 bytes then you still end up having to process each byte on average in the time it takes one byte to be transmitted.

Rate of received data is the big factor. If it's slow enough one could use SERIN, and M2 HSERIN will be even better, but most data packets in my experience will require X2 background HSERIN.

There is also the issue of a second packet arriving while you are still dealing with the first. For SERIN and M2 HSERIN one may miss that subsequent packet, entirely or part of it, and missing part of it can mean it's hard to get back into sync. X2 background HSERIN should mean less chance of packets being lost and can also help in recovering sync.

PICAXE X2 chips also have 4K of program space compared to the M2 chip's 2K which may also prove useful in allowing receive and recovery code to be more robust.

Do you have control over the phone app internals, what it sends to the PICAXE and how ?

 

[micrometal]

Do you have control over the phone app internals, what it sends to the PICAXE and how ?

In the case of the 2 x 08M2 project, this is essentially a data logging application and I am writing the Android app, so have complete control. The messages into the 08M2 are simple - "Start", "Stop" and "Send me your data" - so I need send only a single command byte. Also they will be infrequent. It sounds like I should be able to sort out something using hserin. I can, I hope, time the serout activity to occur in the quiet periods between the timed input events.

Thank you also, @Flenser - your explanation is very clear.

 

[AllyCat]

Hi,

Yes indeed, it is perfectly possible to use an interrupt routine with the HSERIN hardware of any M2 PICaxe, but there are two provisos: Firstly, the M2 has no internal interrupt capability so it's necessary to link an "interruptable" pin to the HSERIN pin, to allow (for example) the start pulse to initiate the interrupt (then the interrupt routine must check if/when a byte has fully arrived). But also, there appears to be a bug in the HSERIN command which makes it much slower than expected, and a much better solution is to use PEEKSFR commands to read the hardware buffer and its flags. There is a long thread on the topic, but some of the nitty gritty is from post #18 here onwards.

Cheers, Alan.

 

[micrometal]

Hello Allycat. I think that I have seen that long thread before when I was looking for something else. Getting something to work on an 08M2 is probably well above my pay-grade at the moment even without in-built bugs, particularly as I cannot spare another interrupt input.

Just at the moment I am back working on my initial topic - receiving and transmitting variable length data. Early days yet, and more questions to follow, probably.

 

[AllyCat]

Hi,

Ah yes, I'd forgotten that ALL the pins on an 08M2 are interruptable (except perhaps C.0) which is actually more than on a 14M2 (and I believe in practice as many as on any of the other M2 chips). So there is no need to link an additional input pin when using the 08M2. Personally, I usually also move HSERIN onto the programming/serial input pin (using the APFCON SFR), but HSERIN on M2s still must be "Idle High" (RS232-TTL format), so I do accept that overall it does need some quite "advanced" programming methods.

Cheers, Alan.

 

[hippy]

Just at the moment I am back working on my initial topic - receiving and transmitting variable length data. Early days yet, and more questions to follow, probably.

The first thing to do is characterise the incoming serial to the PICAXE before deciding what is the best way to deal with it or which PICAXE would be best to use.

If it's possible to make the data packet fixed size that may make things easier for the PICAXE while not having a detrimental effect.

 

[lbenson]

Here is my sample M2 background receive code:

symbol command= b27

start:
  pause 1000
  sertxd ("Hello from 08relay",13,10)
  hsersetup B2400_4, %00001000 ' input inverted ("T"-bit2=0), disable hserout

main: 
  do
    command = 0xFF ' invalid value
' see if we have serial input
    hserin Tmpvar
    select case command ' falls through if command still = 0xFF
      case "1" ' do something
      case "2" ' do something else    
    end select
  loop

 

[hippy]

A fast option for spinning in a loop waiting for a byte to be received is ...

Do
  w0 = -1
  Do
    HSerin w0
  Loop Until b1 = 0
  Select Case b0
    ...

This is much faster than letting a SELECT-CASE fall-through.

It also allows all bytes 0-255 to be received. The trick is that HSERIN writes to the word 'w0' clearing the MSB 'b1' and setting 'b0' to byte received when one has been.

The direct SFR code I believe would be, untested -

Symbol PIR1  = $11 ; $011
Symbol RCREG = $79 ; $199
Do
  Do
    PeekSfr PIR1, b0
  Loop Until bit5 = 1
  PeekSfr RCREG, b0
  Select Case b0
    ...

 

[micrometal]

Okay - I had been working on this before I read the last three posts, so bear that in mind. I am back on the original quest of this thread, and using an 20X2. I have no problem receiving data - I have checked that I am receiving what I am sending (not shown in the code below). I would like help in sending data back (the easy part!). I am unsure about the roles of the scratchpad pointer "ptr" and the hserin pointer "hserptr" - I have tried both but not successfully. Here is my code ...

' Bluetooth 20X2 wrap test : Re-transmit variable length incoming data

' Data is received in the background and stored in scratchpad memory.
' The main loop checks the scratch pad for new data. When found the
' data is retransmitted after a short delay.

    #NO_DATA                    ' Code only
    
    symbol    BT_out    = C.0        ' Bluetooth Tx
    symbol    BT_in    = B.6        ' Bluetooth Rx
    symbol    size    = b1        ' Length of received data   
    
            setfreq        M32                ' Run at 32MHz
    
            hsersetup    B9600_32, 1        ' 9600 baud, background mode
                
    main:   
            if hserflag > 0 then goto data_received
            pause     500
            goto main           
            
    data_received:
            pause 2000                    ' Ensure all data received
            size = hserptr                ' Remember data length
            debug                        ' Check message received
            hserptr = 0                    ' Point to first data byte
            do
                hserout    0,(hserptr)
                hserptr = hserptr + 1
            loop while hserptr < size
            hserptr = 0                    ' Reset hserin pointer
            hserflag = 0                ' Clear flag
            goto main

Having looked at the 20X2 for the first time, I think that I should be using it with the other project I have mentioned (the timer data logger). As I understand it the 20X2 has three pins that can support independent interrupts - exactly what I need.

 

[Aries]

ptr is a pointer to a place in the scratchpad. It is the one normally used when programming. hserptr is a pointer which hserin uses to indicate the next byte in scratchpad to be written. It is not generally used in any other way.

hserout takes a list of variables/constants. In your example
hserout 0,(hserptr)
you are sending one value, which is the value of hserptr (i.e. 0 on the first pass, 1 on the second, etc). What you want to send is the value in the location indicated (location 0 in scratchpad, location 1, etc). This is where it is easier to use ptr, because @ptr means "the value in the location pointed to by ptr". So,
hserout 0,(@ptr) ought to work. You can also simplify things by using "ptrinc" which uses ptr and then increments it by 1.
Your loop then becomes

ptr = 0
do
  hserout 0,(@ptrinc)
loop while ptr < size

 

[micrometal]

Actually using hserout was a second string when my first attempts all failed. I had started with exactly the code that you show. I have just tried it again, still with no success. Here is the code that I am using now ...

    data_received:
            pause 2000                    ' Ensure all data received
            size = hserptr                ' Remember data length
            debug                        ' Check message received
            ptr = 0                        ' Point to first data byte
            do
                hserout    0,(@ptrinc)
            loop while ptr < size
            hserptr = 0                    ' <==== NB below
            ptr = 0                        ' Reset hserin pointer (?)
            hserflag = 0                ' Clear flag
            goto main

Note that I am still resetting hserptr to zero. If I don't do this, successive incoming data packets carry on filling up the scratchpad, following on from the previous data. Maybe this is a factor - I don't know, but I feel that I am missing something obvious and will be kicking myself later.

EDIT : Yes - I was right (and wrong)! I had mis-plugged the output lead - It wasn't in C.0!

 

[inglewoodpete]

You can either reset hSerPtr at the end of processing a message (if the event can clearly be determined in software) or restrict the size of the receive data space.

I have limited the hSerIn range and used the range as a circular buffer with the following code when I had to share the scratchpad with other functionality:

Symbol False         = 0
Symbol True          = 1
Symbol mskSerInRange = %00111111 'Bytes 0-63 or 0-32 etc
Symbol mskSerialOnly = %00100000
Symbol flgSerialOnly = %00100000

      'Include the following line in your initialisation (startup) code
      SetIntFlags flgSerialOnly, mskSerialOnly        'Set hSerial to interrupt


Interrupt:If hSerInFlag = True Then                   'Restrict pointer to 0-63 (Must appear
            hSerInPtr = hSerInPtr And mskSerInRange   '        first in the interrupt routine)
            hSerInFlag = False
          EndIf
          SetIntFlags flgSerialOnly, mskSerialOnly    'Set hSerial to interrupt
          Return

 

[Aries]

One thing you should always be doing (if you are not doing it at the moment), is to pepper your code with "print statements" - i.e. sertxd statements to output what is going on. So, for example, if you had written

                sertxd(CR,LF)
do
                hserout    0,(hserptr)
                sertxd(" ",#hserptr)               ' use # to output value in printable form, rather than raw data
                hserptr = hserptr + 1
            loop while hserptr < size

you would have realised that even if your output was working, it was outputting the wrong values.

As far as resetting hserptr to zero, there is no inherent problem in setting it to any value, so as to use whatever part of scratchpad you want to. If you have some of your own variables stored in scratchpad, you will want to start hserptr after those (if they are at the beginning), or finish hserptr before they start (if they are at the end). InglewoodPete's nethod of masking works as long as you are in the range 0-(2^n)-1. Otherwise, if your range is 0-RANGE you can replace the And statement with

hserptr = hserptr // RANGE

 

[hippy]

This is the code I would use to show what characters were received in the PE6 Terminal window -

#Picaxe 20X2
#Terminal 138400
#No_Data

Symbol reserveW0 = w0 ; b1:b0
Symbol rx        = b2
Symbol msb       = b3
Symbol lsb       = b4

PowerOnReset:
  SetFreq M32
  HSerSetup B9600_32, 1
  SerTxd("Started", CR, LF)

MainLoop:
  Do
    Gosub ReadRxChar
    Gosub ShowRxChar
  Loop

ReadRxChar:
  Do : Loop While ptr = hSerPtr
  rx = @ptrInc
  Return

ShowRxChar:
  msb = rx / 16 + "0" : If msb > "9" Then : msb = msb + 7 : End If
  lsb = rx & 15 + "0" : If lsb > "9" Then : lsb = lsb + 7 : End If
  If rx >= $20 and rx < $7F Then
    SerTxd("Got $", msb, lsb, " ", #rx, " '", rx, "'", CR, LF)
  Else
    SerTxd("Got $", msb, lsb, " ", #rx, CR, LF)
  End If
  Return

That uses the entire Scratchpad as a circular HSERIN buffer, will be emptying the buffer while it's being filled so under most circumstances won't lose any data. There's very little to it; 'ReadRxChar' waits until there has been a character received, then gets what that was. ' ShowRxChar' simply prints it to the Terminal as hex and decimal and ASCII if a printable character.

How to move on from there is the next question. First thing to do is to wait for a data packet, stop further data packets arriving, show what one has received, re-enable reception and repeat, untested -

PowerOnReset:
  SetFreq M32
  SerTxd("Started", CR, LF)

MainLoop:
  Do
    HSerSetup B9600_32, 1
    Gosub WaitForDataPacket
    Pause 8000              ; Wait 1 second at 32MHz
    HSerSetup OFF
    Do
      Gosub ReadRxChar
      Gosub ShowRxChar
    Loop Until ptr = hSerPtr
    SerTxd("---", CR, LF)
  Loop

WaitForDataPacket:
  Do : Loop While ptr = hSerPtr
  Return

 

[inglewoodpete]

One thing you should always be doing (if you are not doing it at the moment), is to pepper your code with "print statements" - i.e. sertxd statements to output what is going on.

A word of caution when using SerTxd to debug "background" data reception and potentially, transmission.

SerTxd is "bit-banged" and requires all system (internal/firmware) interrupts to be disabled while transmitting data. This is in order to maintain control of the critical timing of SerTxd's data bit transmission and reception. This can play havoc with incoming reception and outgoing transmission and results in data corruption.

 

[micrometal]

Thanks, everyone, for your suggestions. As you might suppose, this project is as much about me learning by doing as anything else. I have done a fair bit of programming in my time but I have always used quite helpful IDEs (which the PICaxe Editor is not) and have mainly been manipulating data. Data tends to do what it is told; hardware, on the other hand, seems full of surprises.

I have written several Android apps and I think that a reliable data link between a 'phone and a PICaxe would be something useful. It is going to take me a while to pick all the bits out of your suggestions, but they are exactly what I am looking for. The code that I posted above is a primitive first pass. I will sort that out. At the moment I am receiving messages reliably, but sending them back is only sometimes successful. As Aries suspected, a serious limitation at the moment is my inexperience of debugging techniques in PICaxe, plus a general ignorance of the PICaxe environment. Thanks again for your help.