New (Beta) crossplatform (axepad-friendly) #include and #define preprocessor!

pleiser

Senior Member
#1
I decided to try to write a preprocessor for #include files for use with AXEpad on MacOS and Linux (presumably Windows too, if for some reason you're not using PE6), and found it was surprisingly easy to implement in python! This is an early revision right now, it doesn't currently support absolute file paths, but works great with relative paths for files in the same folder!

It works by opening the file passed as an argument, then copying it one line at a time into the output file, checking each line for #include statements, when it finds one it recursively opens that file and repeats the same process.

Any feedback and suggestions are welcome, hope you guys find it useful :)

here's the source code, save it as picaxeinclude.py in the same folder as the picaxe files you want to include to/from
Code:
#!/usr/bin/python

#PICAXE #include preprocessor
#Created by Patrick Leiser
import sys, getopt, os
inputfilename = ''
outputfilename = 'compiled.bas'
def main(argv):
    global inputfilename
    global outputfilename
    try:
        opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])
    except getopt.GetoptError:
        print 'test.py -i <inputfile> -o <outputfile>'
        sys.exit(2)
    for opt, arg in opts:
        if opt == '-h':
            print 'test.py -i <inputfile> -o <outputfile>'
            sys.exit()
        elif opt in ("-i", "--ifile"):
            inputfilename = arg
        elif opt in ("-o", "--ofile"):
            outputfilename = arg
    print 'Input file is ', inputfilename
    print 'Output file is ', outputfilename
    path=os.path.dirname(os.path.abspath(__file__))
    with open (path+'/'+outputfilename, 'w') as output_file:
        output_file.write("'-----PREPROCESSED BY picaxeinclude.py-----\n\n")
        output_file.write("'----SAVING AS "+outputfilename+" ----\n")
        output_file.write("'---BEGIN "+inputfilename+" ---\n")
    progparse(inputfilename)
        
def progparse(curfilename):
    print curfilename
    path=os.path.dirname(os.path.abspath(__file__))
    with open(path+'/'+curfilename) as input_file:
        for i, line in enumerate(input_file):
            workingline=line.lstrip()
            if workingline.lower().startswith("#include"):
                print"Found an include!"#TODO: Include logic
                workingline=workingline[9:].lstrip().split()[0]
                workingline=workingline.strip('"')
                print(workingline)
                with open (path+'/'+outputfilename, 'a') as output_file:
                    output_file.write("'---BEGIN "+workingline+" ---\n")
                progparse(workingline)
            else:
                with open (path+'/'+outputfilename, 'a') as output_file:
                    output_file.write(line)
                print line,
        print "{0} line(s) printed".format(i+1)
        with open (path+'/'+outputfilename, 'a') as output_file:
            output_file.write("\n'---END "+curfilename+"---\n")
    

if __name__ == "__main__":
    main(sys.argv[1:])
use it in a terminal like the following example:
Code:
/Path/to/program/directory/picaxeinclude.py -i infile.bas -o compiledoutput.bas
Hope this is useful! I'm short on time right now, but will be continuing to improve it, and welcome suggestions and feedback.
 

pleiser

Senior Member
#2
I've continued working on it, and now it works with both relative and absolute paths (assuming the absolute paths start with a "/" as they typically do on UNIX anyway). I also added full support for the #DEFINE preprocessor directive! I'm planning to work on support for #MACRO next.

I wonder if there's any chance that when complete this could be integrated into AXEpad? Or at least some optional modifications to AXEpad would make this much easier to use (specifically the ability to refresh the program's changes on disk without closing and reopening it, and the ability to suppress syntax errors on #include statements (although I suppose in most cases there would still be syntax errors with labels and symbols defined in other files, so this change might not help much) @hippy is there any chance that it could be integrated into axepad when complete, or at least some of the other changes to improve usability?

Here's the current version of the code, supporting both #include and #define:
Code:
#!/usr/bin/python

#PICAXE #include and #define preprocessor
#Created by Patrick Leiser
import sys, getopt, os, datetime
inputfilename = ''
outputfilename = 'compiled.bas'
outputpath=""
definitions=dict()
def main(argv):
    global inputfilename
    global outputfilename
    global outputpath
    try:
        opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])
    except getopt.GetoptError:
        print 'picaxepreprocess.py -i <inputfile> -o <outputfile>'
        sys.exit(2)
    for opt, arg in opts:
        if opt == '-h':
            print 'picaxepreprocess.py -i <inputfile> -o <outputfile>'
            sys.exit()
        elif opt in ("-i", "--ifile"):
            inputfilename = arg
        elif opt in ("-o", "--ofile"):
            outputfilename = arg
    print 'Input file is ', inputfilename
    print 'Output file is ', outputfilename, '\n'
    path=os.path.dirname(os.path.abspath(__file__))
    if outputfilename.startswith("/"):
        outputpath=""
    else:
        outputpath=path+'/'
    with open (outputpath+outputfilename, 'w') as output_file:   #desribe output file info at beginning in comments
        output_file.write("'-----PREPROCESSED BY picaxeinclude.py-----\n")
        output_file.write("'----UPDATED AT "+ datetime.datetime.now().strftime("%I:%M%p, %B %d, %Y") + "----\n")
        output_file.write("'----SAVING AS "+outputfilename+" ----\n\n")
        output_file.write("'---BEGIN "+inputfilename+" ---\n")
    progparse(inputfilename)   #begin parsing input file into output
        
def progparse(curfilename):
    global definitions
    print("including file " + curfilename)
    path=os.path.dirname(os.path.abspath(__file__))+"/"
    if curfilename.startswith("/"):    #decide if an absolute or relative path
        curpath=""
    else:
        curpath=path
    with open(curpath+curfilename) as input_file:
        for i, line in enumerate(input_file):
            workingline=line.lstrip()
            if workingline.lower().startswith("#include"):
                workingline=workingline[9:].lstrip().split("'")[0].split(";")[0].rstrip()     #remove #include text, comments, and whitespace
                workingline=workingline.strip('"')         #remove quotation marks around path
                #print(workingline)
                with open (outputpath+'/'+outputfilename, 'a') as output_file:
                    output_file.write("'---BEGIN "+workingline+" ---\n")
                progparse(workingline)
            elif workingline.lower().startswith("#define"):     #Automatically substitute #defines
                workingline=workingline[8:].lstrip().split("'")[0].split(";")[0].rstrip()
                definitions[workingline.split()[0]]=(workingline.split(None,1)[1])   #add to dictionary of definitions
                with open (outputpath+'/'+outputfilename, 'a') as output_file:
                    output_file.write(line.rstrip()+"      'DEFINITION PARSED\n")
            else:
                for key,value in definitions.items():
                    if key in line:
                        #print("substituting definition")
                        line=line.replace(key,value)
                        line=line.rstrip()+"      'DEFINE: "+value+" SUBSTITUTED FOR "+key+"\n"
                with open (outputpath+outputfilename, 'a') as output_file:
                    output_file.write(line)
                #print line,
        #print "{0} line(s) printed".format(i+1)
        with open (outputpath+'/'+outputfilename, 'a') as output_file:
            output_file.write("\n'---END "+curfilename+"---\n")
    #print (definitions)
    

if __name__ == "__main__":
    main(sys.argv[1:])
 

Technical

Technical Support
Staff member
#3
Were you aware that #define, #ifdef, #ifndef etc. all already work in axepad on Linux? Therefore adding another partial implementation for #define into a third party preprocessor is therefore not ideal, it needs to be complete (ie include all #ifdef #endif type statements as well) to be much use.
 

pleiser

Senior Member
#4
@Technical, yes, I know that #define works with #ifdef and #ifndef etc, in axepad, however axepad’s implementation doesn’t work with the new behavior in PE6 of letting you define code substitutions, like the example in the manual of
Code:
 #DEFINE SetBackLedOn    b0 = 255 : Gosub SendBackLED
Also my preprocessor doesn’t remove the #define statements from the output code, so when run through axepad (that’s the current procedure I’ve been using, open the output code in axepad to program to the PICAXE), the #ifdef type behavior still functions properly.

In other news I think I’ve got #macro working! It’s more complex than the others, more testing needed, but it’s giving me valid results right now :).

TL;DR regarding the #define, I am aware of the partial implementation of #define in axepad, and my preprocessor is design to complement it’s functionality, but not to replace it. In my testing using my preprocessor and axepad together results in proper behavior for both functions of #define.
 

pleiser

Senior Member
#5
I just tested it again with both behaviors of #define, and after a small modification to fix a minor bug, it works great. here's an example input and output from the preprocessor of both types of define:
Input:
Code:
#define testing
#define message sertxd("New Define")

main:
     #ifdef testing
         sertxd("Old Defines Still Work")
     #endif
     message
and the output:
Code:
#define testing      'DEFINITION PARSED
#define message sertxd("New Define")      'DEFINITION PARSED

main:
     #ifdef testing
         sertxd("Old Defines Still Work")
     #endif
     sertxd("New Define")      'DEFINE: sertxd("New Define") SUBSTITUTED FOR message
as you can see, the program preserves the #defines and #ifdefs, leaving them for axepad to deal with as it always does.
Here's some examples of it's other functionality too:
Code:
'in file 1.bas
goto init
#include "2.bas"    ;test
#include "/Users/username/Documents/Programming/Picaxe Include/4.bas

init:
toggle 1
main:
    gosub testloop
    toggle 2
     SetHeadingSpeed(23, 7)
     pause 500
     SetHeadingSpeed(92,34)
Code:
'in file 2.bas
#define MAGIC_NUMBER "83838838"

#DEFINE SetBackLedOn    b0 =255 : toggle 2


#MACRO SerialMacro(num)

    sertxd("this is a macro ",num)

#ENDMACRO

#Macro Assign( var, expression )
     Let var = expression
#EndMacro

Assign( w0, 1 * 2 + 3 )

testloop:
    sertxd(MAGIC_NUMBER)
    SetBackLedOn
    SerialMacro("37")
    pause 1000
    toggle 1
return
Code:
'in 4.bas
    toggle 2
    pause 500
 
#MACRO SetHeadingSpeed(H, S)
       w1 = H
    b0 = S
       Gosub SendHeadingSpeed
#ENDMACRO


SendHeadingSpeed:
debug
    return
and the output of the preprocessor:
Code:
'-----PREPROCESSED BY picaxepreprocess.py-----
'----UPDATED AT 11:30AM, October 01, 2018----
'----SAVING AS 3.bas ----

'---BEGIN 1.bas ---

goto init
'---BEGIN 2.bas ---

#define MAGIC_NUMBER "83838838"      'DEFINITION PARSED

#DEFINE SetBackLedOn    b0 =255 : toggle 2      'DEFINITION PARSED

'PARSED MACRO SerialMacro
'PARSED MACRO Assign

'--START OF MACRO: Assign
  Let w0 = 1 * 2 + 3
'--END OF MACRO: Assign( w0, 1 * 2 + 3 )

testloop:
    sertxd("83838838")      'DEFINE: "83838838" SUBSTITUTED FOR MAGIC_NUMBER
    b0 =255 : toggle 2      'DEFINE: b0 =255 : toggle 2 SUBSTITUTED FOR SetBackLedOn
    '--START OF MACRO: SerialMacro

sertxd("this is a macro ", "37")

'--END OF MACRO: SerialMacro("37")
    pause 1000
    toggle 1
return
'---END 2.bas---
'---BEGIN /Users/username/Documents/Programming/Picaxe Include/4.bas ---
    toggle 2
    pause 500
 
'PARSED MACRO SetHeadingSpeed

SendHeadingSpeed:
debug
    return
'---END /Users/username/Documents/Programming/Picaxe Include/4.bas---
#define testing      'DEFINITION PARSED
#define message sertxd("New Define")      'DEFINITION PARSED

init:
toggle 1
main:
    gosub testloop
    toggle 2
     '--START OF MACRO: SetHeadingSpeed
       w1 = 23
       b0 = 7
       Gosub SendHeadingSpeed
'--END OF MACRO: SetHeadingSpeed(23, 7)
     pause 500
     '--START OF MACRO: SetHeadingSpeed
       w1 = 92
       b0 = 34
       Gosub SendHeadingSpeed
'--END OF MACRO: SetHeadingSpeed(92,34)
  
  
goto main

'---END 1.bas---
Sorry if the readability as to what the code's doing isn't great, it's what I wrote to test the functions as I added them (though I did try to remove a few redundantly demonstrated functions).
The main thing I haven't implemented yet is the use of defines when there's a parameter passed in like with macros, but that should be relatively simple now that I've got that behavior working with macros.
Also note that it is not limited to including files of the format number.bas, that's just what I've been using with my testing files, but it is perfectly capable of using any filename.
 
Last edited:

pleiser

Senior Member
#6
here's the latest form of the preprocessor's source code:
Code:
#!/usr/bin/python

#PICAXE #include, #define, and #macro preprocessor
#todo: make defines behave like single line macros, allowing(parameters)
#todo: more thoroughly test macro behaviors, especially with parentheses
#Created by Patrick Leiser
import sys, getopt, os, datetime, re
inputfilename = ''
outputfilename = 'compiled.bas'
outputpath=""
definitions=dict()
macros=dict()
def main(argv):
    global inputfilename
    global outputfilename
    global outputpath
    try:
        opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])
    except getopt.GetoptError:
        print 'picaxepreprocess.py -i <inputfile> -o <outputfile>'
        sys.exit(2)
    for opt, arg in opts:
        if opt == '-h':
            print 'picaxepreprocess.py -i <inputfile> -o <outputfile>'
            sys.exit()
        elif opt in ("-i", "--ifile"):
            inputfilename = arg
        elif opt in ("-o", "--ofile"):
            outputfilename = arg
    print 'Input file is ', inputfilename
    print 'Output file is ', outputfilename, '\n'
    path=os.path.dirname(os.path.abspath(__file__))
    if outputfilename.startswith("/"):
        outputpath=""
    else:
        outputpath=path+'/'
    with open (outputpath+outputfilename, 'w') as output_file:   #desribe output file info at beginning in comments
        output_file.write("'-----PREPROCESSED BY picaxepreprocess.py-----\n")
        output_file.write("'----UPDATED AT "+ datetime.datetime.now().strftime("%I:%M%p, %B %d, %Y") + "----\n")
        output_file.write("'----SAVING AS "+outputfilename+" ----\n\n")
        output_file.write("'---BEGIN "+inputfilename+" ---\n")
    progparse(inputfilename)   #begin parsing input file into output
        
def progparse(curfilename):
    global definitions
    savingmacro=False
    print("including file " + curfilename)
    path=os.path.dirname(os.path.abspath(__file__))+"/"
    if curfilename.startswith("/"):    #decide if an absolute or relative path
        curpath=""
    else:
        curpath=path
    with open(curpath+curfilename) as input_file:
        for i, line in enumerate(input_file):
            workingline=line.lstrip()
            if workingline.lower().startswith("#include"):
                workingline=workingline[9:].lstrip().split("'")[0].split(";")[0].rstrip()     #remove #include text, comments, and whitespace
                workingline=workingline.strip('"')         #remove quotation marks around path
                #print(workingline)
                with open (outputpath+'/'+outputfilename, 'a') as output_file:
                    output_file.write("'---BEGIN "+workingline+" ---\n")
                progparse(workingline)
            elif workingline.lower().startswith("#define"):     #Automatically substitute #defines
                workingline=workingline[8:].lstrip().split("'")[0].split(";")[0].rstrip()
                try:
                    definitions[workingline.split()[0]]=(workingline.split(None,1)[1])   #add to dictionary of definitions
                except:
                    print("old define found, leaving intact")
                
                with open (outputpath+'/'+outputfilename, 'a') as output_file:
                    output_file.write(line.rstrip()+"      'DEFINITION PARSED\n")
            elif workingline.lower().startswith("#macro"):     #Automatically substitute #macros
                savingmacro=True
                workingline=workingline[7:].lstrip().split("'")[0].split(";")[0].rstrip()
                macroname=workingline.split("(")[0].rstrip()
                print macroname
                with open (outputpath+outputfilename, 'a') as output_file:
                    output_file.write("'PARSED MACRO "+macroname)
                macrocontents=workingline.split("(")[1].rstrip()
                macros[macroname]={}
                argnum=0
                while(1):
                    argnum+=1
                    if macrocontents.strip()==")":
                        print("no parameters to macro")
                        macros[macroname][0]="'Start of macro: "+macroname
                        print macros
                        break
                    else:
                        macrocontents=macrocontents.rstrip(")").strip("(")
                        macros[macroname][argnum]=macrocontents.split(",")[0].rstrip()   #create spot in dictionary for macro variables, but don't populate yet
                        if "," in macrocontents:
                            macrocontents=macrocontents.split(",")[1].strip().rstrip()
                        else:
                            print("finished parsing macro contents")
                            macros[macroname][0]="'--START OF MACRO: "+macroname+"\n"
                            break
            elif savingmacro==True:
                if workingline.lower().startswith("#endmacro"):
                    savingmacro=False
                    macros[macroname][0]=macros[macroname][0]+"'--END OF MACRO: "+macroname
                    #print macros
                else:
                    macros[macroname][0]=macros[macroname][0]+line
            else:
                for key,value in definitions.items():
                    if key in line:
                        #print("substituting definition")
                        line=line.replace(key,value)
                        line=line.rstrip()+"      'DEFINE: "+value+" SUBSTITUTED FOR "+key+"\n"
                for key, macrovars in macros.items():
                    if key in line:
                        params={}
                        argnum=0
                        macrocontents=line.split(key)[1]
                        macrocontents=macrocontents.strip("(").strip(")")
                        while(1):
                            argnum+=1
                        
                            if "," in macrocontents:
                                params[argnum]=macrocontents.split(",")[0].rstrip()   #
                                
                                macrocontents=macrocontents.split(",")[1].strip().rstrip()
                            else:
                                print("finished parsing macro contents")
                                params[argnum]=macrocontents.split(",")[0].rstrip()
                                #params[argnum]=params[argnum].strip("(").strip(")").strip()
                                print params
                                break
                        line=line.replace(key,macrovars[0])
                        print macrovars
                        for num, name in macrovars.items():
                                if name in line:
                                    if num>0:
                                        line=re.sub(r"\b%s\b" % name,params[num],line)
                        line=line[:line.rfind(")", 0, line.rfind(")"))]+line[line.rfind(")", 0, line.rfind(")"))+1:]
                with open (outputpath+outputfilename, 'a') as output_file:
                    output_file.write(line)
                #print line,
        #print "{0} line(s) printed".format(i+1)
        with open (outputpath+'/'+outputfilename, 'a') as output_file:
            output_file.write("\n'---END "+curfilename+"---\n")
    #print (definitions)
    

if __name__ == "__main__":
    main(sys.argv[1:])
It did get a bit more messy when implementing #macro, Ill probably try to clean up unnecessary parts (like the redundant .strip() calls) at some point.
 
Last edited:

steliosm

Senior Member
#7
Were you aware that #define, #ifdef, #ifndef etc. all already work in axepad on Linux? Therefore adding another partial implementation for #define into a third party preprocessor is therefore not ideal, it needs to be complete (ie include all #ifdef #endif type statements as well) to be much use.
Any ideas when #include is also going to be supported in LinAxePad?
 

pleiser

Senior Member
#8
By the way, I've more thoroughly documented and published the code for the preprocessor on github, for anyone interested. I've been successfully using the preprocessor with my robotics team for the last several months, and hope others have found it useful as well. If you have, I'd love to hear about it!
 
Top